Experimental: Unified LSP server in rewatch #8243
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add optional OTLP tracing export to rewatch, controlled by the OTEL_EXPORTER_OTLP_ENDPOINT environment variable. When set, rewatch exports spans via HTTP OTLP; when unset, tracing is a no-op. Instrument key build system functions (initialize_build, incremental_build, compile, parse, clean, format, packages) with tracing spans and attributes such as module counts and package names. Restructure main.rs to support telemetry lifecycle (init/flush/shutdown) and fix show_progress to use >= LevelFilter::Info so -v/-vv don't suppress progress messages. Also print 'Finished compilation' in plain_output mode during watch full rebuilds. Introduce a new Vitest-based test infrastructure in tests/rewatch_tests/ that replaces the bash integration tests. Tests spawn rewatch with an OTLP endpoint pointing to an in-process HTTP receiver, collect spans, and snapshot the resulting span tree for deterministic assertions. Update CI, Makefile, and scripts/test.js to use the new test runner.
When stdin is a pipe (not a TTY), spawn a background thread that monitors for EOF. This allows a parent process (such as the test harness) to signal a graceful shutdown by closing stdin, without relying on signals or lock file removal.
Add mtime and content-hash based deduplication to filter out phantom and duplicate file system events. Normalize event kinds from atomic writes (temp file + rename) so they are treated as content modifications rather than create/remove cycles that trigger unnecessary full rebuilds. This fixes issues on macOS (Create events from atomic writes), Linux (duplicate inotify IN_MODIFY events), and Windows (Remove+Rename sequences from atomic writes).
On Windows, bsc writes CRLF to stdout in text mode. When the original source file uses LF line endings, the formatted output would introduce unwanted CRLF conversions. Detect the original file's line ending style and normalize the formatted output to match.
Propagate parent span through rayon in build.parse so build.parse_file spans are properly nested under build.parse instead of appearing as orphaned root spans. Enrich build.compile_file span with package, suffix, module_system, and namespace attributes for better observability. Handle invalid config changes gracefully during watch mode: replace .expect() with match to report the error and continue watching, allowing the user to fix the config without restarting the watcher.
Add 7 new fixture packages to cover more configuration dimensions: - commonjs: CommonJS module output with .bs.js suffix - namespaced: namespace package with TestNS - noop-ppx: lightweight cross-platform no-op PPX for testing - with-deps: package depending on rescript-bun for clean tests - with-dev-deps: multi-source dirs with dev dependencies - with-jsx: JSX v4 with @rescript/react - with-ppx: PPX integration using noop-ppx Enhance test helpers: - Normalize CRLF line endings in process output for Windows - Support .bs.js artifacts in sandbox cleanup detection - Add createCli, readFileInSandbox, writeFileInSandbox helpers - Add OTEL config for build.parse_file and enriched compile_file spans - Exclude noop-ppx from biome linting (CommonJS required)
Add tests for core build functionality: - Build from a package subdirectory - No stale artifacts on second build - Filter flag to compile only matching modules Add build error tests: - Parse error reporting with file location - Type error reporting - Errors when a dependency module is deleted - Circular dependency detection
Add module operation tests: - File rename with and without dependents - Duplicate module name detection - Interface file compilation and error cases Add namespace package tests: - Build with namespace flag - Namespace in compiler args - File rename in namespaced package Add dev-dependency tests: - Dev source compiles with dev dependencies - Non-dev source cannot use dev dependencies - Clean removes dev source artifacts
Add build config tests: - Experimental feature flags (valid, invalid key, invalid format) - After-build hook execution (success and failure) - Warning configuration in compiler args - Warn-error CLI override - Deprecated and unknown config field warnings Add module system tests: - CommonJS package with .bs.js suffix - CommonJS in compiler args - Suffix change triggers rebuild - Duplicate package-spec suffix error Add PPX integration tests using lightweight noop-ppx: - PPX build produces output - PPX flags in parser args - PPX flags not in compiler args Add JSX tests: - JSX v4 build with @rescript/react - JSX flags in parser args - JSX preserve flag
Add tests for scoped clean, node_modules dependency cleanup, and verifying no false compiler-update message after clean+rebuild.
Add format tests: - Stdin formatting for .res and .resi - Single file and all-files formatting - Subdirectory-scoped formatting - Check mode (pass and fail cases) Add compiler-args tests: - CWD invariance (same output from root and subdirectory) - Warning flags in both parser and compiler args
Verify that a concurrent build is prevented while watch mode holds the lock file.
Add watch mode tests: - New file creation triggers compilation - Warning persistence across incremental builds - Config change triggers full rebuild - Changes outside source dirs are ignored - Missing source folder does not crash watcher - Invalid config change recovery (watcher keeps running) - File rename removes old artifacts and compiles new file - File deletion removes artifacts
Tracing spans are thread-local, so compile_file spans created inside Rayon's par_iter had no parent connection to the compile_wave span on the main thread. Pass the wave span explicitly via `parent: &wave_span` to establish the correct parent-child relationship.
When a file is saved in the LSP, only compile the saved file and its transitive dependencies instead of every module in the project. After the initial LSP build (TypecheckOnly), all modules sit at CompilationStage::TypeChecked. A TypecheckAndEmit build targets Built, so every module would enter the compile universe. In a large project this means the first save compiles the entire codebase to JS. Fix this by computing the downward dependency closure of the saved file and temporarily promoting modules outside that closure to Built. After the incremental build, promoted modules are restored to TypeChecked. Modules already at Built from a previous save are left untouched. Also change mark_file_parse_dirty to return Option<String> (the module name) so did_save can identify the entry point for the closure walk.
package get build to js.
Add single-file typecheck on unsaved edits (didChange). The unsaved buffer content is written to a temp file in the build directory and passed to bsc directly with TypecheckOnly. Diagnostics are remapped back to the original source path. Refactor didSave into two phases: - compile_dependencies (TypecheckAndEmit): compile the saved file and its transitive imports to produce JS output. - typecheck_dependents (TypecheckOnly): re-typecheck modules that transitively import the saved file to surface errors from API changes, without emitting JS. This means saving Library.res immediately shows type errors in App.res without needing to save App.res first. Other changes: - Extract find_module_for_file helper on BuildCommandState - Add get_dependent_closure (reverse dependency traversal) - Use #[instrument] consistently for OTEL spans in the lsp/ folder - Register new OTEL spans in test-context.mjs
Add an internal `-bs-read-stdin` flag to bsc that reads source from stdin instead of from the file argument. The filename argument is still required for error locations, file kind classification, and output prefix derivation. Update the LSP didChange handler to pipe unsaved buffer content directly to bsc's stdin instead of writing temporary files to disk. This eliminates unnecessary filesystem I/O on every keystroke. Key changes: - compiler: add `Js_config.read_stdin` flag and `-bs-read-stdin` CLI option - compiler: add `Res_io.read_stdin` and `Res_driver.parse_*_from_stdin` - compiler: disable `binary_annotations` when reading from stdin (avoids Digest.file call on non-existent source file) - rewatch: replace temp file write/cleanup in did_change.rs with stdin piping
Add a new `completion-rewatch` subcommand to the analysis binary that receives all needed context (pathsForModule, opens, package config) via JSON on stdin, bypassing the expensive project discovery that the existing `completion` command performs. Analysis binary changes: - Add `CommandsRewatch.ml` with JSON parsing and package construction - Add `CompletionFrontEnd.completionWithParserFromSource` to parse from a source string instead of reading from disk - Add `Completions.getCompletionsFromSource` that takes source + package - Add `Cmt.loadFullCmtWithPackage` that uses a pre-built package record instead of calling `Packages.getPackage` Rust LSP changes: - Track open buffers in `Backend.open_buffers` (updated on didChange) - Enable completion_provider capability with trigger characters - Add `lsp/completion.rs` that builds the JSON blob with all module/ package context, spawns `rescript-editor-analysis.exe completion-rewatch`, and deserializes the LSP-conformant response - If no .cmt exists yet (completion before any didChange), run a typecheck first to produce it Test infrastructure: - Add `completeFor` helper to lsp-client.mjs - Add `lsp.completion` span to OTEL summary - Add completion integration test
Add stdin support to `bsc -format` via `-bs-read-stdin`, enabling formatting of unsaved buffer content without temp files. Compiler changes: - Add `Res_multi_printer.print_source` to format from a source string - Update `format_file` to read from stdin when `-bs-read-stdin` is set - Simplify `format_stdin` in rewatch to use `-bs-read-stdin` instead of temp files LSP changes: - Add `textDocument/didOpen` handler to store buffer content - Add `textDocument/formatting` handler that pipes buffer to bsc - Advertise `documentFormattingProvider` capability - Extract `uri_to_file_path` helper to reduce boilerplate
tests - Add open_close capability to TextDocumentSyncOptions - Add did_close handler to remove buffers from open_buffers - Add OTEL tracing spans for did_open and did_close - Add closeFile method to LSP test client - Register lsp.did_open and lsp.did_close in OTEL span summaries - Refactor all LSP tests to call openFile before editFile - Add runtime guard in editFile that throws if openFile wasn't called first - Update LSP.md to reflect didOpen/didClose are implemented
- Rename completionRewatch/hoverRewatch to completion/hover in CommandsRewatch.ml since the module name already implies rewatch - Extract toUri() and assertOpen() helpers in lsp-client.mjs - Add assertOpen guard to completeFor, hoverFor, and formatFor - Remove unused content parameter from openFile - Add openFile calls before hoverFor in hover tests
- Add `definition` subcommand to analysis binary (CommandsRewatch.ml) - Extract `withRewatchContext` helper in CommandsRewatch.ml to reduce boilerplate across completion, hover, and definition - Add definition.rs with handle/run/parse_definition_response - Wire up goto_definition handler in lsp.rs Refactor shared patterns into analysis.rs: - resolve_source: open buffer lookup with disk fallback and trace warning - resolve_module: find module, package, and source file for a file path - original_path: compute original source path from package + source file - ensure_cmt: check .cmt existence and run did_change if missing Add LSP test helpers (assertOpen guard, toUri, definitionFor) and definition tests covering module value and module jumps.
- Add `typeDefinition` subcommand to analysis binary (CommandsRewatch.ml) - Extract `withLocItem` helper to reduce nesting in definition and typeDefinition - Add type_definition.rs reusing definition::parse_definition_response - Wire up goto_type_definition handler in lsp.rs - Make parse_definition_response public for reuse - Add fixture type (Library.user) and test for type definition - Update completion snapshot for new fixture value
uri.path() returns "/C:/Users/..." on Windows which breaks the test path normalizer. Use uri_to_file_path + display() consistent with all other handlers.
Add references endpoint following the same architecture as definition: the Rust LSP builds a JSON context blob, shells out to rescript-editor-analysis.exe rewatch references, and parses the result. - Add references() to CommandsRewatch.ml using withLocItem and allReferencesForLocItem - Add references subcommand routing in analysis binary main.ml - Add references.rs with handle/run functions and tracing spans - Wire up references_provider and handler in lsp.rs - Replace manual parse_location JSON parsing with direct serde deserialization in both definition.rs and references.rs, since Location derives Deserialize in lsp-types - Add test helpers (ReferencesRequest, referencesFor) and test for finding all references to a value within a package
and watch tests The typeDefinition endpoint added `let appAdmin = Library.admin` to App.res, creating a cross-package dependency. When build-modules tests renamed Library.res, App.res would fail to compile, causing snapshot mismatches (extra dirty modules, compile errors). Fix by moving the type-definition test to work within Library.res itself (cursor on `admin` jumps to `type user` in the same file), removing the cross-package reference from App.res. Also use Files.canonicalizeUri in references() for consistency with definition/typeDefinition, and simplify hover.rs to use direct serde deserialization since Hover derives Deserialize.
Add document symbol endpoint for outline view, breadcrumbs, and Ctrl+Shift+O navigation. Unlike other endpoints, documentSymbol is purely syntactic - it only needs the file path, not source content, cursor position, or package context. - Refactor DocumentSymbol.command to return a string instead of printing directly, so both the old CLI and rewatch wrapper can use it - Add documentSymbol() to CommandsRewatch.ml and routing in main.ml - Add document_symbol.rs with handle/run functions and tracing spans - Wire up document_symbol_provider and handler in lsp.rs - Add test helpers and test verifying symbols for let bindings, types with children, and record fields
Add prepareRename and rename endpoints to the rewatch-based LSP server.
OCaml: Add prepareRename() and rename() to CommandsRewatch.ml with
LSP-conformant JSON output. prepareRename outputs {range, placeholder}
matching PrepareRenameResponse::RangeWithPlaceholder. rename wraps
results in {documentChanges: [...]} matching WorkspaceEdit, using
Files.canonicalizeUri for URIs. Add newName field to rewatch_context.
Rust: Add rename.rs with both handlers following the established
pattern. The rename handler injects newName into the context JSON blob.
Enable rename_provider with prepare_provider support in capabilities.
Add textDocument/prepareRename, textDocument/rename, and
completionItem/resolve endpoints to the rewatch-based LSP server.
prepareRename: outputs {range, placeholder} matching serde's
PrepareRenameResponse::RangeWithPlaceholder.
rename: outputs {documentChanges: [...]} matching WorkspaceEdit,
with file renames and text document edits built in a single fold.
Uses Files.canonicalizeUri for URIs.
completionResolve: lazily populates module documentation when a user
selects a FileModule completion item. Extracts modulePath from the
item's data field, loads the target module's cmt, and returns its
docstring.
OCaml: Add newName and modulePath fields to rewatch_context. Add
prepareRename(), rename(), and completionResolve() to
CommandsRewatch.ml with LSP-conformant JSON output.
Rust: Add rename.rs with both rename handlers. Add handle_resolve
to completion.rs. Enable rename_provider with prepare_provider and
resolve_provider in CompletionOptions.
Add module docstring to Library.res fixture for testing. Update
LSP.md status table.
Add signature help support to the rewatch LSP server, providing function parameter information when the cursor is inside a function call. Changes: - Add signatureHelp subcommand to the analysis binary (CommandsRewatch.ml, main.ml) - Refactor SignatureHelp.ml to accept ~text and optional ?package - Add signature_help.rs LSP handler in rewatch - Wire up textDocument/signatureHelp in lsp.rs Fix bsc -bs-read-stdin to produce .cmt files: - The -bs-read-stdin flag was disabling binary_annotations entirely because Cmt_format.save_cmt calls Digest.file on the source path, which has stale content when reading from stdin - Add Clflags.skip_source_digest flag to skip the digest computation instead of suppressing .cmt generation altogether - Nobody in this codebase reads cmt_source_digest, so skipping it is safe and enables LSP features that depend on up-to-date .cmt files Always publish diagnostics for the origin file after didChange/didSave: - Split publish_diagnostics into two functions: one for the initial build and publish_diagnostics_for_file for incremental edits - publish_diagnostics_for_file always sends a notification for the changed file, even when empty, so clients know the typecheck completed - Update did-save test to expect empty diagnostics entries for clean files
Add codeLens support that shows function type signatures above function definitions. This follows the same analysis-binary pattern as the other LSP endpoints (hover, documentSymbol, etc.). Analysis side: - Refactor Hint.codeLens into reusable parts (collectCodeLensLenses, codeLensForFull) and add codeLensFromSource that parses from a source string and uses loadFullCmtWithPackage instead of reading from disk - Add CommandsRewatch.codeLens and wire it in main.ml Rust side: - Add code_lens.rs handler with ensure_cmt, build_context, and analysis_binary tracing spans - Register code_lens_provider capability in initialize Tests: - Add codeLensFor to lsp-client.mjs - Add code-lens.test.mjs with telemetry snapshot - Add greet function to Library.res fixture - Update test-context.mjs with new span names and attrs
Add inlayHint support that shows inline type annotations for variable bindings within the editor's visible range. Follows the same analysis-binary pattern as codeLens and other endpoints. Analysis side: - Refactor Hint.inlay into reusable parts (collectInlayHints, inlayForFull) and add inlayFromSource that parses from a source string and uses loadFullCmtWithPackage instead of reading from disk - Add maxLength field to rewatch_context for controlling hint length - Add CommandsRewatch.inlayHint and wire it in main.ml Rust side: - Add inlay_hint.rs handler with ensure_cmt, build_context, and analysis_binary tracing spans - Pass visible line range via pos field and maxLength in JSON blob - Direct serde deserialization since OCaml output is already LSP-conformant - Uncomment inlay_hint_provider capability in initialize Tests: - Add inlayHintFor to lsp-client.mjs - Add inlay-hint.test.mjs with telemetry snapshot - Update test-context.mjs with new span names and attrs
Refactor SemanticTokens.ml to extract makeIterator from the monolithic command function, enabling commandFromSource to parse from source strings without reading from disk. Add semantic_tokens.rs handler following the established pattern: resolve source buffer, build context JSON, shell out to analysis binary, and deserialize the flat u32 delta-encoded token array via serde. Register semantic token legend with 8 token types matching the existing editor extension: Operator, Variable, Type, JsxTag, Namespace, EnumMember, Property, JsxLowercase.
Add the last standard analysis LSP endpoint to the rewatch LSP server. Code actions provide refactoring suggestions at a cursor position or selection range (e.g. "Add Documentation template", "Exhaustive switch", "Expand catch-all", "Add type annotation"). OCaml side: - Add findTypeOfExpressionAtLocFromSource in CompletionFrontEnd.ml - Extract extractTypeFromExprImpl to share logic between disk and source-based variants - Extract mkImplementationPrinters/mkInterfacePrinters to deduplicate parse functions - Refactor ExpandCatchAllForVariants and ExhaustiveSwitch to use xformImpl with ~extractType parameter, eliminating ~350 lines of duplicated code - Add extractCodeActionsFromSource using Cmt.loadFullCmtWithPackage - Extend rewatch_context with endPos field for range support - Add CommandsRewatch.codeAction and dispatch in main.ml Rust side: - Add rewatch/src/lsp/code_action.rs with tracing spans and endPos injection into the JSON context blob - Wire up code_action_provider capability and handler in lsp.rs Tests: - Add code_action span names and attrs to test-context.mjs - Add codeActionFor method to lsp-client.mjs - Add code-action.test.mjs verifying "Add Documentation template"
Introduce AnalysisContext struct that decouples subprocess execution from the ProjectMap lock lifetime. Previously, every analysis handler held the Mutex<ProjectMap> lock through the entire analysis binary spawn + wait, serializing all LSP requests. Now the lock is only held for fast HashMap lookups, .cmt generation, and JSON context building. The expensive subprocess execution runs after the lock is released. This also eliminates ~80% boilerplate duplication across 13 handler modules by centralizing module resolution, ensure_cmt, context JSON building, and binary spawning into AnalysisContext::new() and spawn(). Other improvements: - Fix document_symbol to use open_buffers instead of reading from disk, so document symbols reflect unsaved editor content - Fix rayon span nesting by passing explicit parent span to initial_build::run, preventing work-stealing from misparenting spans - Add waitForSpan to runLspTest for stable OTEL span collection - Fix runRewatchTest waitForSpan to work with non-LSP tests - Use Value::Null instead of empty string for inlay hint maxLength
On Windows, `canonicalize()` produces verbatim paths (`\\?\D:\...`) while `read_packages` strips the prefix via `StrippedVerbatimPath`. This caused `find_independent_projects` to not recognize already-covered packages, re-discovering them as independent projects — resulting in duplicate `lsp.initial_build` spans, duplicate watcher patterns, and `watcher_count=43` instead of `35`. Apply `StrippedVerbatimPath::to_stripped_verbatim_path` to the `canonicalize()` call in `find_independent_projects` to match the path format used by `read_packages`.
rescript
@rescript/darwin-arm64
@rescript/darwin-x64
@rescript/linux-arm64
@rescript/linux-x64
@rescript/runtime
@rescript/win32-x64
commit: |
When bsc reads source from stdin (used by the LSP for didOpen/didChange typechecking), error reporting would crash with "index out of bounds" because it tried to re-read the source from disk for code frame display. The disk file content doesn't match the stdin content, causing position mismatches. Add Location.stdin_source to hold the stdin content and use it as a fallback in Location.print and error_message_utils extract_text_at_loc before falling back to disk reads. Also add LSP integration tests covering: - Type error diagnostics from buffer content before initial build - didOpen now triggers a typecheck for immediate feedback - Post-build recheck of open buffers for correct diagnostics
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This branch explores embedding a full LSP server directly into the
rescriptbinary (rescript lsp), replacing the current architecture where a Node.js extension mediates between the editor and separate build/analysis processes.The core idea
Today, the ReScript editor experience involves three processes: a Node.js VS Code extension, the
rescriptbuild watcher, and therescript-editor-analysis.exebinary. They communicate through files on disk — the editor extension launches builds, waits for artifacts, then shells out to the analysis binary for each request.This branch collapses the build system and LSP server into a single Rust process using
tower-lsp. The build state lives in memory, and analysis requests shell out to the samerescript-editor-analysis.exebut with source code passed via stdin instead of being read from disk.No temp files — stdin everywhere
Both
bscand the analysis binary receive source code via stdin rather than through temporary files. FordidChange(unsaved edits),bsc -bs-read-stdinproduces diagnostics without writing anything to disk. For analysis requests (hover, completion, code actions, etc.), the analysis binary receives a JSON blob on stdin containing the source text, cursor position, and package metadata. The OCaml analysis code was refactored withFromSourcevariants that parse from a string rather than opening files — so everything works correctly on unsaved editor buffers.Separate build profile:
lib/lspThe LSP server writes its build artifacts to
lib/lsp/instead oflib/bs/. This means it doesn't conflict withrescript buildorrescript build -wrunning in a terminal — both can operate independently on the same project without stepping on each other's artifacts.Initial build: typecheck only
On
initialized, the server runs a full build but only goes as far as producing.cmt/.cmifiles (theTypecheckOnlyprofile). It deliberately skips JS emission. This gets the editor operational as fast as possible — type information for hover, completion, go-to-definition etc. is all available, without paying the cost of generating JavaScript for every module upfront.Smart incremental builds on save
When a file is saved, the server runs a two-phase incremental build:
Emit JS for the dependency closure — the server computes the transitive imports of the saved file and only emits JavaScript for that file and its dependencies. Modules outside this closure are skipped entirely. So saving a module produces JS for it and any imports that haven't been compiled yet — not the entire project.
Typecheck reverse dependencies — modules that transitively depend on the saved file are re-typechecked to surface errors caused by API changes (e.g. a removed export). This gives you project-wide diagnostics on save — if you rename a function, you immediately see errors in every file that uses it, even files you don't have open. No JS is emitted for these — they get their JS when they are themselves saved.
What's implemented
All standard analysis endpoints are wired up: completion (with resolve), hover, signature help, go to definition, type definition, references, rename (with prepare), document symbols, code lens, inlay hints, semantic tokens, code actions, and formatting.
Observability
Every LSP request and build operation is traced with OpenTelemetry spans, viewable in Jaeger. This makes it straightforward to profile request latency and understand what the server is doing.
Test infrastructure
Each endpoint has integration tests using
vscode-languageserver-protocolthat boot a real LSP server in a sandbox, send requests, and snapshot both the results and the OTEL trace structure.What's not here yet
workspace/didChangeWatchedFiles— handling external file changes (git checkout, etc.)createInterfaceandopenCompiledcustom commandsThis is an experiment to validate the architecture. If it proves useful, individual pieces can be split into focused PRs.