feat: hand-written fault-tolerant parser, LSP completion and signature help

[ascandone]

Why

The ANTLR parser throws away whole subtrees on typical in-progress edits, so the LSP has almost nothing to work with while the user is typing. Some examples with the old parser:

input	old AST
send [COIN $	SentValue, Source, Destination all nil
source = { @a @ }	wrong: destination parsed as an account inside the source (the @ lexer mode leaks), destination lost
source = @a allowing	the whole source is nil, even though @a is valid
send $m + ( …rest of program	( read as a parenthesized expression, source and destination lost

Expression completion needs both a meaningful partial AST and "what is expected at the cursor", which ANTLR's Go runtime can't provide cleanly (no maintained antlr4-c3 port).

What

Parser (internal/parser): a hand-written lexer + recursive descent parser that produces the same AST types, so the checker, interpreter and compiler are untouched.

• Missing tokens are reported without being consumed. Unexpected tokens are skipped until a token that an enclosing construct can handle (destination, ), }, send, …).
• Error messages are clearer, e.g. expected `destination`, found `destinaton` (and the misspelled clause is still parsed).
• ParseForCompletion(src, cursor) records what the parser expected at the cursor: keywords, expression slots with the expected type, account parts, var types, statements, and the enclosing function calls.
• About 5x faster than the ANTLR parser on a typical script (6.8µs vs 36.6µs, 154 vs 589 allocs).

LSP:

• textDocument/completion (triggered on $, @, :):
  • keywords valid at the cursor
  • variables filtered by the expected type (account in a source, portion in an allotment, function parameter types, …)
  • builtin functions filtered by return type (snippets when the client supports them)
  • accounts and assets already used in the document
  • @users:$id interpolation, var types, statement snippets
• textDocument/signatureHelp for builtin calls, including unclosed ones.

How it's verified

• Differential test against the old parser, which is kept as a test-only oracle in internal/parser/internal/legacy. The corpus is every .num fixture, every Numscript string literal in the repo's Go tests, and handwritten edge cases (precedence, [$asset -1] juxtaposition, nested comments, …). Every program is also mutated by deleting, duplicating and swapping words, for ~45k programs in total. Whenever ANTLR accepts a program, the new parser accepts it and builds an identical AST. Whenever ANTLR rejects a program, the new parser rejects it too.
• All existing AST snapshots for valid programs are unchanged. Only error-related snapshots changed.
• Robustness: running the analysis (check, hover, goto definition, completion, signature help) on every prefix of every fixture never panics. There's also a FuzzAnalysis fuzz test (1.6M executions locally, no crash).

Behaviour changes to review

• Parse error messages changed. Tests matching ANTLR messages were updated (mcp_impl, specs_format snapshot).
• A // comment on the last line no longer needs a trailing newline. That was a parse error before.
• Comment.Content no longer includes the trailing newline.
• Partial nodes are kept, so semantic diagnostics can now show up next to parse errors on incomplete programs. I added a nil guard in unifyNodeWith for that.

Follow-ups

• Remove the legacy package (and the root .g4 grammars?) once this has been released for a while. CONTRIBUTING.md explains that for now both must be updated when the syntax changes.
• Possible next LSP features on top of ParseForCompletion: completing metadata keys and feature flags in #![feature(...)].

https://claude.ai/code/session_013WheaNzvuvbKPqmRbyREjs

[NumaryBot]

🟠 [major] Avoid quadratic scanning of unterminated nested comments

An input containing many consecutive /* openers and no closing delimiter makes matchFrom recursively scan every remaining suffix. Although results are memoized, each invocation still walks its entire suffix after nested matches return -1 (lines 466–480), so total work is quadratic in the number of openers, with linear recursion depth. Such an incomplete document is a normal input to the new fault-tolerant parser; a sufficiently large pasted comment can monopolize the LSP process rather than promptly producing an unterminated-comment diagnostic.

Suggestion: Replace suffix-by-suffix recursive matching with a bounded/linear-time delimiter scan or precomputed matching information while preserving the documented ANTLR fallback semantics. Add a regression test or benchmark with a large sequence of unmatched comment openers to cover recovery complexity.