Conversation
Adds a small xUnit project (Tests/ColorCode.Core.UnitTests) covering the JSON language's Regex_String. Partial_json_does_not_catastrophically_backtrack feeds an array with an unterminated string value and asserts highlighting finishes within a time budget; against current main it does not (O(2^n) backtracking, issue CommunityToolkit#45), so this commit fails and the following fix commit makes it pass. Valid_json_still_highlights_keys_strings_and_escapes pins the tokenisation of valid JSON so the fix can be shown to change nothing for good input. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: SandRock <sandrock@sandrock.fr>
…olkit#45) The string pattern used Friedl's problem shape: the repeating group (?:\[^\r\n]|[^"\\]*)* has an alternative ([^"\\]*) that can match the empty string, so when a match ultimately fails (e.g. a string value in an array, matched by the key rule but with no ':' following) the engine backtracks through every way to partition the string body — O(2^n) states. A 25-character unterminated value already exceeds seconds; longer values hang indefinitely. Replace it with Friedl's unrolled loop, "[^"\\]*(?:\\.[^"\\]*)*", which removes the ambiguity: linear time, and it matches valid strings (including escaped quotes and backslashes) identically to before. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: SandRock <sandrock@sandrock.fr>
Verify the built-in language registry is populated, every language is findable by id, and all rule patterns compile as valid regular expressions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Feed malformed / partial / oversized input to every language and assert that highlighting completes within a bounded time (failsafe: a catastrophic rule fails fast rather than hanging the run). Generalizes the JSON backtracking regression (CommunityToolkit#45) across the whole language set. One case is intentionally red at this commit: 'koka' hangs on an unterminated string — the same catastrophic-backtracking class as CommunityToolkit#45 — and is fixed in a following commit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…kit#13) The normal string, verbatim string, and char-literal rules each used a nested quantifier (?:X+|...)* that backtracks exponentially on an unterminated literal, hanging the highlighter — the same class of bug as the JSON parser (CommunityToolkit#45). Rewritten with Friedl's unrolled loop: linear, and preserving both matching and the escape captures on valid input. Turns the cross-language robustness test (previous commit) green, and adds targeted KokaBacktrackingTests covering all three literal kinds plus a valid-string sanity check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verify tokens receive their scope CSS class, the container div carries the language class, source text is HTML-encoded, and a stylesheet is produced. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses #13 with an automated, cross-platform xUnit suite for the core library — the
first one that actually runs assertions (the existing
BasicTestsis a console harness,and the UWP/WinUI test projects are sample apps). Everything targets
net8.0and runsanywhere
dotnet testdoes.What's covered
by id, and every language's rule patterns compile as valid regexes (a malformed
pattern now fails a test instead of a consumer's runtime).
input and must finish within a bounded time. Generalizes the JSON catastrophic-backtracking
regression (Json.cs: Catastrophic regex backtracking in Regex_String causes extreme slowdown with JSON arrays #45) across the whole language set; the time bound makes a bad rule fail fast
instead of hanging CI.
the language class, source is HTML-encoded, and a stylesheet is produced.
Assertions are structural (class present, text encoded) rather than exact-markup, so the
suite isn't coupled to incidental formatting.
A bug the robustness suite caught: Koka ReDoS
Writing the robustness net immediately surfaced a second catastrophic-backtracking bug —
in
Koka.cs. Its normal-string, verbatim-string, and char-literal rules each used a nestedquantifier
(?:X+|…)*that goes exponential on an unterminated literal (same class as #45).The commits are ordered so it's easy to see:
Add language registry smoke testsAdd cross-language highlighting robustness tests— the Koka case is red here (bug demonstrated)Fix catastrophic backtracking in the Koka string rules— Friedl's unrolled loop; therobustness test goes green, plus targeted
KokaBacktrackingTests(three literal kindsAdd HtmlClassFormatter output testsThe Koka fix is behaviour-preserving on valid input and keeps the escape captures intact.
Happy to split it into its own PR if you'd rather keep this one tests-only — just say.
Note on base
This is stacked on #46 (the JSON fix), so until that merges this PR also shows its two
commits; I'll rebase once #46 lands.
286 tests, all green, ~1s.
Signed-off-by: SandRock sandrock@sandrock.fr
Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com