Skip to content

Decode character references above U+FFFF correctly - #251

Merged
Peter Gromov (donnerpeter) merged 1 commit into
JetBrains:masterfrom
sebxt:fix/supplementary-char-references
Oct 20, 2026
Merged

Peter Gromov (donnerpeter) merged 1 commit into
JetBrains:masterfrom
sebxt:fix/supplementary-char-references

Conversation

@sebxt

Copy link
Copy Markdown
Contributor

EntityConverter.replaceEntities converted the decoded code point with Int.toChar(), which truncates anything above U+FFFF. 😀 rendered as U+F600 and 𝔄 as 프; this affects every numeric reference outside the BMP and the 133 named entities above U+FFFF.

Changes:

  • Build the replacement with Compat.codePointToString, so supplementary code points become a proper surrogate pair.
  • Replace �, surrogate code points and values above U+10FFFF with U+FFFD, as the CommonMark spec requires.
  • Limit decimal references to 7 digits and hex references to 6 digits, per the spec, so � stays literal text.

Tests:

  • New EntityConverterTest in commonTest for decimal, hex and named references above U+FFFF, invalid code points and over long references.
  • Un-ignore CommonMark spec examples 26 and 28 and GFM spec example 322, which now pass.
  • data/html/entities.txt had captured the old output for � � (a truncated ੸ and a raw NUL byte); it now expects � �, matching the spec.

./gradlew jvmTest jsTest wasmJsTest passes locally. I could not run the native targets (no Xcode on this machine); the change is in common code only.

Fixes #250

Signed-off-by: Sebastian Turner <sebastiant3484@gmail.com>
@donnerpeter
Peter Gromov (donnerpeter) merged commit a1ff2e7 into JetBrains:master Oct 20, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Character references above U+FFFF are decoded to the wrong character

2 participants