Repository navigation
fix(lexer): close the group after an escaped backslash - #23
Merged
Merged
Conversation
The scanner treated any "{", "}", "\" or newline right after a backslash as escaped, so in "\}" the brace was taken as escaped too: the escaped backslash escaped the next character as well. The group never closed, and every token after it was off by one bracket. Word writes exactly this inside field instructions such as {\fldinst SYMBOL 183 \f "Symbol"}, and documents containing them lost everything that followed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lexer::scantreats any{,},\or newline that follows a backslash as escaped. An escaped backslash (\\) is itself a complete escape, though, so in\\}the brace was taken as escaped as well. The group never closed, and every token after it was one bracket off.Word writes exactly this inside field instructions, for example the bullets it stores as
so a reader skipping the field's instruction group never saw it end, and lost the rest of the document. I found it while running a few hundred real-world RTF files (shareware help sources from 1990s CD-ROMs) through Paperback, which uses this crate.
The fix tracks whether the previous backslash still escapes the current character, so the second backslash of
\\escapes nothing.scan_escaped_backslash_before_closing_bracketcovers it: before the change it givesPlainText("\\}after")and one closing bracket too few.