fix(tests): repair mojibake'd city names in CorrectResults.json - #83
Merged
Conversation
Twenty of the 237 city names in the test fixture had lost their non-ASCII
characters to a double-transcode: the file's UTF-8 bytes were at some point
decoded as a single-byte charset and re-encoded, leaving one substitution
marker per *byte* of the original rather than per character.
Reykjavík -> Reykjav??k (í, 2 UTF-8 bytes -> 2 markers)
Islāmābād -> Isl??m??b??d (ā x3, 6 bytes -> 6 markers)
Both markers used are lossy — '?' (0x3F) and U+FFFD — so the original bytes
are unrecoverable from the file alone and the names were restored by hand.
Each restoration is checked two ways: the non-ASCII characters must encode to
exactly as many UTF-8 bytes as there were markers, and the surviving ASCII
skeleton must be unchanged. All 20 pass both.
No assertion depended on the damaged spellings — the only name-keyed lookup is
for "London", which was undamaged — but the names are interpolated into the
sunrise/sunset failure messages, so a regression in e.g. Reykjavík reported as
"Reykjav??k: ... not close to ...".
Only the `city` field is touched; every latitude, longitude, sunrise and sunset
value is byte-identical (20 changed lines, all `"city"`).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PYPBCzQiXBvadC3y6Ae6FH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Twenty of the 237 city names in
SolarTests/CorrectResults.jsonhad lost their non-ASCII characters:Reykjav??kT??rshavnIsl��m��b��dChi�?in��uTwo lossy markers appear, sometimes in the same name:
?(0x3F, a codec's "can't encode this") and U+FFFD�(0xEF 0xBF 0xBD, a codec's "can't decode this").Diagnosis
The tell is the marker count — there is one marker per byte of the original UTF-8, not one per character:
A single failed encode would drop one marker per character (
Reykjav?k). Two means the file's UTF-8 bytes were read as though they were a single-byte charset — which splits each two-byte sequence into two separate garbage characters — and then written back out to something that couldn't represent those either. A double-transcode. The mix of?and U+FFFD suggests the two passes used different substitution policies.Because both markers discard the original byte value, this is not reversible from the file alone:
í,óandéall collapse to??. The names were restored by hand.Verification
Each restoration is checked two ways before being applied:
All 20 pass both.
Saint John�??s→Saint John’sis the only three-byte case (U+2019).All 20 repairs
Blast radius
Deliberately minimal — only the
cityfield is touched. Every latitude, longitude, sunrise and sunset value is byte-identical; the fix edits the raw JSON text rather than re-serialising, so formatting is untouched too. The diff is 20 changed lines, all"city".No assertion depended on the damaged spellings: the only name-keyed lookup is
first(where: { $0.name == "London" }), and London was undamaged. What this fixes is:city.nameis interpolated into the#expectmessages atSolarTests.swift:44and:60, so a Reykjavík regression previously reported asReykjav??k: ... not close to ....isDaytimecase is Reykjavík or Tórshavn — both damaged. Writing that test meant pasting mojibake into a string literal, which an editor or linter would later "helpfully" normalise and silently break the lookup.Testing
swift test— 14 tests, 0 failures, before and after. Repaired names confirmed to round-trip through Swift's JSON decoding into test output:The file is valid UTF-8 with no U+FFFD remaining.
🤖 Generated with Claude Code