Skip to content

fix(tests): repair mojibake'd city names in CorrectResults.json - #83

Merged
ceeK merged 1 commit into
mainfrom
chowell/fix-fixture-city-name-encoding
Aug 31, 2026
Merged

fix(tests): repair mojibake'd city names in CorrectResults.json#83
ceeK merged 1 commit into
mainfrom
chowell/fix-fixture-city-name-encoding

Conversation

@ceeK

@ceeK ceeK commented Aug 31, 2026

Copy link
Copy Markdown
Owner

Problem

Twenty of the 237 city names in SolarTests/CorrectResults.json had lost their non-ASCII characters:

Stored Should be
Reykjav??k Reykjavík
T??rshavn Tórshavn
Isl��m��b��d Islāmābād
Chi�?in��u Chișinău

Two lossy markers appear, sometimes in the same name: ? (0x3F, a codec's "can't encode this") and U+FFFD (0xEF 0xBF 0xBD, a codec's "can't decode this").

Diagnosis

The tell is the marker count — there is one marker per byte of the original UTF-8, not one per character:

Reykjavík   í   = 2 UTF-8 bytes  ->  2 markers
Islāmābād   ā×3 = 6 UTF-8 bytes  ->  6 markers
Chișinău    ș+ă = 4 UTF-8 bytes  ->  4 markers

A single failed encode would drop one marker per character (Reykjav?k). Two means the file's UTF-8 bytes were read as though they were a single-byte charset — which splits each two-byte sequence into two separate garbage characters — and then written back out to something that couldn't represent those either. A double-transcode. The mix of ? and U+FFFD suggests the two passes used different substitution policies.

Because both markers discard the original byte value, this is not reversible from the file alone: í, ó and é all collapse to ??. The names were restored by hand.

Verification

Each restoration is checked two ways before being applied:

  1. Byte-count checksum — the non-ASCII characters of the correction must encode to exactly as many UTF-8 bytes as there were markers in the damaged string.
  2. ASCII skeleton — the surviving ASCII characters must be unchanged, in order.

All 20 pass both. Saint John�??sSaint John’s is the only three-byte case (U+2019).

All 20 repairs
Asunci??n           ->  Asunción
Bogot?�             ->  Bogotá
Bras??lia           ->  Brasília
Bucure�?ti          ->  București
Chi�?in��u          ->  Chișinău
H?� N��?i           ->  Hà Nội
Hag?�t?�a           ->  Hagåtña
Isl��m��b��d        ->  Islāmābād
Lom??               ->  Lomé
Mat?�'utu           ->  Matā'utu
Noum??a             ->  Nouméa
Panam?�             ->  Panamá
Port-aux-Fran??ais  ->  Port-aux-Français
Reykjav??k          ->  Reykjavík
S?�o Tom??          ->  São Tomé
Saint John�??s      ->  Saint John’s
San Jos??           ->  San José
T??rshavn           ->  Tórshavn
Tehr��n             ->  Tehrān
Yaound??            ->  Yaoundé

Blast radius

Deliberately minimal — only the city field is touched. Every latitude, longitude, sunrise and sunset value is byte-identical; the fix edits the raw JSON text rather than re-serialising, so formatting is untouched too. The diff is 20 changed lines, all "city".

No assertion depended on the damaged spellings: the only name-keyed lookup is first(where: { $0.name == "London" }), and London was undamaged. What this fixes is:

  • Failure output. city.name is interpolated into the #expect messages at SolarTests.swift:44 and :60, so a Reykjavík regression previously reported as Reykjav??k: ... not close to ....
  • A trap for the next test author. The suite already looks cities up by name, and the natural place to add a high-latitude isDaytime case is Reykjavík or Tórshavn — both damaged. Writing that test meant pasting mojibake into a string literal, which an editor or linter would later "helpfully" normalise and silently break the lookup.

Testing

swift test — 14 tests, 0 failures, before and after. Repaired names confirmed to round-trip through Swift's JSON decoding into test output:

City(name: "Chișinău", coordinate: ...)
City(name: "Hà Nội", coordinate: ...)

The file is valid UTF-8 with no U+FFFD remaining.


🤖 Generated with Claude Code

Twenty of the 237 city names in the test fixture had lost their non-ASCII
characters to a double-transcode: the file's UTF-8 bytes were at some point
decoded as a single-byte charset and re-encoded, leaving one substitution
marker per *byte* of the original rather than per character.

    Reykjavík -> Reykjav??k       (í, 2 UTF-8 bytes -> 2 markers)
    Islāmābād -> Isl??m??b??d     (ā x3, 6 bytes -> 6 markers)

Both markers used are lossy — '?' (0x3F) and U+FFFD — so the original bytes
are unrecoverable from the file alone and the names were restored by hand.

Each restoration is checked two ways: the non-ASCII characters must encode to
exactly as many UTF-8 bytes as there were markers, and the surviving ASCII
skeleton must be unchanged. All 20 pass both.

No assertion depended on the damaged spellings — the only name-keyed lookup is
for "London", which was undamaged — but the names are interpolated into the
sunrise/sunset failure messages, so a regression in e.g. Reykjavík reported as
"Reykjav??k: ... not close to ...".

Only the `city` field is touched; every latitude, longitude, sunrise and sunset
value is byte-identical (20 changed lines, all `"city"`).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PYPBCzQiXBvadC3y6Ae6FH
@ceeK
ceeK marked this pull request as ready for review August 31, 2026 15:23
@ceeK
ceeK merged commit 4e951c4 into main Aug 31, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant