[codethinki]

split was the slowest part of loading the word list at runtime (about 1 ms for the 785 KB list on gcc). It only runs at runtime without #embed (msvc) and in the tests and benchmark.

split

• libstdc++ 16: one std::simd::vec<char, 64> compare per 64 chars, indices from the mask with countr_zero
• everything else: per 256 char block every index is written and the count only advances on a separator, instead of an if per char
• the gate is __cpp_lib_simd || __glibcxx_simd, libstdc++ 16 does not define __cpp_lib_simd yet and clang 23 cannot use its <simd>

word_ends

• split moves to lbs::dev, the public lbs::word_ends appends data.size() as an end sentinel
• every word is [previous end + 1, end), filter_and_hash no longer special cases the last word
• splits is renamed to wordEnds everywhere
• declarations of dev::split, word_ends and filter_and_hash before the implementations
• the benchmark measures word_ends instead of split

libstdc++ bug

partial_load drops the third char when exactly 4 chars of a register remain, __memcpy_chunks in bits/simd_details.h copies bytes 0, 1 and 3 in its __bytes >= 2 branch. The tail is copied into a zero filled block and loaded with unchecked_load instead. Still present on gcc trunk.

tests

word_ends against a plain loop for every size from 0 to 300, for only separators, for a word without separators and for empty input. This caught the bug above (sizes 4 and 228).

results (word list split, pinned core, median of 7)

	gcc 16	clang 23
before	1060 us	472 us
std::simd	109 us
compaction	248 us	209 us