Add chess.chesstb: pure-Python prober for chesstb endgame tablebases

[noobpwnftw]

Add chess.chesstb: pure-Python prober for chesstb endgame tablebases

What this adds

A new module, chess.chesstb, that probes the chesstb endgame
tablebase format directly from chess.Board positions — in the same spirit as
the existing chess.syzygy and chess.gaviota modules.

chesstb ships five table types per material:

Table	Extension	Answer
WDL	.lzw	50-move-rule-aware win/draw/loss with cursed/blessed classes
DTZ	.lzdtz	distance-to-zeroing (plies to the next capture, promotion or pawn move)
DTC	.lzdtc	one pack giving distance-to-conversion at any halfmove clock: the pawn pushes a win still owes, and the plies to the next zeroing move on the line that owes them
DTM	.lzdtm	unbounded distance-to-mate
DTM50	.lzdtm50	one pack giving both the unbounded DTM and the exact 50MR DTM at any halfmove clock

Each pack embeds an unbounded table as its first row, which makes that table
redundant wherever the pack ships: DTM comes out of the DTM50 pack, DTZ out of
the DTC pack. Both are read, with the pack preferred, so material shipping only
.lzdtm still answers probe_dtm and material shipping only .lzdtz still
answers probe_dtz.

Files are looked up in the wdl/, dtz/, dtc/, dtm/ and dtm50/
subdirectories of each search directory, and in the directory itself, so a flat
dump of table files is also probeable.

API

import chess, chess.chesstb

with chess.chesstb.open_tablebase("/path/to/chesstb") as tb:
    board = chess.Board("8/8/8/5k2/8/8/1Q6/K7 w - - 0 1")
    tb.probe_wdl(board)        # 2  (+2 win .. -2 loss, like syzygy)
    tb.probe_dtz(board)        # 19 (signed distance-to-zeroing)
    tb.probe_dtm(board)        # 19 (signed distance-to-mate, ignoring 50MR)
    tb.probe_dtm50(board)      # 19 (rule-true signed distance at the board's clock)
    tb.probe_dtc(board)        # (0, 19): (pushes owed, signed plies) at that clock

Every distance is signed from the side to move, 0 where nothing is decisive —
a draw, or a clocked metric whose clock has taken the win. The clocked metrics
hold only win/draw/loss (cursed and blessed are unreachable under 50MR), so the
sign carries the class and no WDL comes back alongside a distance. The one value
a sign cannot express is a mate already on the board, a loss at zero plies: it
reads 0 like a draw, and board.is_checkmate() settles that without a table.

get_wdl / get_dtz / get_dtm are non-raising variants returning a default
(None) when no table is available. probe(board, rule50=0) returns the full
structured result — every metric at once, off a single walk, with every class
field explicit, zero plies included.

probe_dtz measures what syzygy's does, but the two are not interchangeable.
Syzygy bases its cursed band off 100 — n > 100 is a cursed win whose zeroing
move is n or n - 100 plies away — so a DTZ magnitude carries the WDL class
along with the count. A chesstb DTZ is a plain distance in every class, the
class being WDL's alone, so substituting one for the other silently changes
what the number means.

DTC, for readers who have not met it

DTZ prices every pawn push at 1, so it reads 1 almost everywhere a pawn can move
and cannot tell a useful push from a wasted one. DTC prices the pushes
separately from the waiting: the first half of the pair is how many of its own
pawn pushes the winning side still owes before a conversion (a capture or a
promotion), and the second is the plies to the next zeroing move on the line
that owes them. The key is pushes first, then plies — a line that converts
without touching a pawn beats one that spends a push, even when the push is
quicker.

Fewer pushes cost a longer wait, so the halfmove clock decides how many are
affordable, and where no budget fits the clock the answer is a 50-move draw:

board = chess.Board("8/8/8/k7/8/8/K4P2/8 w - - 0 1")
tb.probe_dtc(board, 0)    # (4, 23): four pushes, first zeroing 23 plies out
tb.probe_dtc(board, 80)   # (5, 19): tighter clock, so one push more, 4 plies sooner
tb.probe_dtc(board, 95)   # (0, 0):  that clock has taken the win

Packs exist for pawnful material only: with no push to budget, every zeroing
move is already a conversion, so the answer is DTZ's own number at nothing owed.

Why pure Python

chess.syzygy is pure Python; this follows suit. The module depends only on
python-chess and the standard library:

• LZMA (DTZ / DTC / DTM / DTM50 blocks) via stdlib lzma with FORMAT_RAW
  (the C++ side uses the LZMA SDK with props appended at each block tail).
• LZ4 (WDL blocks) via a small bundled pure-Python LZ4-block decoder
  (~40 lines) supporting the optional LZ4 dictionary. No new dependency --
  but python-lz4, if the environment happens to have it, is picked up and used
  instead: the same C library the tables were written with, and it releases the
  GIL, so WDL blocks then decode in parallel like the lzma ones. A test pins the
  two decoders together block for block.

The position index (symmetry canonicalization, king/pawn slice managers, the
binomial piece-group ranking, the radix-composed board index and the
index-permutation layout) and the probe orchestration (dropped-frame one-ply
minimax reconstruction for shrunk files, the en-passant overlay, and the layer
selection both packs do at a halfmove clock) are faithful re-implementations of
the C++ src/probe library. Square numbering already matches python-chess
exactly (a1=0 … h8=63), so boards are consumed directly.

Table kinds that share a layout share a reader here, as they do on the C++ side.
DTZ and DTM are byte-for-byte twins on disk — as src/probe/dtm_file.cpp says
of its own traits — so one reader carries both, with the magic and the value
decode as the whole of what separates them. DTC and DTM50 likewise share the
changepoint container (header, block decode, the prefix indexes) and differ only
in what a record means: DTM50 takes one layer per read and halves its value by
the parity its class implies, while DTC decodes a whole record, since its answer
is a pair and its ranks hold plies outright.

Design notes

• Mapped, not read. Tables are memory-mapped and decoded a block at a time,
  with one shared LRU holding decoded blocks across every open table against a
  single budget (open_tablebase(..., block_cache_bytes=...)).
• One transport seam. _TableFile._open_source is the only place the
  transport is decided, and the file classes are named as class attributes on
  Tablebase, so serving tables from something other than a mapping is a
  subclass per kind and a _find. Nothing above that seam asks for a span wider
  than one block.
• Thread-safe. Probes register as readers so close() can wait for them
  before unmapping; table opens are double-checked under a per-kind lock, and
  block decoding is guarded per (color, block) rather than per table.

Validation

Every value is validated bit-for-bit against the reference C++ prober
(tools/probe_fen) by enumerating positions and comparing WDL, DTZ, DTC, DTM
and DTM50 in lockstep:

• All 145 shipped ≤5-man materials, ~72k positions at halfmove clock 0 —
  0 mismatches.
• Layered DTM50 at halfmove clocks 1, 30, 40, 80, 98, 99 across all
  materials (exercising the CONST/SINGLE/DOUBLE/MULTI changepoint state machine,
  the draw-end hint, and recover_mate_at_hmc) — 0 mismatches.
• Dropped-frame reconstruction: both the symmetric-mirror path (e.g. KRKR)
  and the asymmetric one-ply-minimax derive (109 of 145 materials ship a dropped
  frame) — 0 mismatches.
• En-passant overlay and color-mirrored material (stronger side is
  Black) — verified against the oracle.
• The pure-Python LZ4 decoder is cross-checked block-for-block against the
  reference lz4 C library on every shipped WDL table -- the same check the
  test suite runs over the fixture tables wherever python-lz4 is installed.

The standalone DTM reader was validated the same way, over random positions and
short playouts across the seven fixture materials (which reach en-passant
squares, mates, stalemates and every capture/promotion sub-material), against
probe_fen pointed at the same directories — 0 mismatches in each shape the
table can take on disk:

Table shape	how produced	positions
frames intact	chesstb --builddtm	22,851
dropped frame	shrink	22,851
loss-only (wins derive at probe time)	transcribe --loss-only	3,453
loss-only and dropped, over shrunk wdl/+dtz/	both	14,187
all four kinds present (pack preferred)	—	8,640

DTC was validated the same way, comparing WDL, DTZ and the DTC pair field by
field — the pack answers DTZ too, so a wrong embedded row shows up as a DTZ
mismatch:

What	positions × clocks	mismatches
KPK, every position, clocks 0 / 40 / 90	994,056	0
KPK, KPKP, KNKP, KRKP, KNNKP, KQK, KRK, clock drawn from 0/20/50/80/99/100	2,800	0
the same, packs shrink-dropped (derive path)	1,400	0
the same, transcribe --loss-only packs (win derives)	1,400	0
en-passant positions with a live ep capture (KPKP)	7,500	0
by record shape on KPPK: CONST / SINGLE / DOUBLE	3,600	0
by record shape on KPPK: 269 MULTI records	807	0
double pushes an enemy pawn can answer en passant (KPKP)	127,098	0

KNNKP is in that list for the cursed band: it is the smallest pawnful material
whose zeroing distances run past 100 (longest win 164), so its pack is the one
that exercises the 2-byte rank tier and the row values a budget cannot settle.
MULTI records are ~0.1% of even a KPPK block, so they were reached by
classifying sampled positions by record state and comparing that bucket rather
than by sampling alone.

The last row is the case that separates a derive from a read: a stored cell
carries no en-passant rights, so a reduced frame rebuilding a position whose
double-push child hands the opponent an ep capture has to price that reply the
way generation does. It needs pawns of both colours to arise at all, which the
single-pawn materials above cannot produce.

Tests

ChesstbTestCase in test.py, against table sets for seven materials
(KBK … KRKR) committed under data/chesstb/, shrink-processed as they would
ship — the DTC pack for KPK included, so the white-to-move answers there run
through the one-ply derive rather than a read.

The logic is proven on the C++ side and by the sweeps above, so these tests
target what is specific to this port rather than re-proving the format: the
per-kind wiring and directory search, the shared DTZ/DTM reader and its value
decodes, rejection of malformed files (and release of the mapping a failed open
had taken), the changepoint decoder at a live clock, the DTC pair against the
clock that picks it (with the DTZ table it embeds, read with no dtz/ in the
path at all), the block-cache budget, the mmap lifecycle, the transport seam,
and probing concurrently from several threads. Two more pin the conventions
above: that a mate already on the board and a draw both read as a zero distance
while probe() still separates them, and that the bundled LZ4 decoder agrees
with python-lz4 on every block of every fixture table wherever that package is
installed.