Repository navigation
recipe: vl-convert-python 2.0.0rc7 - #127
Open
ndonkoHenri wants to merge 10 commits into
Open
ndonkoHenri wants to merge 10 commits into
ndonkoHenri wants to merge 10 commits into
Conversation
…nd sysroot `crate_patches` in meta.yaml patches a Rust package's dependencies, which the sdist-level `patches` cannot reach: cargo fetches them at build time. forge takes the version Cargo.lock resolves, checks the download against the lockfile checksum, unpacks it into `forge-crates/`, applies the recipe's patches and adds a `[patch.crates-io]` entry to the root Cargo.toml. Rust builds also get `BINDGEN_EXTRA_CLANG_ARGS_<target>`. bindgen runs libclang with no sysroot, so -sys crates that generate bindings at build time could not find <stdlib.h> on Android, and its spelling of aarch64-apple-ios-sim is a triple clang rejects. Scoped to the target, so host build scripts are untouched.
Vega-Lite and Vega to SVG/PNG/PDF on device (flet#6907). vl-convert runs the Vega JavaScript inside Deno; this recipe switches Deno's engine from V8 to QuickJS-ng through deno_core's `quickjs` feature, since V8 has no Android build and its build-time snapshot cannot cross configurations. The runtime boots without a snapshot. Six Deno crates get small cfg patches for Android/iOS via crate_patches. armeabi-v7a is excluded: v8x's pre-generated V8 bindings assert 64-bit layouts.
rustc links cdylibs with -nodefaultlibs, so libclang_rt.builtins never reaches the link, and Rust's compiler_builtins lacks some symbols C code in -sys crates calls. arm64 libffi's __clear_cache is one: the extension links, then dlopen fails with "cannot locate symbol". x86_64 never references it, so CI's x86_64 emulator cannot see this. Only members for still-undefined symbols are pulled.
CI run 1 built every Android wheel; on device the JS worker died at startup.
Fixes, each found on an arm64 emulator:
- Deno records extension sources as build-machine paths and reads them only
while snapshotting. With no snapshot the worker read them on the phone and
panicked ("Failed to initialize a JsRuntime: No such file or directory").
deno_core and deno_runtime patches embed them instead.
- Rust has no native TLS on Android, so each thread_local! costs a pthread key;
a worker took ~80 of bionic's 128 and the next native module aborted ("out of
TLS keys"). forge_tls.c multiplexes the extension's keys onto one.
- pyo3 without abi3: the abi3 cross build linked libpython3.10 on iOS.
- reqwest's platform verifier panics on Android without JNI setup; use the
bundled Mozilla roots there.
- A worker that dies at startup now reports its panic message.
7/7 tests pass on an arm64 Android emulator (Python 3.14), including Altair's
chart.save. All three iOS slices build and link; the iOS on-device run is CI's.
…tches Catalogue entries for __clear_cache (arm64-only, now fixed in forge), out of TLS keys (std spends a pthread key per thread_local! on Android), deno_core sources read from build paths, the Android rustls-platform-verifier panic, bindgen sysroot/triple, aws-lc-sys bindings, pyo3 abi3 on iOS, crate cfg gaps and a build script's bare cmake. new-mobile-recipe documents crate_patches and the Deno shape; local-recipe-testing records that CI never tests arm64.
Two patched versions of one crate get keys like deno_core-0.411.0, which TOML reads as a dotted key.
- forge_tls.c leaves values with no destructor in place at thread exit, as bionic does; clearing them hid Rust std's thread handle from its own cleanup and leaked one per exiting thread. - The Google Fonts blocking client also gets the Android TLS roots. - The TLS key test detects Android with sys.getandroidapilevel: sys.platform is "linux" there before Python 3.13, so CI's 3.12 leg skipped it. - Test with altair==6.3.0, the version run on device. - Example: hide the image until the first chart (an empty src draws Flet's error box) and disable the picker during a conversion. - README: a >=2.0.0rc7 bound instead of a pin, a throwaway conversion instead of warm_up_workers (which does not load Vega), per-output font fallback, iOS timings and sizes, TLS keys as an Android-wide problem, absolute cache path.
…oots From review of flet-dev#127: - crate_patches recognised only a literal "[patch.crates-io]" line, so a commented or quoted header got a second table and an entry for an already-patched crate a duplicate key, both invalid TOML. It now finds the header by regex, refuses a crate upstream already patches, and re-parses the result. - bindgen shell-splits BINDGEN_EXTRA_CLANG_ARGS, so sysroots are quoted. Paths without special characters come out unchanged. - No str.removesuffix, and tarfile's filter= only where it exists: forge declares Python >=3.8.
forge_tls.c never reuses a deleted key ID, which caps creates at 4,095 per process. Nothing in this extension churns keys, so the code stays; the limit is now written down in the shim and in the failure catalogue.
ndonkoHenri
force-pushed
the
vl-convert-python
branch
from
October 4, 2026 19:21
fd83a8e to
3901c2d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a recipe for vl-convert-python 2.0.0rc7, which converts Vega-Lite and Vega specs to SVG, PNG, JPEG and PDF and is what Altair's
chart.save("chart.png")calls. Requested in flet#6907.Upstream ships no Android or iOS wheels: its desktop wheels embed V8, which it cannot ship for either. Only the 2.0 series can run Deno on another engine (1.9 is deno_core 0.307, V8 only), so the recipe builds 2.0.0rc7, the newest release.
Recipe shape
vl-convert runs Vega inside Deno. V8 was rejected: rusty_v8 publishes no Android builds, its iOS builds are arm64-only and jitless, and vl-convert's host-made V8 snapshot cannot load on either. deno_core 0.411's
quickjsfeature runs Deno on QuickJS-ng through thedeno_v8facade, which makes the engine an ordinary C dependency.mobile.patchswitches to it and fixes what running on a phone exposed:Failed to initialize a JsRuntime: No such file or directory.deno_core.patchanddeno_runtime.patchembed them. A QuickJS snapshot made on the build host was rejected: v8x replays snapshot callbacks by op index, Deno registers some ops per OS, and CI builds Android on Linux and iOS on macOS.thread_local!costs a pthread key. One worker used ~80 of bionic's 128, and the next native module to load (rpds-py, under Altair) aborted withfatal runtime error: out of TLS keys.forge_tls.cis linked into the extension with hidden visibility and multiplexes all of its keys onto one real key, leaving the rest of the process untouched.webpki-root-certs.abi3(the abi3 cross build linkslibpython3.10), aws-lc-sys with itsbindgenfeature, and a cargo[env]CMake toolchain file for the barecmakecalls in v8x (WAMR) and aws-lc-sys.deno_fs,deno_io,deno_node,deno_os,deno_signals) get one patch each through the newcrate_patcheskey; each preamble says why.armeabi-v7ais excluded: v8x ships pre-generated V8 API bindings for 64-bit targets only, with compile-time size asserts.Forge changes
crate_patches(new meta.yaml key). The broken code sits in transitive crates.io dependencies, not in the sdist, sopatches:cannot reach it. forge downloads the exact version Cargo.lock pins, checks it against the lockfile checksum, applies the recipe's patches and wires the copy in through[patch.crates-io]. A crate that no longer matches the lock is refused rather than silently skipped. An existing[patch.crates-io]table in upstream's manifest is joined, a crate it already patches is refused, and the manifest is re-parsed after the edit.BINDGEN_EXTRA_CLANG_ARGS_<target>on every Rust build: the clang target and sysroot, so bindgen finds the cross headers (the iOS simulator needs the-simulatortriple). The sysroot is shell-quoted, since bindgen splits the variable that way.-nodefaultlibs, which left__clear_cacheundefined on arm64 only. A static archive contributes only the members something references, so this adds nothing to a link that already resolved.Validation
53380e7. Since then the recipe changed by one code comment, and the forge changes were checked by runningpatch_crateson the real sdist: cargo resolves all seven Deno crates to the patched copies. On device: 7 passed / EXIT 0 on the Android x86_64 emulator, 6 passed, 1 skipped / EXIT 0 on the iOS simulator; the skip is the TLS key test, which is Android-only.__clear_cacheand the TLS key exhaustion, neither of which CI's x86_64 emulator can see.chart-galleryexample runs on an arm64 Android emulator and the iOS Simulator, rendering bar, line and LOESS charts to PNG. The first chart takes ~10 s with an empty bytecode cache and ~3 s with it on the emulator (~4.4 s / ~0.6 s on the simulator), and later charts ~0.5 s..sofor/Users/…or/home/….js/.tspaths finds only vl-convert's two module specifiers.Review follow-ups (
fd6e690,3901c2d):crate_patchesno longer writes invalid TOML when upstream already has a[patch.crates-io]table, the bindgen sysroots are quoted, forge keeps running on its declared Python 3.8 floor, and the TLS shim's lifetime cap of 4,095 key creates is written down (nothing in this extension comes near it).Changes
recipes/vl-convert-python/:meta.yaml,mobile.patchplus six crate patches, 7 on-device tests (SVG, PNG scale, asyncio, text without system fonts, bytecode cache location, TLS key budget, Altairchart.save),README.md, and the example.src/forge/:crate_patches(build + schema), the bindgen args and the compiler-rt builtins..claude/skills/: failure-catalogue entries for each failure above, how to authorcrate_patches, and a note that a green CI says nothing about arm64.Consumer notes
A Flet app can turn a Vega-Lite spec or an Altair chart into a PNG or SVG on the device, offline, and show it with
ft.Image. The first conversion in a process takes seconds while the JavaScript runtime starts, so point the bytecode cache at app storage and warm it up early. The install bound, thetarget_archline, the storage wiring and the font caveats are in the recipe README.