Skip to content

recipe: vl-convert-python 2.0.0rc7 - #127

Open
ndonkoHenri wants to merge 10 commits into
flet-dev:mainfrom
ndonkoHenri:vl-convert-python
Open

ndonkoHenri wants to merge 10 commits into
flet-dev:mainfrom
ndonkoHenri:vl-convert-python

Conversation

@ndonkoHenri

@ndonkoHenri ndonkoHenri commented Oct 4, 2026 •

Copy link
Copy Markdown

Adds a recipe for vl-convert-python 2.0.0rc7, which converts Vega-Lite and Vega specs to SVG, PNG, JPEG and PDF and is what Altair's chart.save("chart.png") calls. Requested in flet#6907.

Upstream ships no Android or iOS wheels: its desktop wheels embed V8, which it cannot ship for either. Only the 2.0 series can run Deno on another engine (1.9 is deno_core 0.307, V8 only), so the recipe builds 2.0.0rc7, the newest release.

Recipe shape

vl-convert runs Vega inside Deno. V8 was rejected: rusty_v8 publishes no Android builds, its iOS builds are arm64-only and jitless, and vl-convert's host-made V8 snapshot cannot load on either. deno_core 0.411's quickjs feature runs Deno on QuickJS-ng through the deno_v8 facade, which makes the engine an ordinary C dependency. mobile.patch switches to it and fixes what running on a phone exposed:

  • No snapshot, so the JavaScript is embedded. Without a snapshot Deno reads its extension sources from build-machine paths at runtime, and the worker panicked on device with Failed to initialize a JsRuntime: No such file or directory. deno_core.patch and deno_runtime.patch embed them. A QuickJS snapshot made on the build host was rejected: v8x replays snapshot callbacks by op index, Deno registers some ops per OS, and CI builds Android on Linux and iOS on macOS.
  • Android TLS keys. Rust has no native TLS on Android, so every thread_local! costs a pthread key. One worker used ~80 of bionic's 128, and the next native module to load (rpds-py, under Altair) aborted with fatal runtime error: out of TLS keys. forge_tls.c is linked into the extension with hidden visibility and multiplexes all of its keys onto one real key, leaving the rest of the process untouched.
  • Android HTTPS roots. reqwest's platform verifier panics outside a JVM-initialised app, so the Android clients use the Mozilla roots from webpki-root-certs.
  • Build plumbing. pyo3 without abi3 (the abi3 cross build links libpython3.10), aws-lc-sys with its bindgen feature, and a cargo [env] CMake toolchain file for the bare cmake calls in v8x (WAMR) and aws-lc-sys.
  • Deno crates with Android/iOS cfg gaps (deno_fs, deno_io, deno_node, deno_os, deno_signals) get one patch each through the new crate_patches key; each preamble says why.

armeabi-v7a is excluded: v8x ships pre-generated V8 API bindings for 64-bit targets only, with compile-time size asserts.

Forge changes

  • crate_patches (new meta.yaml key). The broken code sits in transitive crates.io dependencies, not in the sdist, so patches: cannot reach it. forge downloads the exact version Cargo.lock pins, checks it against the lockfile checksum, applies the recipe's patches and wires the copy in through [patch.crates-io]. A crate that no longer matches the lock is refused rather than silently skipped. An existing [patch.crates-io] table in upstream's manifest is joined, a crate it already patches is refused, and the manifest is re-parsed after the edit.
  • BINDGEN_EXTRA_CLANG_ARGS_<target> on every Rust build: the clang target and sysroot, so bindgen finds the cross headers (the iOS simulator needs the -simulator triple). The sysroot is shell-quoted, since bindgen splits the variable that way.
  • compiler-rt builtins on Android Rust links. rustc links with -nodefaultlibs, which left __clear_cache undefined on arm64 only. A static archive contributes only the members something references, so this adds nothing to a link that already resolved.

Validation

  • CI 6/6 jobs green across 3.12 / 3.13 / 3.14 × android / iOS, run 37205829688 on 53380e7. Since then the recipe changed by one code comment, and the forge changes were checked by running patch_crates on the real sdist: cargo resolves all seven Deno crates to the patched copies. On device: 7 passed / EXIT 0 on the Android x86_64 emulator, 6 passed, 1 skipped / EXIT 0 on the iOS simulator; the skip is the TLS key test, which is Android-only.
  • 7/7 on an arm64 Android emulator by hand. That run found __clear_cache and the TLS key exhaustion, neither of which CI's x86_64 emulator can see.
  • The chart-gallery example runs on an arm64 Android emulator and the iOS Simulator, rendering bar, line and LOESS charts to PNG. The first chart takes ~10 s with an empty bytecode cache and ~3 s with it on the emulator (~4.4 s / ~0.6 s on the simulator), and later charts ~0.5 s.
  • No build paths in the shipped extension: grepping the arm64 Android and iOS .so for /Users/… or /home/… .js/.ts paths finds only vl-convert's two module specifiers.

Review follow-ups (fd6e690, 3901c2d): crate_patches no longer writes invalid TOML when upstream already has a [patch.crates-io] table, the bindgen sysroots are quoted, forge keeps running on its declared Python 3.8 floor, and the TLS shim's lifetime cap of 4,095 key creates is written down (nothing in this extension comes near it).

Changes

  • recipes/vl-convert-python/: meta.yaml, mobile.patch plus six crate patches, 7 on-device tests (SVG, PNG scale, asyncio, text without system fonts, bytecode cache location, TLS key budget, Altair chart.save), README.md, and the example.
  • src/forge/: crate_patches (build + schema), the bindgen args and the compiler-rt builtins.
  • .claude/skills/: failure-catalogue entries for each failure above, how to author crate_patches, and a note that a green CI says nothing about arm64.

Consumer notes

A Flet app can turn a Vega-Lite spec or an Altair chart into a PNG or SVG on the device, offline, and show it with ft.Image. The first conversion in a process takes seconds while the JavaScript runtime starts, so point the bytecode cache at app storage and warm it up early. The install bound, the target_arch line, the storage wiring and the font caveats are in the recipe README.

…nd sysroot

`crate_patches` in meta.yaml patches a Rust package's dependencies, which the
sdist-level `patches` cannot reach: cargo fetches them at build time. forge takes
the version Cargo.lock resolves, checks the download against the lockfile
checksum, unpacks it into `forge-crates/`, applies the recipe's patches and adds
a `[patch.crates-io]` entry to the root Cargo.toml.

Rust builds also get `BINDGEN_EXTRA_CLANG_ARGS_<target>`. bindgen runs libclang
with no sysroot, so -sys crates that generate bindings at build time could not
find <stdlib.h> on Android, and its spelling of aarch64-apple-ios-sim is a
triple clang rejects. Scoped to the target, so host build scripts are untouched.
Vega-Lite and Vega to SVG/PNG/PDF on device (flet#6907). vl-convert runs the
Vega JavaScript inside Deno; this recipe switches Deno's engine from V8 to
QuickJS-ng through deno_core's `quickjs` feature, since V8 has no Android build
and its build-time snapshot cannot cross configurations. The runtime boots
without a snapshot. Six Deno crates get small cfg patches for Android/iOS via
crate_patches. armeabi-v7a is excluded: v8x's pre-generated V8 bindings assert
64-bit layouts.
rustc links cdylibs with -nodefaultlibs, so libclang_rt.builtins never reaches
the link, and Rust's compiler_builtins lacks some symbols C code in -sys crates
calls. arm64 libffi's __clear_cache is one: the extension links, then dlopen
fails with "cannot locate symbol". x86_64 never references it, so CI's x86_64
emulator cannot see this. Only members for still-undefined symbols are pulled.
CI run 1 built every Android wheel; on device the JS worker died at startup.
Fixes, each found on an arm64 emulator:

- Deno records extension sources as build-machine paths and reads them only
  while snapshotting. With no snapshot the worker read them on the phone and
  panicked ("Failed to initialize a JsRuntime: No such file or directory").
  deno_core and deno_runtime patches embed them instead.
- Rust has no native TLS on Android, so each thread_local! costs a pthread key;
  a worker took ~80 of bionic's 128 and the next native module aborted ("out of
  TLS keys"). forge_tls.c multiplexes the extension's keys onto one.
- pyo3 without abi3: the abi3 cross build linked libpython3.10 on iOS.
- reqwest's platform verifier panics on Android without JNI setup; use the
  bundled Mozilla roots there.
- A worker that dies at startup now reports its panic message.

7/7 tests pass on an arm64 Android emulator (Python 3.14), including Altair's
chart.save. All three iOS slices build and link; the iOS on-device run is CI's.
…tches

Catalogue entries for __clear_cache (arm64-only, now fixed in forge), out of
TLS keys (std spends a pthread key per thread_local! on Android), deno_core
sources read from build paths, the Android rustls-platform-verifier panic,
bindgen sysroot/triple, aws-lc-sys bindings, pyo3 abi3 on iOS, crate cfg gaps
and a build script's bare cmake. new-mobile-recipe documents crate_patches and
the Deno shape; local-recipe-testing records that CI never tests arm64.
Two patched versions of one crate get keys like deno_core-0.411.0, which TOML
reads as a dotted key.
- forge_tls.c leaves values with no destructor in place at thread exit, as
  bionic does; clearing them hid Rust std's thread handle from its own cleanup
  and leaked one per exiting thread.
- The Google Fonts blocking client also gets the Android TLS roots.
- The TLS key test detects Android with sys.getandroidapilevel: sys.platform is
  "linux" there before Python 3.13, so CI's 3.12 leg skipped it.
- Test with altair==6.3.0, the version run on device.
- Example: hide the image until the first chart (an empty src draws Flet's
  error box) and disable the picker during a conversion.
- README: a >=2.0.0rc7 bound instead of a pin, a throwaway conversion instead
  of warm_up_workers (which does not load Vega), per-output font fallback, iOS
  timings and sizes, TLS keys as an Android-wide problem, absolute cache path.
…oots

From review of flet-dev#127:
- crate_patches recognised only a literal "[patch.crates-io]" line, so a
  commented or quoted header got a second table and an entry for an
  already-patched crate a duplicate key, both invalid TOML. It now finds the
  header by regex, refuses a crate upstream already patches, and re-parses
  the result.
- bindgen shell-splits BINDGEN_EXTRA_CLANG_ARGS, so sysroots are quoted.
  Paths without special characters come out unchanged.
- No str.removesuffix, and tarfile's filter= only where it exists: forge
  declares Python >=3.8.
forge_tls.c never reuses a deleted key ID, which caps creates at 4,095 per
process. Nothing in this extension churns keys, so the code stays; the limit
is now written down in the shim and in the failure catalogue.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant