feat: use released tpcgen-cli 4.0.0 from PyPI by kevinjqliu · Pull Request #114 · microsoft/LakeBench

Kevin Liu (kevinjqliu) · 2026-10-20T02:53:32Z

Why this change is needed

tpcgen-cli 4.0.0 is released on PyPI. LakeBench currently vendors a pre-release binary built from a fork, plus a lot of logic to size, rename, and package its output.

How

Use the PyPI package and make the integration a thin wrapper around the CLI.

Dependency: tpch_datagen / tpcds_datagen install tpcgen-cli==4.0.0. Removes the vendored binaries, hatch build hook, wheel scripts, and native-wheels workflow.
Wrapper: datagen/_tpcgen.py builds the command and runs it. compression, target_row_group_size_mb, and num_threads are passed straight through.
Parts: the only thing LakeBench decides. One SF1000 size table per benchmark, targeting ~256 MiB files.
Output: tpcgen-cli's layout, <table>/<table>.<part>.<ext>.

Behavior changes:

File names: lineitem-00001.zstd.parquet → lineitem.1.parquet. Native files are no longer renamed to dbgen/dsdgen names. Loading is unaffected.
target_row_group_size_mb is now the uncompressed size (tpcgen-cli's --row-group-bytes).
compression_factor is removed.
TPC-DS native output now accepts num_threads.

Test

Unit tests pass (pytest tests/ --ignore=tests/integration), also on Python 3.8.
Generated TPC-H and TPC-DS with tpcgen-cli 4.0.0: Parquet at SF10 (files come out ~250 MiB) and native at SF1.
Loaded native SF0.1 data into DuckDB and ran queries.
DuckDB/Polars integration tests: 7/8 pass. The failing DuckDB TPC-DS queries (DeltaScan serialization not implemented) fail the same way with the legacy DuckDB generator.

Closes issue

N/A

Replace the vendored pre-release tpcgen-cli binaries with the tpcgen-cli 4.0.0 package from PyPI (tpch_datagen / tpcds_datagen extras), and reduce the integration to a thin wrapper around the CLI. - One _tpcgen module builds the tpcgen-cli command and runs it. Options (compression, row-group size, threads) are passed straight through. - Part counts come from one measured SF1000 size table per benchmark, targeting ~256 MiB files. The compression-factor, native-ratio, and TPC-DS row-count scaling tables are removed. - Output keeps tpcgen-cli's layout (<table>/<table>.<part>.<ext>); the renaming to LakeBench and dbgen/dsdgen file names is removed. - Remove native/tpcgen, the hatch build hook, wheel build scripts, and the native-wheels workflow. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Kevin Liu (kevinjqliu)

removed a lot of wrapper code 😄

Kevin Liu (kevinjqliu) · 2026-10-20T03:24:35Z

note, we just released https://pypi.org/project/tpcgen-cli/ today so msft's internal pypi mirror hasn't picked it up yet. I'll regenerated the file again once its picked up

Miles Cole (mwc360) · 2026-10-22T14:37:56Z

Tom Zeng (@tomz) - FYI this was anticipated work once tpcgen-cli was released.

Jorge Docampo (jdocampo) · 2026-10-29T16:12:30Z

+        tables_by_parts = defaultdict(list)
+        for table in self.table_list:
+            # tpcgen-cli skips files that already exist; clear any previous output.
+            shutil.rmtree(self.target_folder / table, ignore_errors=True)


Please let cleanup failures stop generation rather than ignoring them. If a file is read-only, locked, or cannot be removed from a mount, the CLI skips surviving output files by default and can return success with stale or mixed-scale data. A missing table directory is normal on the first run, so only delete it when it exists:

Suggested change

shutil.rmtree(self.target_folder / table, ignore_errors=True)

table_folder = self.target_folder / table

if table_folder.exists():

shutil.rmtree(table_folder)

This change passes the local regression tests for both TPC-H and TPC-DS, including first-time generation and propagation of a simulated permission error without invoking the CLI.

Jorge Docampo (jdocampo) · 2026-10-29T16:12:31Z

+            raise ValueError("output_format must be either 'parquet' or 'native'.")
+        if output_format == "native" and (target_row_group_size_mb is not None or compression is not None):
+            raise ValueError("target_row_group_size_mb and compression apply only to output_format='parquet'.")
+        table_list = list(table_list or self.SF1000_SIZE_GB)


Please distinguish an omitted selection from an explicitly empty one. [] is falsy, so this expression currently selects every table; run() then deletes and regenerates all their output directories. That is particularly risky when the list is computed dynamically. The previous implementation rejected empty selections.

Suggested change

table_list = list(table_list or self.SF1000_SIZE_GB)

table_list = list(self.SF1000_SIZE_GB if table_list is None else table_list)

if not table_list:

raise ValueError("table_list must contain at least one table.")

This keeps None as "all tables" while rejecting [] before touching existing output.

- Raise when removing previous table output fails, instead of letting tpcgen-cli skip surviving files and report success with stale data. - Reject table_list=[] instead of treating it as "all tables". - Add tpcgen-cli 4.0.0 to uv.lock now that the package mirror has it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The linear SF1000 estimate over-split TPC-DS tables that grow sublinearly: inventory got 164 parts at SF1000 (~15 MiB files) and 1,636 at SF10000. Estimate size as the measured SF10 size times row growth from SF10, which is linear except for the TPC-DS tables that follow the spec's row counts. Re-measure SF10 sizes with tpcgen-cli 4.0.0. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The fallback was temporary until tpcgen-cli was released on PyPI. With it gone, TPCDSDataGenerator is a plain TpcgenDataGenerator subclass like TPCHDataGenerator: no backend switch, no wrapper class, and no _tpc module. Regenerate uv.lock with uv for the removed tpcds_duckdb_datagen extra. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Look up sublinear TPC-DS row counts at the nearest defined scale factor instead of interpolating, with counts in millions. - Document generator parameters once, on the shared constructor. - Drop --no-progress; tpcgen-cli hides progress bars when stderr is captured. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Default --row-group-bytes per benchmark (TPC-H 320, TPC-DS 176 MiB) so the largest tables keep ~128 MiB ZSTD(1) row groups on disk, as before. - Size native output from measured native sizes, and step the file size from 128 MiB to 1 GiB with table size, as before. - Replace the stale TPC-DS profiler with scripts/measure_tpcgen_sizes.py, which produces SF10_SIZE_MIB for both benchmarks. - Bump to 3.0.0: backend= and compression_factor= are removed, output files use tpcgen-cli names, and target_row_group_size_mb is uncompressed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Kevin Liu (kevinjqliu) · 2026-10-29T17:29:02Z

I did another round of fixes and review. Addressed all the comments and improved the PR as much as I can:

cleanup errors now stop generation, and table_list=[] is rejected
regenerated uv.lock, the internal mirror has picked up tpcgen-cli 4.0.0 now
file and row-group sizes match main again, including native output
removed the DuckDB TPC-DS fallback, it was meant to be temporary until tpcgen-cli was released
bumped to 3.0.0 since there are breaking changes (see description)

I mainly opened this PR to let the project know 4.0 is released. I won't have more time to work on it, so handing it over to the maintainers. Feel free to push to the branch or pick what's useful into a new PR.

Couple of notes:

DuckDB TPC-DS q95/q23a/q23b fail with DeltaScan serialization not implemented. Same failure on main, not from this PR
if tpcgen-cli adds a target file size option, LakeBench can drop the size tables and just pass that through

Kevin Liu (kevinjqliu) force-pushed the kevinjqliu-use-tpcgen-cli-4-0 branch 2 times, most recently from 1c48bb8 to 83ab200 Compare October 20, 2026 03:07

Kevin Liu (kevinjqliu) force-pushed the kevinjqliu-use-tpcgen-cli-4-0 branch from 83ab200 to 7875d4c Compare October 20, 2026 03:14

Kevin Liu (kevinjqliu) commented Oct 20, 2026

View reviewed changes

Jorge Docampo (jdocampo) requested changes Oct 29, 2026

View reviewed changes

Kevin Liu (kevinjqliu) and others added 5 commits October 29, 2026 09:59

Miles Cole (mwc360) mentioned this pull request Oct 29, 2026

Restore per-table file and row-group sizing on top of tpcgen-cli 4.0.0 kevinjqliu/LakeBench#1

Closed

Navigation Menu

Repository navigation

feat: use released tpcgen-cli 4.0.0 from PyPI - #114

feat: use released tpcgen-cli 4.0.0 from PyPI#114
Kevin Liu (kevinjqliu) wants to merge 6 commits into
microsoft:mainfrom
kevinjqliu:kevinjqliu-use-tpcgen-cli-4-0

Kevin Liu (kevinjqliu) commented Oct 20, 2026 •

edited

Loading

Uh oh!

Kevin Liu (kevinjqliu) left a comment

Uh oh!

Kevin Liu (kevinjqliu) Oct 20, 2026

Uh oh!

Miles Cole (mwc360) commented Oct 22, 2026

Uh oh!

Jorge Docampo (jdocampo) Oct 29, 2026

Uh oh!

Jorge Docampo (jdocampo) Oct 29, 2026

Uh oh!

Kevin Liu (kevinjqliu) commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

-            shutil.rmtree(self.target_folder / table, ignore_errors=True)
+            table_folder = self.target_folder / table
+            if table_folder.exists():
+                shutil.rmtree(table_folder)

-        table_list = list(table_list or self.SF1000_SIZE_GB)
+        table_list = list(self.SF1000_SIZE_GB if table_list is None else table_list)
+        if not table_list:
+            raise ValueError("table_list must contain at least one table.")

Conversation

Kevin Liu (kevinjqliu) commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why this change is needed

How

Test

Closes issue

Uh oh!

Kevin Liu (kevinjqliu) left a comment

Choose a reason for hiding this comment

Uh oh!

Kevin Liu (kevinjqliu) Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

Miles Cole (mwc360) commented Oct 22, 2026

Uh oh!

Jorge Docampo (jdocampo) Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

Jorge Docampo (jdocampo) Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

Kevin Liu (kevinjqliu) commented Oct 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Kevin Liu (kevinjqliu) commented Oct 20, 2026 •

edited

Loading