LoBiFlow

LoBiFlow is a history-conditioned flow-matching model for generating level-2 limit order books. The package includes the model, chronological dataset builders, raw-data preparation tools, training and benchmark entry points, and tests.

Installation

LoBiFlow requires Python 3.10 or newer. From a source checkout:

python -m pip install -e .

Install only the optional dependencies needed for your workflow:

python -m pip install -e ".[test]"          # pytest and test data support
python -m pip install -e ".[data]"          # pandas and Parquet support
python -m pip install -e ".[databento]"     # Databento preparation
python -m pip install -e ".[download]"      # provider download clients
python -m pip install -e ".[visualization]" # plots and image exports
python -m pip install -e ".[dev]"           # tests, linting, and builds

Editable installation makes lobiflow importable directly; no PYTHONPATH configuration is needed.

Package layout

code/lobiflow/
  data/       bundled synthetic calibration metadata
  datasets/   L2 representation, loaders, splits, and preparation tools
  models/     LoBiFlow, baseline models, conditioning, and configuration
  trainers/   training, evaluation, and maintained benchmark entry points
  utils/      metrics, visualization, and catalog export
  tests/      unit, training, and evaluation tests
results/      generated outputs plus explicitly tracked historical artifacts

Data locations and prepared format

External prepared datasets are read from LOBIFLOW_DATA_DIR, which defaults to data relative to the working directory. The named dataset defaults are:

$LOBIFLOW_DATA_DIR/cryptos/cryptos_binance_spot_monthly_1s_l10.npz
$LOBIFLOW_DATA_DIR/optiver/optiver_train_8stocks_l2.npz
$LOBIFLOW_DATA_DIR/databento/es_mbp_10.npz

Set LOBIFLOW_DATA_DIR before starting Python to use another data root, or pass --data-path to the training command. Synthetic generation uses calibration metadata bundled with the package and does not require an external NPZ.

The NPZ loader accepts either of these finite, non-empty schemas:

• ask_p, ask_v, bid_p, and bid_v, each with shape [T, L]; or
• params_raw with shape [T, 4L] together with mids of shape [T].

An archive may contain both representations when their timeline lengths agree. For concatenated sessions, segment_ends is an optional strictly increasing integer array whose final value is T; windows never cross those boundaries. segment_groups may assign one value to each segment when several adjacent segments belong to the same trading day or time group. Each group must be contiguous, and chronological splits are placed only between groups. When params_raw is supplied, its delta_mid column must agree with mids and reset to zero at every segment boundary. Timestamp and provider metadata arrays may also be stored and are ignored by the core loader. Chronological train/validation/test splits fit normalization statistics on the training portion only.

Preparing datasets

The generic converter accepts JSON Lines snapshots (plain, gzip, or a zip containing one file) and wide CSV files. JSON records may use a/b or asks/bids price-size pairs. CSV columns are inferred from names such as ask_price_1, ask_qty_1, bid_price_1, and bid_qty_1.

python -m lobiflow.datasets.lob_prepare_dataset \
  --dataset jsonl_l2 \
  --input path/to/books.jsonl.gz \
  --output data/custom/books.npz \
  --levels 10 \
  --history-len 256

python -m lobiflow.datasets.lob_prepare_dataset \
  --dataset csv_l2 \
  --input path/to/books.csv \
  --output data/custom/books.npz \
  --levels 10 \
  --history-len 256 \
  --time-col timestamp

Provider-specific converters create segment-aware archives:

# Optiver Kaggle Parquet data; requires the data extra.
python -m lobiflow.datasets.prepare_optiver \
  --input-root path/to/optiver \
  --output data/optiver/optiver_train_8stocks_l2.npz \
  --history-len 128 \
  --max-stocks 8

# Tardis monthly public samples; requires the data and download extras.
# Keep --max-files small while checking a new configuration.
python -m lobiflow.datasets.prepare_cryptos \
  --symbols BTCUSDT \
  --symbol-start-dates BTCUSDT:2024-01-01 \
  --end-date 2024-03-01 \
  --max-files 3 \
  --cache-root data/cryptos/tardis_cache \
  --output data/cryptos/cryptos_binance_spot_monthly_1s_l10.npz

# Databento MBP-10; requires the databento extra.
python -m lobiflow.datasets.prepare_databento \
  --start 2026-02-10 \
  --end 2026-03-10 \
  --cache-root data/databento/databento_cache \
  --output data/databento/es_mbp_10.npz

prepare_databento reads authentication only from DATABENTO_API_KEY; set it through your shell, secret manager, or CI environment rather than passing a key on the command line. A complete local cache can be processed without network authentication.

Data provenance and rights

This code repository does not grant rights to Tardis, Optiver/Kaggle, Databento, LOBSTER, or any other third-party data. Obtain data through an authorized source and comply with each provider's access, use, attribution, and redistribution terms. Do not publish raw or derived archives unless the applicable terms permit it.

Training and evaluation

The maintained single-run entry point is lobiflow.trainers.train:

python -m lobiflow.trainers.train \
  --dataset synthetic \
  --steps 12000 \
  --out-dir results/main

python -m lobiflow.trainers.train \
  --dataset npz_l2 \
  --data-path data/custom/books.npz \
  --levels 10 \
  --out-dir results/main

A normal run trains LoBiFlow and evaluates the validation and test splits. Additional work is opt-in:

• --run-speed-quality evaluates an NFE sweep;
• --run-ablations trains the selected ablation set;
• --run-rollout evaluates multiple rollout horizons; and
• --export-qualitative exports one qualitative trajectory.

For example:

python -m lobiflow.trainers.train \
  --dataset cryptos \
  --preset quality \
  --run-rollout \
  --rollout-horizons 60,300,900 \
  --out-dir results/main

Use python -m lobiflow.trainers.train --help for architecture, optimization, sampling, and evaluation overrides. Training and evaluation are compute-heavy; choose a suitable device and budget. Each run requires a new or empty output directory, and dataset/evaluation capacity is checked before training starts.

Benchmarks

The maintained benchmark modules are:

# LoBiFlow quality, speed, and architecture sections.
python -m lobiflow.trainers.benchmark_lobiflow \
  --datasets synthetic,optiver,cryptos,es_mbp_10 \
  --sections quality,speed,architecture \
  --out-root results/benchmark_lobiflow

# TRADES, CGAN, TimeCausalVAE, TimeGAN, and KoVAE baselines.
python -m lobiflow.trainers.benchmark_baselines \
  --datasets synthetic,optiver,cryptos,es_mbp_10 \
  --out-root results/benchmark_baselines

# Structured regularization training curves.
python -m lobiflow.trainers.benchmark_regularization \
  --dataset cryptos \
  --variants baseline_fm,local_causal_ot,current_matching \
  --out-root results/benchmark_regularization

These commands run training and evaluation. Narrow --datasets, --sections, --seeds, --checkpoints, or related budgets when appropriate. Benchmark commands default to the bundled synthetic data when no dataset is specified. Each benchmark otherwise requires a new or empty output root. Resume for benchmark_lobiflow is opt-in through --resume and validates cached run metadata, data identity, configuration, and runtime details before reuse.

The main comparison reports score_main plus TSTR macro-F1, discriminator AUC gap, unconditional W1, and conditional W1. Diagnostics include distribution, temporal, microstructure, and validity measures. Rollout timing is named efficiency_ms_per_rollout because it measures a complete generated trajectory, not one autoregressive step.

Metric catalogs

The catalog exporter has working checkout defaults:

python -m lobiflow.utils.export_model_metric_catalogs

With no arguments it reads the tracked historical LoBiFlow summary and baseline catalog, then refreshes results/model_metric_catalogs. To combine newly generated benchmark summaries without touching that archive, use explicit paths:

python -m lobiflow.utils.export_model_metric_catalogs \
  --lobiflow-summary results/benchmark_lobiflow/overall_summary.json \
  --baseline-summaries results/benchmark_baselines/overall_summary.json \
  --out-dir results/current_metric_catalogs

Tests

The default suite excludes tests marked as training or evaluation:

python -m pytest

Run the compute-bearing groups explicitly when intended:

python -m pytest -m training
python -m pytest -m evaluation
python -m pytest -m "training or evaluation"

Result directories

Current commands write new, ignored-by-default working outputs to:

• results/main/
• results/benchmark_lobiflow/
• results/benchmark_baselines/
• results/benchmark_regularization/

The following tracked directories are historical reference artifacts. Their names are retained for stable citations; they are not active module names or the default destinations of the current training and benchmark commands:

• results/benchmark_lobiflow_paper_ready/
• results/model_metric_catalogs/
• results/regularization_ablation/
• results/additional_results_slots/
• results/additional_results_slots_abstract_aligned/
• results/optiver_resmlp_confirm/

Treat tracked artifacts as archived evidence. Write new studies to a fresh output directory unless you intentionally mean to regenerate a catalog.

License

LoBiFlow is released under the MIT License.