hvantk

Hail-based toolkit for multi-omics variant annotation and analysis.

hvantk turns public omics resources (variants, genes, proteins and expression) into datasets with provenance, and integrates them in analysis tools that help interpret genetic variants. It uses Hail for genome-scale tables and AnnData for expression matrices.

• Data-source plugins. 21 providers, among them ClinVar, gnomAD, GTEx, UniProt, dbNSFP and Expression Atlas. Each dataset builds with one command, hvantk reprocess <provider>:<dataset>.
• Provenance on every dataset. A build records its source fingerprint, schema, plugin version and parameters, and a derived dataset keeps its parents.
• Drift detection. hvantk drift compares a source's live fingerprint with the committed one. A scheduled job re-checks every source twice a month and opens pull requests for what changed.
• Analysis tools. Joint genotyping, ancestry inference, QTL cascades, enrichment and burden tests, PTM variant classification, pathogenicity-score evaluation and multi-omic gene re-ranking.

Installation

Requires Python 3.10–3.12, Java 11 (for Hail and Spark) and Poetry 2.x.

git clone https://github.com/bigbio/hvantk
cd hvantk
poetry install
eval "$(poetry env activate)"
hvantk utils check-install   # verify the install

The base install is lean. Plotting, machine-learning and a few provider-specific dependencies are opt-in extras: poetry install --extras "ancestry psroc", or poetry install --all-extras for everything. Installation → Optional features lists what each extra adds and which commands need it.

Quick start

# Every data source and its datasets
hvantk plugins list

# Download, parse and build the HGNC gene table, then check its source for drift
hvantk reprocess hgnc:lookup --raw-dir data/hgnc --output hgnc.ht

A second run into the same --raw-dir stops with FileExistsError because the raw file is already there. Add --skip-download to rebuild from it, or use a fresh --raw-dir.

The output is a Hail table with a hgnc.ht.provenance.json sidecar. The Quick start continues from here to gene sets and an enrichment test, and Building datasets covers every build option.

Workflow skills

Twelve skills let a coding agent run hvantk for you, on a laptop or on a Slurm cluster from the container image: joint genotyping and callset delivery, dataset builds and drift triage, the gene index, cohort burden, re-ranking and QTL cascades. Every command the agent runs is a deterministic hvantk command. Install them in Claude Code with /plugin install hvantk --marketplace bigbio/hvantk, or point any harness that reads the Agent Skills layout at skills/. The catalog lists them and says how to set up the site profile. They are not the data-source plugins under hvantk/skills/.

Toolkit

Tool	Description	Command	Docs
Dataset builds	Build any plugin dataset (download → parse → build → drift check)	hvantk reprocess <provider>:<dataset>	Building datasets
Downloads	Fetch raw files for the sources that allow it	hvantk download <source>	Data sources
HGC	Joint genotyping: gVCF combining, QC, format conversion	hvantk hgc	HGC
Ancestry	Ancestry inference (PCA + Random Forest)	hvantk ancestry-inference	Ancestry
Annotate	Build the gene spine, map sources onto it, compose a gene × feature matrix	hvantk annotate	Gene annotation matrix
Cohort	Validate external cohorts, run a gene burden test, attach them to the matrix	hvantk cohort	External cohorts
Re-rank	Re-rank genes by multi-omic credibility from a YAML config	hvantk rerank	Re-rank
EnrichEx	Gene-set enrichment: overlap tests and rare-variant burden	hvantk enrichex	EnrichEx
Gene sets	Build gene-set files from ClinGen, GenCC, COSMIC or your own panels	hvantk genesets	Gene sets
PS-ROC	Pathogenicity-score ROC evaluation against ClinVar labels	hvantk psroc	PS-ROC
PTM	Post-translational modification variant classification	hvantk ptm	PTM
QTL cascade	Molecular QTL integration (eQTL → pQTL cascade, ABF colocalisation)	hvantk qtlcascade	QTL cascade
Expression	Expression matrices: describe, summarise, marker genes	hvantk expression	Expression analysis

Plus the registry and operational commands: hvantk plugins and hvantk tools (inspect the plugin and tool registries), hvantk catalog (search the aggregated dataset catalog), hvantk drift (compare a plugin's live source fingerprint against the committed one), and hvantk utils (format conversion, BGZF validation, install diagnostics).

Architecture

hvantk is organized in four code layers (core/, algorithms/, skills/, tools/) plus a substrate-level data registry (resources/). Imports flow one way, enforced by hvantk/tests/test_dependency_directions.py: tools/ composes skills/ and algorithms/, and those two never import each other. They meet only through the dataset types in core/.

A data-source adapter in skills/ can break when its upstream changes without breaking any algorithm, and algorithms/ evolve without touching the adapters. core/ and resources/ are the stable substrate; neither imports upward.

Data model

Four dataset types live in hvantk/core/models/, each backed by a native engine:

Dataset type	Backends	On-disk format	Used for
AnnotationTable	hail / pandas	.ht/ or .parquet	variants, gene-disease pairs, eQTLs, PTM sites
ExpressionMatrix	anndata	.h5ad	bulk + single-cell expression, proteomics matrices
VariantMatrix	hail-mt	.mt/	multi-sample variant cohorts (genotypes × samples × multi-field entries)
GeneSet	(in-memory frozenset)	.geneset.json	curated gene collections (ClinGen, GenCC, COSMIC CGC, MSigDB, …)

Every dataset carries a Provenance record: where it came from (its dataset id, such as clinvar:variants, and the source fingerprint), its schema id, the builder commit, build parameters and timestamp, and a parents chain. The @algorithm decorator stamps input provenances onto output datasets, so the build graph is preserved end to end.

AnnotationTable exposes a portable query API (filter, select, join, with_columns, group_by().agg()) through the col(...) expression DSL, compiled to Hail or pandas at execution time. When an algorithm needs the raw native object, core_io.load_native(path) returns (native_obj, Provenance) at no cost.

Plugin contract

Each data source is a self-contained plugin under hvantk/skills/<plugin>/, declared by a plugin.yaml manifest that names its builder, drift probe and, where the source allows an automated fetch, its downloader. Sources behind a licence gate, or too large to mirror, document their acquisition instead. run_builder_for_spec computes the source fingerprint, calls the builder, validates the built dataset against the manifest's artifact_type and schema_id, and saves it with a .provenance.json sidecar. The loader discovers manifests on its own; there is no registry to edit.

The architecture guide has the build sequence, an annotated plugin.yaml, the two-pass loader and the project structure.

How to extend

Add	Where	Pattern
A new data source	hvantk/skills/<plugin>/	Write plugin.yaml + builder.py (returns one of the four dataset types) + drift_probe.py. The loader discovers it.
A new algorithm	hvantk/algorithms/<domain>/	Decorate with @algorithm(name=…, backends=[…], inputs={…}, outputs={…}). Operate on dataset inputs (or use load_native for Hail-heavy work).
A new CLI command	hvantk/tools/<domain>/	Add the click command, a .tool.yaml manifest and an entry in _LAZY_COMMANDS in hvantk/hvantk.py, so the command's imports load only when it runs (details).
A new on-disk format	hvantk/core/io/_formats.py + dispatch in __init__.py	Add save_<type>_<ext> / load_<type>_<ext>.

Documentation

Full docs site: https://bigbio.github.io/hvantk

• Installation and Quick start
• Building datasets and Data sources: every dataset and how to build it
• Running on HPC: the container image and Slurm jobs
• Examples: walk-throughs for each tool
• Architecture: design and extension points
• Changelog: what changed in each release

Citation

If you use hvantk in your research, please cite it as below. GitHub's "Cite this repository" button reads the same metadata from CITATION.cff.

@software{hvantk2024,
  title = {hvantk: Hail-based toolkit for multi-omics variant annotation and analysis},
  author = {Perez-Riverol, Yasset and Audain, Enrique},
  year = {2024},
  url = {https://github.com/bigbio/hvantk}
}

Contributing

See CONTRIBUTING.md for the development workflow, code style and tests (pytest -q runs the fast suite).

License

MIT License. See LICENSE.

Support

• GitHub Issues
• Documentation