RepDBmaker Pipeline

A Snakemake workflow for building taxonomically annotated protein sequence databases from public resources and custom genome collections.

Contents

• RepDBmaker Pipeline
  • Contents
  • Overview
  • Installation
    • Requirements
  • Docker
  • Quick start
  • Reproducibility
    • Pinned source snapshots
    • Reproducing a release from a pinned universe
  • Troubleshooting
  • Documentation
  • Citation
  • License

Overview

RepDBmaker assembles protein sequence databases from multiple sources, annotates them with taxonomy, and builds searchable indices for Diamond, MMseqs2 and BLAST.

It draws on:

• prokaryotes from GTDB
• eukaryotes from EukProt, P10K, and UniProt
• viruses from NCBI Virus
• your own genomes or transcriptomes, alongside the public sources

with optional taxonomic clustering and contamination filtering along the way.

Installation

Requirements

• Snakemake (v8.11.6)
• Conda or Miniconda
• Internet access for external downloads
• Sufficient disk space for genome and database files

Pick how the workflow schedules its jobs (local machine, SLURM, LSF, or a site-specific profile) in Choosing an executor.

If using --sdm conda, Snakemake will automatically create the following environments from workflow/envs/:

• workflow/envs/python.yaml: Python, pandas, matplotlib, polars
• workflow/envs/homology.yaml: diamond, mmseqs2, blast
• workflow/envs/utils.yaml: taxonkit, csvtk, ncbi-datasets-cli, newick_utils, seqkit, jq, and standard CLI tools (wget, tar, unzip, gzip)
• workflow/envs/R.yaml: R and visualization/taxonomy packages
• workflow/envs/krona.yaml: Krona (interactive charts)

Create all of them up front, without running the pipeline:

snakemake --sdm conda --conda-create-envs-only

These .yaml specs are intentionally loose (minimum bounds only where a feature requires it) so a fresh install resolves against current packages. To instead reproduce the exact package builds used for the published RepDB v1.0, a pin file (workflow/envs/<name>.linux-64.pin.txt) sits next to each .yaml and is picked up automatically by --sdm conda, no extra flag needed.

Docker

A Docker image is available at Docker Hub, built from the pinned envs so it reproduces the RepDB v1.0 toolchain exactly, without needing conda or Snakemake installed locally. From the repository root:

docker run --rm -t -v $(pwd):/app/data gmuttiirb/repdbmaker:v1.0 \
  snakemake --configfile config/default.yaml config/repdb.yaml --cores <N> \
  --directory /app/data --sdm conda --conda-prefix /conda-envs

Both flags are required: --sdm conda turns on each rule's conda: env, and --conda-prefix /conda-envs must be exactly this value to reuse the image's pre-built environments instead of rebuilding them from scratch. Add -n to preview the plan without running anything; -t just gets you Snakemake's usual colored output.

--configfile takes two files here, config/default.yaml explicitly: --directory changes the base Snakemake resolves relative paths against, which includes the Snakefile's own configfile: "config/default.yaml" directive. Passing only config/repdb.yaml after --directory /app/data makes Snakemake look for config/default.yaml inside /app/data (your mounted, otherwise-empty directory) instead of the image's own /app, silently failing to load it. Passing both explicitly sidesteps that.

For reproducible pulls, use the image's immutable digest instead of the mutable :v1.0 tag:

docker run --rm -t -v $(pwd):/app/data \
  gmuttiirb/repdbmaker@sha256:c3458c7c5dd8e1d1a7dcf7ea61d02b97f4807c309ade381c6a65a72c68fa628a \
  snakemake --configfile config/default.yaml config/repdb.yaml --cores <N> \
  --directory /app/data --sdm conda --conda-prefix /conda-envs

Quick start

The workflow is two phases that meet at the universe, the enriched table of every available proteome (id, source, 7 ranks, completeness). Both phases read the same config file; pick the phase with the target:

command	produces
snakemake sample	Pipeline 1 (curation): results/universe/universe.tsv + results/universe/repdb.ids + the taxonomy QC (no sequences fetched)
snakemake build	Pipeline 2 (construction), the databases: repdb and, if configured, its clustered sibling repdb_clustered (+ decontamination, stats, _meta.tsv)
snakemake	both (all)

build produces the universe first if needed, so it's self-contained; run sample on its own to stop at the universe and review it before building.

To just inspect the available proteomes first:

snakemake --configfile config/repdb.yaml -j 1 --until available_proteomes
# -> results/meta/available_proteomes.tsv, for picking a proteome subset

To build everything with the default config:

snakemake --configfile config/repdb.yaml -j 14

--configfile config/repdb.yaml is required even just for available_proteomes: the Snakefile reads config["dbs"]["type"] while building its rule graph, before it even knows which target you asked for, and config/repdb.yaml is what supplies dbs: on top of the Snakefile's own auto-loaded config/default.yaml.

For more, see Example commands: a fast smoke test on a tiny subset, building only a custom database, running on a scheduler, reproducing a release, and more.

Producing an actual versioned release (freezing the universe, staging assets, publishing to GitHub/Zenodo) is a longer process; see docs/releasing.md.

Reproducibility

RepDBmaker supports reproducibility at three levels; pick the one your use case needs.

Level	What is fixed	How
Parameter	thresholds, which DBs, which subsets	the config file
Composition	which proteomes and their taxonomy	a pinned universe (below)
Artifact	exact tool builds and, ideally, the exact sequences	conda pin files + Docker digest + a Zenodo deposit of the FASTA

Pinned source snapshots

External sources drift, so pin the snapshots under versions: in config/repdb.yaml:

• gtdb: a specific release (e.g. release226), never latest.
• unieuk: the exported UniEuk taxonomy version.
• EukProt: the EukProt version.
• taxdump: a dated NCBI taxdump archive (or latest).

For sources without stable versioned hosting (UniProt reference-proteome release, RefSeq virus catalog, P10K), the universe below is what actually freezes them, so record the retrieval date alongside your run too.

Reproducing a release from a pinned universe

Because several sources are unversioned, re-running the full selection later yields a different set of proteomes. Building from a frozen universe instead of the live sources fixes that: taxonomy harmonization is skipped entirely, and the selection re-runs deterministically on the pinned composition.

snakemake build --configfile resources/releases/v1/config.yaml --sdm conda -j 8

resources/releases/v1/config.yaml is the self-contained build config for that release, produced alongside its pinned universe.tsv when the release was made. See docs/releasing.md for how a release like this gets produced, and where to download or publish its assets.

That same command also pins the exact sequences: resources/releases/v1/config.yaml points at the RepDB v1.0 Zenodo deposit too, so the fasta, cluster table, decontamination report and taxdump are fetched and checksum-verified instead of reassembled. That's artifact-level reproduction, not just composition-level. See Getting RepDB v1.0 if that (or just downloading the data directly, no pipeline needed) is all you're after.

Troubleshooting

Sometimes things can go wrong while downloading a proteome: rule db_stats fails if any gzipped fasta is malformed, blocking the database fasta until it's resolved.

cut -f1 results/dbs/<db>/genome_table.tsv | xargs -I {} sh -c 'gzip -t "{}" || echo "Failed: {}"'

Delete the problematic files and re-run the pipeline. If the problem persists, the files may be broken at the source or the current downloading script may be failing on them; we recommend excluding them and finding the most suitable alternative.

Documentation

Everything beyond this README lives under docs/:

• Getting RepDB v1.0: download the published data directly, or fetch + build indices with the pipeline
• Example commands: a cookbook covering common tasks, from a fast smoke test to reproducing a release
• Choosing an executor: local, SLURM, LSF, site profiles
• Configuration: full config reference plus eukaryote downsampling
• Custom databases: adding your own database or proteomes
• Clustering: the additive cluster: block, how it works, repdb_clustered
• Decontamination: the cross-domain contamination filter
• Outputs and quality control: where everything lands, and the QC reports
• Utilities and benchmarking: helper scripts, the NR/PhyloDB comparison
• Reproducing the paper figures: the two-stage HPC → laptop notebook split, and the palette checks
• Building & releasing RepDB: the full release process

Citation

If you use RepDBmaker or RepDB, please cite:

Mutti G. and Gabaldón T. Automated reconstruction of reproducible protein databases with RepDBmaker. Protein Science (2026). DOI:

License

See the LICENSE file for details.