Introduction

DNA derived data are increasingly being used to document taxon occurrences. To ensure these data are useful to the broadest possible community, GBIF published a guide entitled "Publishing DNA-derived data through biodiversity data platforms." This guide is supported by the DNA derived data extension for Darwin Core, which incorporates MIxS terms into the Darwin Core standard.

This use case draws on both the guide and the extension to build a workflow for creating Darwin Core Archives from eDNA data.

Current output model options:

• Occurrence Core + DNA Derived Extension.
• Event Core + Occurrence Extension + DNA Derived Extension.

The latest version of edna2obis (v3) builds on the original workflow with:

• Script architecture (single command) instead of a notebook workflow
• Configuration-first setup (set parameters in files, not code)
• Support for both generic FAIRe and FAIRe NOAA formats (compatible with Ocean DNA Explorer)
• Improved taxonomic mapping performance and accuracy (with caching)
  • Choose WoRMS or GBIF APIs
  • Optionally skip species-level matching for selected assays
  • taxa_mapping_INFO_<API>.xlsx with transparent candidate tracking (selected_match is the chosen row)
• Run report generation via edna2obis_report.html
• Optional generation of eMoF.csv, project-level eml.xml, and meta.xml
• Taxonomy-only mode via taxamapper.py (same taxa_mapping_INFO_<API>.xlsx style; filter selected_match = True for the carried-forward mapping)

Citation

If you find edna2obis helpful for your work, please consider citing the software:

Bayden Willms, & Katherine Silliman. (2026). aomlomics/edna2obis: v3.2.0 (Version v3.2.0) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.23667395

Tutorial

Need help running edna2obis?

Example data abstract:

Seawater was collected on board the NOAA ship Ronald H. Brown as part of the fourth Gulf of Mexico Ecosystems and Carbon Cycle (GOMECC-4) cruise from September 13 to October 21, 2021. Sampling for GOMECC-4 occurred along 16 coastal-offshore transects across the entire Gulf of Mexico and an additional line at 27N latitude in the Atlantic Ocean. We also collected eDNA samples near Padre Island National Seashore (U.S. National Parks Service), a barrier island located off the coast of south Texas. Vertical CTD sampling was employed at each site to measure discrete chemical, physical, and biological properties. Water sampling for DNA filtration was conducted at 54 sites and three depths per site (surface, deep chlorophyll maximum, and near bottom) to capture horizontal and vertical gradients of bacterial, protistan, and metazoan diversity across the Gulf. The resulting ASVs, their assigned taxonomy, and the metadata associated with theircollection are the input data for the OBIS conversion scripts presented here.

Published data

• GBIF
• OBIS

Input Data Format

Quick Overview

• Metadata tables come from running FAIReSheets (projectMetadata, sampleMetadata, experimentRunMetadata; and the NOAA data format variation requires an analysisMetadata sheet per analysis run).
• Raw ASV taxonomy and abundance tables come from Tourmaline (or any amplicon sequence processing workflow) as long as they follow Tourmaline's output structure.

Supported Data Formats

edna2obis supports two FAIRe eDNA data formats (both formats can be generated by FAIReSheets):

• FAIRe generic: Standard FAIRe layout. Analysis settings live in projectMetadata. Sequence ID column is typically seq_id in raw tables.
• FAIRe NOAA: Adds analysisMetadata per analysis run; optimized for Ocean DNA Explorer compatibility.

Both share the same core table ideas; examples below show the NOAA flavor. Differences for the generic flavor are noted along the way.

Metadata: NOAA Omics FAIR eDNA-based metadata template

This FAIRe NOAA Google Sheet template was developed by NOAA Omics at AOML and is based on the FAIRe eDNA data standard. Run FAIReSheets to generate these templates in Google Sheets. Here is a filled-in example:

FAIRe_NOAA_noaa-aoml-gomecc4_SHARING

projectMetadata

Project wide (project_level) project metadata, and metadata unique to each assay

Note: In generic FAIRe format, analysis metadata is included in this sheet rather than in separate analysisMetadata files.

term_name	project_level	ssu16sv4v5-emp (1st assay)	ssu18sv9-emp (2nd assay)
recordedBy	Luke Thompson
recordedByID	https://orcid.org/0000-0002-3911-1280
project_contact	Luke Thompson
institution	NOAA/AOML
institutionID	https://www.aoml.noaa.gov/omics
project_name	eDNA from Gulf of Mexico Ecosystems and Carbon Cruise 2021 (GOMECC-4)
project_id	noaa-aoml-gomecc4
parent_project_id	noaa-aoml-gomecc
study_factor	water column spatial series
assay_type	metabarcoding
sterilise_method	After sampling, run ~1 L of 5% bleach through tubing lines, then rep...
checkls_ver	FAIRe_checklist_v1.0.xlsx
mod_date	2024-10-31
license	http://creativecommons.org/publicdomain/zero/1.0/legalcode
rightsHolder	US Government
accessRights	no rights reserved
assay_name		ssu16sv4v5-emp	ssu18sv9-emp
ampliconSize		411	260
code_repo	https://github.com/aomlomics/gomecc
biological_rep	3

sampleMetadata

Contextual data about the samples collected. Each row is a distinct sample (Event)

Note: This structure is largely the same between generic FAIRe and FAIRe NOAA formats.

samp_name	materialSampleID	geo_loc_name	eventDate	decimalLatitude	decimalLongitude	sampleSizeValue	sampleSizeUnit	env_broad_scale	env_local_scale	env_medium	samp_collect_device	samp_vol_we_dna_ext	samp_mat_process	size_frac
GOMECC4_27N_Sta1_Deep_A	GOMECC4_27N_Sta1_Deep	USA: Atlantic Ocean, east of Florida (27 N)	2021-09-14T11:00-04:00	26.997	-79.618	1920	mL	marine biome [ENVO:00000447]	marine mesopelagic zone [ENVO:00000213]	sea water [ENVO:00002149]	Niskin bottle on CTD rosette	1920 mL	Pumped through Sterivex filter (0.22-µm) using peristaltic pump	0.22 µm
GOMECC4_27N_Sta1_Deep_B	GOMECC4_PANAMACITY_Sta1_Deep	USA: Atlantic Ocean, east of Florida (27 N)	2021-09-20T23:13-04:00	26.997	-79.618	1920	mL	marine biome [ENVO:00000447]	marine mesopelagic zone [ENVO:00000213]	sea water [ENVO:00002149]	Niskin bottle on CTD rosette	1920 mL	Pumped through Sterivex filter (0.22-µm) using peristaltic pump	0.22 µm

experimentRunMetadata

Library preparation and sequencing details

Note: This structure is largely the same between generic FAIRe and FAIRe NOAA formats.

samp_name	assay_name	pcr_plate_id	lib_id	seq_run_id	mid_forward	mid_reverse	filename	filename2	input_read_count
GOMECC4_NegativeControl_1	ssu16sv4v5-emp	not applicable	GOMECC16S_Neg1	20220613_Amplicon_PE250	TAGCAGCT	CTGTGCCTA	GOMECC16S_Neg1_S499_L001_R1_001.fastq.gz	GOMECC16S_Neg1_S499_L001_R2_001.fastq.gz	29319
GOMECC4_NegativeControl_2	ssu16sv4v5-emp	not applicable	GOMECC16S_Neg2	20220613_Amplicon_PE250	TAGCAGCT	CTAGGACTA	GOMECC16S_Neg2_S500_L001_R1_001.fastq.gz	GOMECC16S_Neg2_S500_L001_R2_001.fastq.gz	30829

analysisMetadata

Bioinformatic analysis configuration metadata. There is one analysisMetadata sheet PER analysis. Append the analysis_run_name to the filename, ex: analysisMetadata_gomecc4_16s_p1-2_v2024.10_241122.tsv

Note: analysisMetadata sheet(s) is only present in FAIRe NOAA format. In generic FAIRe format, this information is included in the projectMetadata sheet.

Field	Value
project_id	noaa-aoml-gomecc4
assay_name	ssu16sv4v5-emp
analysis_run_name	gomecc4_16s_p1-2_v2024.10_241122
sop_bioinformatics	https://github.com/aomlomics/gomecc
trim_method	cutadapt
trim_param	qiime cutadapt trim-paired
demux_tool	qiime2-2021.2; bcl2fastq v2.20.0
merge_tool	qiime2-2021.2; DADA2 1.18
min_len_cutoff	200
otu_db	Silva SSU Ref NR 99 v138.1; 515f-926r region; 10.5281/zenodo.8392695
otu_seq_comp_appr	Tourmaline; qiime2-2021.2

Raw Data: ASV Taxonomies and Abundance Tables

You must have two raw data files associated with each analysis (analysisMetadata) in your submission. These files are generated by Tourmaline v2, AOML Omics' amplicon sequence processing workflow.

Note: Raw data structure is largely the same between formats, with one key difference: in generic FAIRe format, the sequence identifier column is named seq_id instead of featureid.

If your data was generated with Qiime2 or a previous version of Tourmaline, you can convert the table.qza, taxonomy.qza, and repseqs.qza outputs to the correct format using the create_asv_seq_taxa_obis.sh shell script.

Example:

# Run this with a qiime2 environment.
bash create_asv_seq_taxa_obis.sh -f \
../gomecc_v2_raw/table-16S-merge.qza -t ../gomecc_v2_raw/taxonomy-16S-merge.qza -r ../gomecc_v2_raw/repseqs-16S-merge.qza \
-o ../gomecc_v2_raw/gomecc-16S-asv.tsv

Your ASV raw data files should look like this:

ASV Taxonomy Features:

featureid	dna_sequence	taxonomy	verbatimIdentification	kingdom	phylum	other_ranks...	species	Confidence
1ce3b5c6d...	TACGA...	Bacteria;Proteobacteria;Alphaproteoba...	d__Bacteria;p__Proteoba...	Bacteria	Proteobacteria	...	Clade_Ia	0.88
4e38e8ced...	GCTACTAC...	Eukaryota;Obazoa;Opisthokonta;Metazoa...	Eukaryota;Obazoa;Opisthokonta;Metazoa...	Eukaryota	Obazoa	...	Clausocalanus furcatus	0.999

Note: In generic FAIRe format, the first column would be named seq_id instead of featureid.

Note: Taxonomy is complex, so edna2obis accepts any list of taxonomic ranks as long as they are between verbatimIdentification and Confidence. For example, 16S and 18S assay data can use different rank names and counts. The code assigns taxonomy based on what ranks each API returns.

verbatimIdentification strings may or may not include prepended rank prefixes with underscores. The code removes these during processing when present.

featureid is a hash of the DNA sequence and acts as a unique identifier.

Note: In generic FAIRe format, this column is named seq_id instead of featureid.

Some field values are truncated (...) for readability in the documentation. Include complete values in your input files.

ASV Taxonomy: what edna2obis uses

• This raw taxonomy table is generated by FAIReSheets in both generic FAIRe and NOAA-FAIRe formats
• You can fill in the raw taxonomy table Google Sheet, or if you run Tourmaline, you can just use the raw taxonomy table from that output (they are the same format).
• edna2obis sends the value in verbatimIdentification to the selected taxonomy API (after cleaning common prefixes and formatting).
• The API response is then used to fill most of the taxonomy fields in your final outputs (e.g., standard ranks and identifiers).

ASV Abundance Tables:

featureid	GOMECC4_BROWNSVILLE_Sta63_DCM_A	GOMECC4_BROWNSVILLE_Sta63_DCM_B	GOMECC4_CAMPECHE_Sta91_DCM_A
1ce3b5c6d...	0	32	2
4e38e8ced...	15	0	45

Each column name after featureid is a sample name, and must correspond with your sampleMetadata.

Note: In generic FAIRe format, the first column would be named seq_id instead of featureid.

If your abundance tables have decimal numbers, that is okay too.

Configuration and Workflow

Configure your run using the files below; the graphic shows how inputs flow through edna2obis to outputs. Current workflow output is Occurrence Core + DNA Derived Extension (with optional eMoF.csv, meta.xml, and eml.xml). Event Core support is planned for future versions.

edna2obis uses several configuration files to customize the conversion process. All configuration files are located in the root directory.

Main Configuration Files

config.yaml

The primary configuration file that controls the overall pipeline behavior:

• Data file paths: Specify paths to your Excel metadata file and raw data files
• Taxonomic mapping: Choose between "WoRMS" or "GBIF" APIs for taxonomic mapping
• Output settings: Configure output directory and file naming
• Custom field handling: Configure how to process specific metadata fields
• Processing parameters: Set parallel processing limits and caching options

data_mapper.yaml

Maps FAIRe metadata fields to Darwin Core terms:

• Field mappings: Maps FAIRe fields to Darwin Core standards
• Format-specific mappings: Different mappings for generic vs NOAA FAIRe formats

EML_config.yaml

Controls Ecological Metadata Language (EML) generation:

• Dataset metadata: Project descriptions, contact information, licensing
• Geographic coverage: Study area descriptions and bounding boxes
• Temporal coverage: Study period and sampling frequency
• Method descriptions: Detailed protocols and methodologies

eMoF Configuration

The Extended Measurement or Fact (eMoF) file can be customized to include additional measurements:

• Template file: raw-v3/eMoF_Fields_edna2obis.xlsx
• Custom measurements: Add your own measurement types and values
• Controlled vocabularies: Use OBIS-approved terms for measurement types and units
• Documentation: See OBIS eMoF formatting guide for detailed instructions

You can add custom fields to the eMoF template to include additional environmental measurements, sampling parameters, or any other data linked to your events. If there are fields in the eMoF template that are not in your data, don't worry, the code will account for that!

Example Configuration Files

config_FAIRe_noaa.yaml

Example configuration for FAIRe NOAA format data:

• Pre-configured for Ocean DNA Explorer compatibility
• Optimized settings for NOAA-specific metadata structure
• Includes NOAA-specific field mappings

config_FAIRe_generic.yaml

Example configuration for generic FAIRe format data:

• Standard FAIRe eDNA data format settings
• Generic field mappings and processing parameters
• Compatible with standard FAIRe workflows

🚀 Setup and Installation

Prerequisites

• Python 3.13 installed
• Either:
  • Conda or Anaconda, or
  • Python venv + pip
• Git installed
• At least 8GB RAM recommended
• Internet connection required (for API calls to WoRMS/GBIF)

Quick Start

1. Clone the Repository

git clone https://github.com/aomlomics/edna2obis.git
cd edna2obis

2. Create Python Environment

Choose one method:

Option A: Conda

# Create the environment from the environment.yml file
conda env create -f environment.yml

# Activate the environment
conda activate edna2obis

Option B: venv + pip

# Create and activate a virtual environment
python -m venv .venv
# Linux/macOS
source .venv/bin/activate
# Windows (PowerShell)
.venv\Scripts\Activate.ps1

# Install dependencies
pip install -r requirements.txt

3. Edit the Configuration Files

Edit the configuration files with your data file paths and other parameters:

Primary configuration:

• Edit config.yaml for main pipeline settings
• You can copy config_FAIRe_noaa.yaml or config_FAIRe_generic.yaml into config.yaml as a starting point based on your data format

Additional configuration (optional):

• Modify data_mapper.yaml for custom field mappings
• Update EML_config.yaml for metadata generation settings, or specify you don't want an EML file in config.yaml
• Customize raw-v3/eMoF_Fields_edna2obis.xlsx for additional measurements, or specify you don't want an eMoF file in config.yaml

Key settings to update in config.yaml:

• excel_file: Path to your FAIRe Excel file (data template)
• datafiles: Paths to your ASV taxonomy and occurrence files
• taxonomic_api_source: Choose "WoRMS" or "GBIF"
• output_dir: Where to save results (default: "processed-v3/")

4. Run the Pipeline

python main.py

The pipeline will:

• Load and clean your metadata (according to OBIS/GBIF)
• Align data to Darwin Core data standard
• Generate an Occurrence Core
• Perform taxonomic mapping via WoRMS or GBIF APIs
• Generate a DNA Derived Extension
• Optionally generate eMoF.csv
• Generate Darwin Core Archive metadata files (meta.xml and optional eml.xml)
• Create an HTML report with results from your run

Run taxonomic mapping only (taxamapper.py)

If you only want to map taxonomy for a list of identifiers (without running the full edna2obis workflow), use taxamapper.py.

• Input: a TSV/CSV with a single column named verbatimIdentification. An example is provided at raw-v3/taxamapper_example_input.tsv.
• Output: taxa_mapping_INFO_<API>.xlsx (same column layout as the full pipeline; sheet taxa_mapping_INFO only). Important: selected_match is True on the row that would be chosen as the official mapping (equivalent to what goes to occurrence in the full pipeline). You may get multiple candidate rows per name; the rest are for review. Candidate count is controlled by -n (or by gbif_match_limit from config.yaml when using --use-config).

Help:

python taxamapper.py --help

Examples:

# Example input, default API and limits from taxamapper.py, writes processed-v3/taxa_mapping_INFO_<API>.xlsx
python taxamapper.py

# Map taxonomies to WoRMS using an input file
python taxamapper.py -i raw-v3/taxamapper_example_input.tsv -a WoRMS

# Up to 5 candidate rows per name, and map to GBIF backbone
python taxamapper.py -i raw-v3/taxamapper_example_input.tsv -a GBIF -n 5

# Explicit output path
python taxamapper.py -i raw-v3/taxamapper_example_input.tsv -a GBIF -o processed-v3/my_run.xlsx

# Matcher options from config.yaml
python taxamapper.py --use-config

# Parallelism
python taxamapper.py -i raw-v3/taxamapper_example_input.tsv -a GBIF --n-proc 3

Options:

• -i, --input: TSV/CSV with column verbatimIdentification (default: raw-v3/taxamapper_example_input.tsv)
• -a, --api: GBIF or WoRMS (omit with --use-config to use taxonomic_api_source from config)
• -n, --limit: max candidate rows per name (omit with --use-config to use gbif_match_limit from config)
• -o, --output: output .xlsx path (not a folder). If omitted, {outdir}/taxa_mapping_INFO_<API>.xlsx
• --outdir: directory when -o omitted (default: processed-v3)
• --use-config: load matcher settings from config.yaml beside taxamapper.py
• --n-proc: parallel workers (optional)

In-file defaults (no CLI needed):

• You can also set defaults directly at the top of taxamapper.py in the DEFAULTS block (input path, API, match limit, outdir/output, n-proc). Any CLI flags you pass will override those defaults.

Output Files

The pipeline generates several files in your output directory:

• processed-v3/
  • edna2obis_report_worms.html / edna2obis_report_gbif.html - Run report.
  • occurrence_core_worms.csv / occurrence_core_gbif.csv - Occurrence Core with assigned taxonomy.
  • taxa_mapping_INFO_WoRMS.xlsx / taxa_mapping_INFO_GBIF.xlsx - Candidate taxonomy matches; use selected_match for the chosen row. The API part of the filename matches taxonomic_api_source in config.yaml.
  • dna_derived_extension.csv - DNA Derived Data extension.
  • eMoF.csv - Extended Measurement or Fact extension (if enabled).
  • meta.xml - Darwin Core Archive mapping metadata.
  • eml.xml - Dataset metadata in EML format (if enabled).

Occurrence Core (GBIF example)

occurrenceID	eventID	verbatimIdentification	kingdom	phylum	class	order	family	genus	scientificName	taxonRank	organismQuantity	organismQuantityType	recordedBy	materialSampleID	eventDate	locality	decimalLatitude	decimalLongitude	basisOfRecord	nameAccordingTo
GU190706-CTD11-220_MiFish_S30_occ_18109634cc2f8e156e5402bf13cf4502	GU190706-CTD11-220_MiFish_S30	Eukaryota;Chordata;Actinopteri;Beloniformes;Exocoetidae;Cheilopogon	Animalia	Chordata		Beloniformes	Exocoetidae	Cheilopogon	Cheilopogon Lowe, 1841	genus	4	DNA sequence reads	Lynsey Wilcox Talbot | Katherine Silliman	GU190706-CTD11-220	2019-07-06 00:00:00	USA: Gulf of Mexico	-85.793	28.662	MaterialSample	GBIF
GU190706-CTD11-220_MiFish_S30_occ_183bc18f3e5eac45c6dd248fb86d64bf	GU190706-CTD11-220_MiFish_S30	Eukaryota;Chordata;Actinopteri;Tetraodontiformes;Tetraodontidae;Lagocephalus;Lagocephalus laevigatus	Animalia	Chordata		Tetraodontiformes	Tetraodontidae	Lagocephalus	Lagocephalus laevigatus (Linnaeus, 1766)	species	5317	DNA sequence reads	Lynsey Wilcox Talbot | Katherine Silliman	GU190706-CTD11-220	2019-07-06 00:00:00	USA: Gulf of Mexico	-85.793	28.662	MaterialSample	GBIF

DNA Derived Extension

eventID	source_mat_id	samp_name	env_broad_scale	env_local_scale	env_medium	samp_vol_we_dna_ext	samp_collect_device	samp_mat_process	size_frac	concentration	lib_layout	seq_meth	nucl_acid_ext	target_gene	target_subfragment	pcr_primer_forward	pcr_primer_reverse	pcr_primer_name_forward	pcr_primer_name_reverse	pcr_primer_reference	pcr_cond	nucl_acid_amp	ampliconSize	otu_seq_comp_appr	otu_db	occurrenceID	DNA_sequence	concentrationUnit	otu_class_appr
GU190706-CTD11-220_MiFish_S30	GU190706-CTD11-220	GU190706-CTD11-220	marine biome [ENVO:00000447]	marine mesopelagic zone [ENVO:00000213]	sea water [ENVO:00002149]	2	Niskin bottle	Samples were vacuum-filtered through a MilliporeSigma 47 mm diameter mixed cellulose ester (MCE) filter with...	0.45	1.57	paired end	Illumina MiSeq [OBI_0002003]	https://doi.org/10.1002/edn3.70074	12S rRNA (SSU mitochondria)	V5-V6	GTCGGTAAAACTCGTGCCAGC	CATAGTGGGGTATCTAATCCCAGTTTGT	MiFish-U-F	MiFish-U-R2	https://doi.org/10.1098/rsos.150088	initial denaturation:98_30s; 40 cycles of denaturation: 98_20s, annealing:60_20s, elongation:72_20s; final elongation:72_5min	not applicable	175	qiime2-2023.5; naive-bayes classifier; scikit-learn 0.24.1	custom	GU190706-CTD11-220_MiFish_S30_occ_18109634cc2f8e156e5402bf13cf4502	CACCGCGGTTATACGAGAGGCCTAAGTTGACAGACAACGGCGTAAAGAGTGGTTAAGGAAAAATTTATACTAAAGCCGAACATCCTCAAGACTGTCGTACGTTTCCGAGGATATGAAGTCCCCCTACGAAAGTGGCTTTAACTCCCCTGACCCCACGAAAGCTGTGAC	ng/µl	qiime2-2023.5; DADA2 1.26.0
GU190706-CTD11-220_MiFish_S30	GU190706-CTD11-220	GU190706-CTD11-220	marine biome [ENVO:00000447]	marine mesopelagic zone [ENVO:00000213]	sea water [ENVO:00002149]	2	Niskin bottle	Samples were vacuum-filtered through a MilliporeSigma 47 mm diameter mixed cellulose ester (MCE) filter with...	0.45	1.57	paired end	Illumina MiSeq [OBI_0002003]	https://doi.org/10.1002/edn3.70074	12S rRNA (SSU mitochondria)	V5-V6	GTCGGTAAAACTCGTGCCAGC	CATAGTGGGGTATCTAATCCCAGTTTGT	MiFish-U-F	MiFish-U-R2	https://doi.org/10.1098/rsos.150088	initial denaturation:98_30s; 40 cycles of denaturation: 98_20s, annealing:60_20s, elongation:72_20s; final elongation:72_5min	not applicable	175	qiime2-2023.5; naive-bayes classifier; scikit-learn 0.24.1	custom	GU190706-CTD11-220_MiFish_S30_occ_183bc18f3e5eac45c6dd248fb86d64bf	CACCGCGGTTATACGATGAAGCCCAAGTTGTTAGCCTTCGGCGTAAAGAGTGGTTAGAGTACCCCAACAAAACTAAGGCCGAACACCTTCAGGGCAGTCATACGCTTTCGAAGGCATGAAGCACACCAACGAAAGTAGCCTTACCAGACTTGAACCCACGAAAGCTAAGAT	ng/µl	qiime2-2023.5; DADA2 1.26.0

Taxonomic Mapping Algorithms

FAQ: Why “mapping” and not “assignment”?
Taxonomic assignment (matching ASVs/OTUs to taxa) happens before edna2obis. Taxonomic mapping to WoRMS (or GBIF) takes those names and maps them to backbone records so the data can be submitted to OBIS/GBIF.

WoRMS

• Clean verbatimIdentification: split by ;, replace _ - / with spaces, trim, remove trailing “sp.”/“spp.” and numbers, drop empty/“unassigned”.
• If it exists as a species by AphiaID in the local reference database file provided (optionally), look up AphiaID directly and fill fields.
• Otherwise, use WoRMS “Match Names”:
  • Start from the most specific level (ex: species) and move broader only if needed.
  • Keep only records with status “accepted” and add every accepted alternative to the review list.
  • Use the most specific level that returns an accepted record for the mapping (ex: favors a genus match over a class match).
• If an assay is set to skip species, drop the last level before searching.
• If nothing accepted, set scientificName=incertae sedis and clear ranks.
• Fill: scientificName, scientificNameID (LSID), taxonRank, ranks (kingdom … species), nameAccordingTo=WoRMS, plus cleanedTaxonomy and match_type_debug.
• Endpoints: AphiaID record; Match Names.

GBIF

• Clean verbatimIdentification as above and remove rank prefixes like d__, p__, etc.
• Try the most specific level first; move broader only if needed.
• For each candidate from the initial lookup, confirm against the GBIF backbone to get a confidence score.
• Choose the highest-confidence accepted match by default; optionally choose by lineage assignment_score first when gbif_use_assignment_score_for_selection is true. Include accepted candidates and unaccepted/synonym review rows in the review list. If none, set scientificName=incertae sedis.
• If an assay is set to skip species, drop the last level before searching.
• Cache results to speed up future runs.
• Fill: scientificName, taxonID (e.g., gbif:<usageKey>), taxonRank, ranks (kingdom … genus), confidence, optional higherClassification, environment when available, nameAccordingTo=GBIF, plus cleanedTaxonomy and match_type_debug.
• Endpoints: name_lookup; name_backbone; name_usage parents.

Settings that affect matching

• taxonomic_api_source: "WoRMS" or "GBIF"
• assays_to_skip_species_match: assays where species level is ignored
• Per-assay maximum depth: inferred from your inputs; used for species skipping
• GBIF candidate limit and caching
• gbif_use_assignment_score_for_selection: GBIF only. If true, lineage assignment_score is used before GBIF confidence when choosing selected_match
• gbif_return_higher_classification: GBIF only. If true, calls the parents endpoint and includes higherClassification
• Local reference database (WoRMS only) for fast AphiaID lookups

Also writes taxa_mapping_INFO_<API>.xlsx (sheet taxa_mapping_INFO). One row can appear per candidate match, not only the match used in the occurrence file.

taxa_mapping_INFO columns

Important column! selected_match is True on the row the pipeline treats as the official mapping for each verbatimIdentification (the one reflected in the occurrence file). For taxamapper.py-only runs, it is True on the row that would be chosen in the full pipeline. When you summarize results, filter on selected_match = TRUE; other mapping rows exist for transparency and QA.

Columns are ordered with selected_match immediately after cleanedTaxonomy in the .xlsx file.

Column	Meaning
verbatimIdentification	Original taxonomic string from your data before matching.
cleanedTaxonomy	Normalized / cleaned version of verbatimIdentification (what is actually used for API lookup.
selected_match	Indicates the taxonomic mapping! True only on the chosen mapping row for that verbatimIdentification (occurrence file in full pipeline; same rule for taxamapper-only output). False on alternate candidate rows kept for review.
ambiguous	WoRMS and GBIF. True when more than one candidate mapping was found for this verbatim string.
replaced_unaccepted	WoRMS and GBIF. True when an unaccepted/synonym name was resolved to an accepted valid name.
unaccepted_match	WoRMS and GBIF. True when the mapping in that row is an unaccepted/synonym name shown for comparison.
ranks_matched	WoRMS and GBIF. Number of taxonomic ranks given in your verbatimIdentification that matched the candidate classification.
ranks_provided	WoRMS and GBIF. Number of taxonomic ranks given in your verbatimIdentification for that taxonomy.
assignment_score	WoRMS and GBIF. Ratio of ranks_matched / ranks_provided.
environment	WoRMS and GBIF habitat labels when available. GBIF habitat values come from GBIF usage data and may be coarser than WoRMS.
scientificName	Scientific name for this candidate row from the backbone.
confidence	GBIF only. Assignment confidence score from 0 to 100.
taxonRank	Rank of scientificName assigned.
scientificNameID	WoRMS only. LSID for the taxon.
taxonID	GBIF only. Backbone taxonomic identifier.
higherClassification	WoRMS and optionally GBIF. Pipe-separated lineage. GBIF uses the name_usage parents endpoint when gbif_return_higher_classification is true.
kingdom	Darwin Core kingdom.
phylum	Darwin Core phylum.
class	Darwin Core class.
order	Darwin Core order.
family	Darwin Core family.
genus	Darwin Core genus.
match_type_debug	Short code for how this row was matched via the API.
nameAccordingTo	Which API you used to assign taxonomy.

eMoF (extended Measurement or Fact)

The eMoF file captures occurrence-level measurements linked to each occurrenceID that made it into the final occurrence file. This creates a larger file but is what GBIF requires for data submission.

• What it contains: One row per occurrence per configured measurement.
• Where it comes from: Measurement values are sourced from sampleMetadata first, otherwise experimentRunMetadata.
• Units:
  • If the eMoF template specifies a literal unit (e.g., m, °C), that unit is used for every emitted row of that measurementType.
  • If the template says provided, a column named <measurementType>_unit must be present in the chosen source sheet and must be non-blank for all emitted rows.
  • If the template leaves the unit blank, output unit is blank (no auto-fallback).
• Template: Configure measurements in raw-v3/eMoF_Fields_edna2obis.xlsx on the input_file sheet. Required columns: measurementType, measurementValue, measurementUnit, measurementTypeID, measurementValueID, measurementUnitID, measurementRemarks.
• Output: Written to processed-v3/eMoF.csv.

Customizing eMoF Measurements

You can add custom measurements to your eMoF file by editing the template file raw-v3/eMoF_Fields_edna2obis.xlsx. This allows you to include:

• Environmental measurements: Temperature, salinity, pH, dissolved oxygen, etc.
• Sampling parameters: Gear type, sampling depth, volume filtered, etc.
• Laboratory measurements: DNA concentration, extraction efficiency, etc.
• Any other data: Linked to your events or occurrences

For detailed guidance on formatting eMoF files and using controlled vocabularies, see the OBIS eMoF formatting guide.

eMoF Preview (example)

Below is a small, illustrative preview of the eMoF structure. Your actual content will depend on your eMoF template and metadata.

eventID	occurrenceID	verbatimMeasurementType	measurementType	measurementValue	measurementUnit	measurementTypeID	measurementValueID	measurementUnitID	measurementRemarks
GU190706-CTD11-220_MiFish_S30	GU190706-CTD11-220_MiFish_S30_occ_ASV_001	temperature	Temperature	25.5	°C	http://vocab.nerc.ac.uk/collection/P01/current/TEMPPR01/		http://vocab.nerc.ac.uk/collection/P06/current/UPAA/
GU190706-CTD11-220_MiFish_S30	GU190706-CTD11-220_MiFish_S30_occ_ASV_002	temperature	Temperature	25.5	°C	http://vocab.nerc.ac.uk/collection/P01/current/TEMPPR01/		http://vocab.nerc.ac.uk/collection/P06/current/UPAA/

Troubleshooting

Common Issues

1. Environment creation fails

 # Try updating conda first
 conda update conda
 conda env create -f environment.yml

2. API timeout errors

• Reduce worms_n_proc or gbif_n_proc in config.yaml

3. Missing data files

• Verify all file paths in config.yaml are correct
• Use absolute paths if relative paths don't work

Getting Help

• Check the HTML report for detailed error messages
• Review the terminal output for specific error details
• Ensure your input data follows the FAIRe NOAA format

Recommended System Requirements

• Processing: 8GB+ RAM, 4+ CPU cores
• Storage: ~1GB free space for large datasets
• Network: Stable internet for API calls

Disclaimer

This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project code is provided on an 'as is' basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.