[grantfitzsimmons]

Fixes #286 (#8666), #6435, and #8175

This PR adds a visual editor for creating, editing, and validating Darwin Core Archive definitions.

The editor supports:

• Darwin Core and GBIF core/extension selection
• Default templates for common Specify mappings
• Automatic term mapping based on Specify fields
• Saved query reuse
• Custom terms and row types
• Required-term, filename, and row-type validation
• Multiple extensions
• Data-model alignment information in Schema Config
• New backend validation before starting an export
• Actionable validation errors when an export definition is invalid

The PR also adds a workflow and generator for periodically updating the bundled GBIF vocabulary catalogs. 🎉

For what it's worth, as of October 1st, the size of this PR is primarily generated vocabulary data:

• gbifCores.json: 3,508 added lines
• gbifExtensions.json: 9,182 added lines
• Generated catalog data total: 12,690 lines

They are generated by generateGbifCatalog.ts and can be refreshed through the included GitHub Actions workflow! No manual updating required (at least not until they migrate from XML to something else).

Most of the other changes are the visual editor itself, then tests, field-to-term matching patterns, default mapping templates, and more validation and localization.

This PR also adds some necessary changes to facilitate certain fields like recordedByID and identifiedByID (e.g., additional table formats and aggregators in a single query). This was implemented by supporting as many (formatted) or (aggregated) query fields as you have available formats or aggregations defined for that table. See:

MultipleTableFormats.mov

Checklist

• Self-review the PR after opening it to make sure the changes look good and self-explanatory (or properly documented)
• Add relevant issue to release milestone
• Add PR to documentation list
• Add automated tests
• Add a reverse migration if a migration is present in the PR — not applicable; this PR has no migrations
• Add migration function to the migration command — not applicable; this PR has no migrations

Testing instructions

There is comprehensive automatic test coverage for the changes in this PR, but the validation will require a lot of test data sets (taken from those who already have set up mappings) and new ones.

Make sure to review and understand Exporting & Publishing Data using Darwin Core (DwC-A) (GBIF, iDigBio, etc.) completely before testing.

We need to validate the produced packages using the GBIF Data Validator, which will require that you make a GBIF account.

I encourage you to also create a DwCA export in v7 and compare it against the export produced in issue-286. They should be identical, any any exports prepared in this branch on the 'Occurrence Core' (Collection Object-based exports) should be supported on main and v7 just fine. EML will still need to be sourced from GBIF (as in the original docs) or generated using this tool from GBIF Norway.

Convert an Existing Mapping

To convert an existing DwCA mapping from a generic XML app resource:

1. Navigate to the existing DwCA app resource defining the mapping (not the EML). Most have DwC somewhere in the title itself!
   
2. Change the 'Mime Type' from text/xml to application/vnd.specify.dwca+xml
   
3. Click Save. Once it refreshes, you should see the visual editor.

Instructions

• Select an Occurrence, Event, or Taxon core and confirm the appropriate default mapping is loaded.
• Add an extension and verify that:
  • extensions with the same display name remain distinct;
  • the correct default template is selected;
  • generated filenames are unique;
  • saved queries can be used as mapping sources.
• Add a custom extension and confirm saving is blocked until both its filename and row type are populated.
• Confirm duplicate filenames and missing required terms prevent saving.
• Reorder mapping fields and confirm mapped read-only fields cannot be moved or displaced.
• Save and reopen the definition, confirming filenames, terms, queries, and mappings are preserved.
• Run an export with a valid definition.
• Attempt an export with an invalid definition and confirm the specific validation error is returned.
• Open Schema Config and verify Darwin Core/GBIF alignments are shown only for fields whose identities match the configured suffix patterns (see coreTermPatterns.ts if you need to see the matches)

Summary by CodeRabbit

• New Features

  • Added a visual editor for Darwin Core Archive definitions, including templates, field mappings, extensions, saved queries, XML validation, and export settings.
  • Added support for Darwin Core definitions as application resources and in the export workflow.
  • Added data-model alignment information to schema field configuration.
  • Added discipline-specific Darwin Core and GBIF term mappings.
  • Added read-only field controls and customizable field behavior in the query builder.
  • Added expandable warning details for resource-scope issues.

• Bug Fixes

  • Improved validation and error reporting for invalid Darwin Core export definitions.
  • Improved handling of external identities without a name.

[JDAM2k4]

It's a bit confusing that once you've chosen a query or mapping from the Choose Template or Query dropdown that it doesn't save the selection. I think I loaded the same query twice before I realized that was the "issue."

I don't think the custom row type currently works, as it turns the save button red with no real indication as to why. The error it gives is just App Resource Datas: Field is required. which is very vague and not intuitive to solving a potential problem in the mapping.

I find it kind of silly that basic view just shoves the rest of the query to the lefthand side and shows you the icon of the main table for that row. It is a bit weird that this doesn't follow any other pre-existing query format in Specify, but maybe I wasn't looking hard enough.

Ultimately, I am not familiar enough with DwCA extensions to comment on them too much. It doesn't seem like they automatically map as well, even when you give it a query built for whatever you are using. I like the warning at the top of the screen if a DwCA term is not mapped, but it took me a minute to understand that that was what it was referring to.

A label to identify the columns in the query might be nice, because the DarwinCore names can be very similar or very different to what is in a database. It would also be nice if there was a color difference in the cell or row when a term is not mapped, like in the Workbench. As of now, it is a little hard to decern Choose a term from the column easily.

It does not need to be implemented now, but perhaps a Default Mapping button could be nice. Or something similar to speed up the RSS export workflow so you do not have to manually change it when you set up the export.