Skip to content

Feat/ct refresh and ig metadata - #3

Merged
billdenney merged 8 commits into
mainfrom
feat/ct-refresh-and-ig-metadata
Oct 1, 2026
Merged

billdenney merged 8 commits into
mainfrom
feat/ct-refresh-and-ig-metadata

Conversation

@billdenney

Copy link
Copy Markdown
Member

No description provided.

billdenney and others added 6 commits September 30, 2026 17:13
…esh to 2026-09-25

The NCI CT parser's header-row detection assumed a codelist's own C-code is
repeated in both its "Code" and "Codelist Code" columns. From 2012 onward,
NCI instead leaves "Codelist Code" blank on that row, which the old check
(raw_code == raw_clst_code) never matched. This silently left
codelist_name NA for every row of ct_sdtm/ct_adam for every release from
2012 on, and made resolving a codelist by name (e.g. PKPARMCD, PKPARM,
PKUNIT) unreliable - exactly what get_ig()/build_domain_spec() need.

Also fixes fetch_current_release_date() for NCI's 2026 change to the
publication-date-stamp file format (now a small table instead of a bare
date on line 1): it now scans every line for a date and returns the newest
one found, rather than assuming the date is on line 1.

All bundled historical releases were re-parsed from scratch with the fix
(re-downloading the ~110 archived releases involved), then the 2026-09-25
SDTM and ADaM CT releases were added on top via the normal fetch scripts.

A handful of older (2007-2010, plus one 2017) archival releases have
further, rarer source-data quirks this could not fully resolve; the
*current* release is verified clean and the historical gap is bounded by
new tests rather than silently tolerated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Lazy-loaded package data (ct_sdtm, ct_adam, datasets_catalogue) is only
wired into the search path when a package is *attached* with
library()/require(). Loading just the namespace - as `::` does, and as
every `pkg::fun()` call does - left an unqualified reference to that data
unresolved even from inside the package's own functions defined in that
namespace: cdiscdata::get_ct("sdtm") without a prior library() call errored
with "object 'datasets_catalogue' not found" (reproduced, and confirmed
fixed, via a fresh callr::r() subprocess).

Fixed with an internal .pkg_data() helper that loads a dataset explicitly
via utils::data() into a throwaway environment, which works identically
regardless of attach/namespace-load state.

Adds a regression test (test-attach.R) that builds and installs the
package into a fresh temporary library and exercises it via callr::r() in
a brand-new subprocess with no library() call - reproducing the original
bug precisely and verifying the fix end to end, not just at the unit level.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds ig_sdtm (SDTM Model Findings general-observation-class variables,
versions 1.4-1.7, plus the SDTMIG PP domain and generic SUPP-- structure,
version 3.2) and ig_adam (ADaMIG ADSL and BDS variable tables, versions
1.0-1.2), a get_ig(standard, version) accessor mirroring get_ct(), and
build_domain_spec(domain, ig_version, ct_version), which joins an IG table
to CT to produce a ready-to-use variable spec (name, label, type, length,
core, order, codelist id) for "PP", "SUPPPP" (via the generic SUPP--
structure - CDISC has no SUPPPP-specific table), or "ADPP" (via the
generic BDS structure - CDISC has no ADPP-specific table either).

Resolved codelist ids match nca.reporter's independently-derived values
exactly (e.g. PPTESTCD -> C85839, PPSTAT -> C66789), cross-validating both.

IG metadata is transcribed from CDISC's own published SDTM
Model/SDTMIG/ADaMIG specification tables, copied read-only (with
attribution) from Bill Denney's private Rsdtm package - see
data-raw/ig_source/README.md for full provenance, licensing notes, and the
known coverage gap (SDTMIG is only available at version 3.2; Rsdtm's
SDTMIG_3.3 copy does not include PP or Supplemental Qualifiers). This
supersedes the package's original plan to leave all IG metadata to a
separate cdiscapi package; the README has been updated accordingly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Version and Description text for the CT refresh, attach-only fix, and IG
metadata additions in this release; see NEWS.md for the full changelog.
Also picks up roxygen2's Config/roxygen2/version field (replacing
RoxygenNote), from running devtools::document() with the currently
installed roxygen2 (8.1.0) - no functional change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
build_domain_spec("ADPP") gains an adsl argument (default TRUE) that unions
the ADaMIG ADSL variables into the BDS-only spec, since a real ADPP dataset
carries ADSL's subject-level variables (treatment, demographics, ...)
alongside its own BDS variables. New rows are marked source = "ADSL" and
core is forced to "Perm" (ADSL's own Core reflects requirements for ADSL
itself, not for merging a variable into ADPP, which is always optional). A
variable both tables define (STUDYID, USUBJID, ...) keeps its BDS version
rather than being duplicated. adsl = FALSE returns the prior BDS-only
table. All domains now carry a source column (SDTMIG for PP/SUPPPP; BDS or
ADSL for ADPP) for provenance.

Documents that PP-inherited traceability variables (PPTESTCD and the rest
of PP's variables that a BDS dataset built from PP typically carries
forward) are not part of either IG table and remain a downstream
derivation choice.

Also documents get_ig()'s SDTMIG-3.2-only coverage of PP/SUPP-- explicitly,
including the specific variables/renames this implies are missing
(PPANMETH, EPOCH, PTAETORD; the PPDTC-to-PPPDTC rename) until a newer
SDTMIG source is available.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
# Conflicts:
#	data-raw/utils_nci.R
@codecov

codecov Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
R/available_versions.R 100.00% <100.00%> (ø)
R/build_domain_spec.R 100.00% <100.00%> (ø)
R/cdiscdata_versions.R 100.00% <100.00%> (ø)
R/get_dataset.R 100.00% <100.00%> (ø)
R/get_ig.R 100.00% <100.00%> (ø)
R/list_datasets.R 100.00% <100.00%> (ø)
R/utils.R 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

billdenney and others added 2 commits September 30, 2026 19:40
…ssing

list_archive_dates() had been silently returning zero dates: NCI rebuilt
evs.nci.nih.gov as a JavaScript single-page app at some point in
2024-2025, so the Archive/ directory URL this scraped for file names now
serves that app's empty HTML shell instead of a plain listing. Replaced
with a call to the JSON API the app itself uses
(GET /ftp1/folder?folder=<path>, an S3 ListObjectsV2-shaped response); see
the comment above list_archive_dates() for that API's one limitation (no
working pagination past 1000 entries) and why it does not affect finding
every release date in practice (verified empirically for both SDTM and
ADaM).

Investigated a reported gap in available_ct_versions(): SDTM CT appeared
to be missing the 2024-06, 2024-12, 2025-06, 2025-12, and 2026-06
quarterly releases. Verified directly against NCI's file listing, not
assumed: none of these five releases exist for either SDTM or ADaM - NCI
shifted SDTM CT from quarterly to semi-annual (March/September) releases
starting in 2024, and ADaM CT has never been reliably quarterly (gaps of
266-448 days recur from 2017 on). Nothing to backfill.

Added gap-detection tests calibrated to each type's actual verified
history (200 days for SDTM, 500 days for ADaM - not a single ~120-day
bound, which would flag the genuine 2024-on semi-annual SDTM gaps, and
most of ADaM's history, as failures) plus a freshness check, so a release
that is actually skipped, or a fetch workflow that silently stops running,
still fails loudly.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every error and warning these two functions raise is now classed
(cdiscdata_error_ig_version_unavailable, cdiscdata_error_no_ig_variables,
cdiscdata_warning_adsl_ignored) via a small internal .cdiscdata_abort()/
.cdiscdata_warn() helper (R/utils.R), so callers can catch or assert on
them by class rather than by matching message text - without adding a
runtime dependency (e.g. rlang) for it, keeping the package's
zero-runtime-dependency design.

build_domain_spec.R was at 93% line coverage (codecov: 4 uncovered lines,
all in the two "no variables found for this domain/dataset" defensive
checks). Neither is reachable via the public API with the currently
bundled ig_sdtm/ig_adam, since every version present there has PP/SUPPQUAL
rows (SDTMIG) or BDS rows (ADaMIG); both are now exercised directly by
mocking get_ig() to return a version that validates but has zero rows for
the requested domain/dataset. Every error-path test now asserts the exact
condition class (and still checks the message), not just that some error
occurred. Confirmed 100.00% coverage package-wide with
covr::package_coverage() before pushing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@billdenney
billdenney merged commit f3158ab into main Oct 1, 2026
9 checks passed
@billdenney
billdenney deleted the feat/ct-refresh-and-ig-metadata branch October 1, 2026 01:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant