Repository navigation
Feat/ct refresh and ig metadata - #3
Merged
Merged
Conversation
…esh to 2026-09-25 The NCI CT parser's header-row detection assumed a codelist's own C-code is repeated in both its "Code" and "Codelist Code" columns. From 2012 onward, NCI instead leaves "Codelist Code" blank on that row, which the old check (raw_code == raw_clst_code) never matched. This silently left codelist_name NA for every row of ct_sdtm/ct_adam for every release from 2012 on, and made resolving a codelist by name (e.g. PKPARMCD, PKPARM, PKUNIT) unreliable - exactly what get_ig()/build_domain_spec() need. Also fixes fetch_current_release_date() for NCI's 2026 change to the publication-date-stamp file format (now a small table instead of a bare date on line 1): it now scans every line for a date and returns the newest one found, rather than assuming the date is on line 1. All bundled historical releases were re-parsed from scratch with the fix (re-downloading the ~110 archived releases involved), then the 2026-09-25 SDTM and ADaM CT releases were added on top via the normal fetch scripts. A handful of older (2007-2010, plus one 2017) archival releases have further, rarer source-data quirks this could not fully resolve; the *current* release is verified clean and the historical gap is bounded by new tests rather than silently tolerated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Lazy-loaded package data (ct_sdtm, ct_adam, datasets_catalogue) is only
wired into the search path when a package is *attached* with
library()/require(). Loading just the namespace - as `::` does, and as
every `pkg::fun()` call does - left an unqualified reference to that data
unresolved even from inside the package's own functions defined in that
namespace: cdiscdata::get_ct("sdtm") without a prior library() call errored
with "object 'datasets_catalogue' not found" (reproduced, and confirmed
fixed, via a fresh callr::r() subprocess).
Fixed with an internal .pkg_data() helper that loads a dataset explicitly
via utils::data() into a throwaway environment, which works identically
regardless of attach/namespace-load state.
Adds a regression test (test-attach.R) that builds and installs the
package into a fresh temporary library and exercises it via callr::r() in
a brand-new subprocess with no library() call - reproducing the original
bug precisely and verifying the fix end to end, not just at the unit level.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Adds ig_sdtm (SDTM Model Findings general-observation-class variables, versions 1.4-1.7, plus the SDTMIG PP domain and generic SUPP-- structure, version 3.2) and ig_adam (ADaMIG ADSL and BDS variable tables, versions 1.0-1.2), a get_ig(standard, version) accessor mirroring get_ct(), and build_domain_spec(domain, ig_version, ct_version), which joins an IG table to CT to produce a ready-to-use variable spec (name, label, type, length, core, order, codelist id) for "PP", "SUPPPP" (via the generic SUPP-- structure - CDISC has no SUPPPP-specific table), or "ADPP" (via the generic BDS structure - CDISC has no ADPP-specific table either). Resolved codelist ids match nca.reporter's independently-derived values exactly (e.g. PPTESTCD -> C85839, PPSTAT -> C66789), cross-validating both. IG metadata is transcribed from CDISC's own published SDTM Model/SDTMIG/ADaMIG specification tables, copied read-only (with attribution) from Bill Denney's private Rsdtm package - see data-raw/ig_source/README.md for full provenance, licensing notes, and the known coverage gap (SDTMIG is only available at version 3.2; Rsdtm's SDTMIG_3.3 copy does not include PP or Supplemental Qualifiers). This supersedes the package's original plan to leave all IG metadata to a separate cdiscapi package; the README has been updated accordingly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Version and Description text for the CT refresh, attach-only fix, and IG metadata additions in this release; see NEWS.md for the full changelog. Also picks up roxygen2's Config/roxygen2/version field (replacing RoxygenNote), from running devtools::document() with the currently installed roxygen2 (8.1.0) - no functional change. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
build_domain_spec("ADPP") gains an adsl argument (default TRUE) that unions
the ADaMIG ADSL variables into the BDS-only spec, since a real ADPP dataset
carries ADSL's subject-level variables (treatment, demographics, ...)
alongside its own BDS variables. New rows are marked source = "ADSL" and
core is forced to "Perm" (ADSL's own Core reflects requirements for ADSL
itself, not for merging a variable into ADPP, which is always optional). A
variable both tables define (STUDYID, USUBJID, ...) keeps its BDS version
rather than being duplicated. adsl = FALSE returns the prior BDS-only
table. All domains now carry a source column (SDTMIG for PP/SUPPPP; BDS or
ADSL for ADPP) for provenance.
Documents that PP-inherited traceability variables (PPTESTCD and the rest
of PP's variables that a BDS dataset built from PP typically carries
forward) are not part of either IG table and remain a downstream
derivation choice.
Also documents get_ig()'s SDTMIG-3.2-only coverage of PP/SUPP-- explicitly,
including the specific variables/renames this implies are missing
(PPANMETH, EPOCH, PTAETORD; the PPDTC-to-PPPDTC rename) until a newer
SDTMIG source is available.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
# Conflicts: # data-raw/utils_nci.R
Codecov Report✅ All modified and coverable lines are covered by tests.
🚀 New features to boost your workflow:
|
…ssing list_archive_dates() had been silently returning zero dates: NCI rebuilt evs.nci.nih.gov as a JavaScript single-page app at some point in 2024-2025, so the Archive/ directory URL this scraped for file names now serves that app's empty HTML shell instead of a plain listing. Replaced with a call to the JSON API the app itself uses (GET /ftp1/folder?folder=<path>, an S3 ListObjectsV2-shaped response); see the comment above list_archive_dates() for that API's one limitation (no working pagination past 1000 entries) and why it does not affect finding every release date in practice (verified empirically for both SDTM and ADaM). Investigated a reported gap in available_ct_versions(): SDTM CT appeared to be missing the 2024-06, 2024-12, 2025-06, 2025-12, and 2026-06 quarterly releases. Verified directly against NCI's file listing, not assumed: none of these five releases exist for either SDTM or ADaM - NCI shifted SDTM CT from quarterly to semi-annual (March/September) releases starting in 2024, and ADaM CT has never been reliably quarterly (gaps of 266-448 days recur from 2017 on). Nothing to backfill. Added gap-detection tests calibrated to each type's actual verified history (200 days for SDTM, 500 days for ADaM - not a single ~120-day bound, which would flag the genuine 2024-on semi-annual SDTM gaps, and most of ADaM's history, as failures) plus a freshness check, so a release that is actually skipped, or a fetch workflow that silently stops running, still fails loudly. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every error and warning these two functions raise is now classed (cdiscdata_error_ig_version_unavailable, cdiscdata_error_no_ig_variables, cdiscdata_warning_adsl_ignored) via a small internal .cdiscdata_abort()/ .cdiscdata_warn() helper (R/utils.R), so callers can catch or assert on them by class rather than by matching message text - without adding a runtime dependency (e.g. rlang) for it, keeping the package's zero-runtime-dependency design. build_domain_spec.R was at 93% line coverage (codecov: 4 uncovered lines, all in the two "no variables found for this domain/dataset" defensive checks). Neither is reachable via the public API with the currently bundled ig_sdtm/ig_adam, since every version present there has PP/SUPPQUAL rows (SDTMIG) or BDS rows (ADaMIG); both are now exercised directly by mocking get_ig() to return a version that validates but has zero rows for the requested domain/dataset. Every error-path test now asserts the exact condition class (and still checks the message), not just that some error occurred. Confirmed 100.00% coverage package-wide with covr::package_coverage() before pushing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.