Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions DESCRIPTION
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
Package: cdiscdata
Title: Versioned CDISC Standards Reference Data
Version: 0.1.0
Version: 0.2.0
Authors@R: person("Bill", "Denney", email = "wdenney@humanpred.com",
role = c("aut", "cre"),
comment = c(ORCID = "0000-0002-5759-428X"))
Description: Provides versioned CDISC Controlled Terminology (SDTM, ADaM)
Description: Provides versioned CDISC Controlled Terminology (SDTM, ADaM),
SDTM Model/SDTMIG and ADaMIG implementation-guide variable metadata,
and Define-XML schemas (XSD) and CDISC XSLT stylesheets. Uses a
validity-date design to store all historical CT versions compactly
without redundant row duplication. Intended as a shared data
Expand All @@ -18,12 +19,14 @@ Suggests:
covr,
pkgdown,
knitr,
rmarkdown
rmarkdown,
callr,
pkgbuild
LazyData: true
LazyDataCompression: xz
Encoding: UTF-8
Config/testthat/edition: 3
URL: https://github.com/humanpred/cdiscdata, https://humanpred.github.io/cdiscdata/
BugReports: https://github.com/humanpred/cdiscdata/issues
VignetteBuilder: knitr
RoxygenNote: 7.3.3
Config/roxygen2/version: 8.1.0
2 changes: 2 additions & 0 deletions NAMESPACE
Original file line number Diff line number Diff line change
@@ -1,9 +1,11 @@
# Generated by roxygen2: do not edit by hand

export(available_ct_versions)
export(build_domain_spec)
export(cdiscdata_versions)
export(get_ct)
export(get_dataset)
export(get_ig)
export(list_datasets)
export(schema_path)
export(stylesheet_path)
124 changes: 124 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,127 @@
# cdiscdata 0.2.0

## Classed conditions and coverage

* Every error and warning `get_ig()` and `build_domain_spec()` raise is now
classed (e.g. `cdiscdata_error_ig_version_unavailable`,
`cdiscdata_error_no_ig_variables`, `cdiscdata_warning_adsl_ignored`), so
they can be caught or asserted on by class rather than by matching
message text. Implemented with a small internal `.cdiscdata_abort()`/
`.cdiscdata_warn()` helper rather than a new dependency (e.g. rlang), to
keep the package's zero-runtime-dependency design.
* `build_domain_spec()` is now at 100% line coverage; two defensive checks
(an SDTMIG/ADaMIG version with no rows for the requested domain/dataset -
not reachable via the public API with the currently bundled `ig_sdtm`/
`ig_adam`, since every version there has PP/SUPPQUAL/BDS rows) are
exercised directly via mocking `get_ig()`.

## CT refresh

* Added the 2026-09-25 NCI SDTM and ADaM CT releases (previous latest:
2026-03-27), while keeping every historical release already bundled.
- SDTM CT 2026-09-25: 41 new/changed codelists, 765 new/changed terms.
- ADaM CT 2026-09-25: 2 new/changed codelists, 11 new/changed terms.
* Fixed a parser bug (`data-raw/utils_nci.R`) that had silently left
`codelist_name` `NA` for every row of `ct_sdtm`/`ct_adam` for every NCI
release from 2012 onward (`codelist_name` and `codelist_code` are used to
look up a codelist by name, e.g. by `get_ig()`/`build_domain_spec()`, so
this previously made resolving PKPARMCD/PKPARM/PKUNIT - or any other
codelist by name - unreliable). The header-row detection this relied on
assumed the NCI source file repeats a codelist's own C-code in both its
"Code" and "Codelist Code" columns; from 2012 on, NCI instead leaves
"Codelist Code" blank on that row, which the previous logic never
matched. All bundled historical releases were re-parsed from scratch with
the fix (re-downloading the ~110 archived releases involved, now cached
as raw text going forward so a future parser fix would not need to). A
handful of older (2007-2010, plus one 2017) archival releases have
further, rarer source-data quirks this could not fully resolve; the
*current* release is verified clean, and the historical gap is bounded by
`tests/testthat/test-data_integrity.R` rather than silently tolerated.
* Fixed `list_archive_dates()`, which had been silently returning zero
dates: NCI rebuilt evs.nci.nih.gov as a JavaScript single-page app at some
point in 2024-2025, so the Archive/ directory URL this scraped for file
names now serves that app's empty HTML shell instead of a plain listing.
It now calls the JSON API the app itself uses
(`GET /ftp1/folder?folder=<path>`) instead; see the comment above
`list_archive_dates()` in `data-raw/utils_nci.R` for that API's one
limitation (no working pagination past 1000 entries) and why it does not
affect finding every release date in practice. `fetch_current_release_date()`
(previously reading NCI's Publication Date Stamp file, which changed
format at least twice - a bare date, then "SDTM Terminology YYYY-MM-DD",
then, as of 2026, a small table - and 403'd from GitHub Actions in
between) was replaced with a derivation from this same, now-working
archive listing (`origin/main` PR "fix/nci-stamp-403-fallback", merged
into this branch).
* Investigated a reported gap in `available_ct_versions()` - SDTM CT
appeared to be missing the 2024-06, 2024-12, 2025-06, 2025-12, and
2026-06 quarterly releases. Verified directly against NCI's own file
listing (not assumed): none of these five releases exist for either
SDTM or ADaM - NCI shifted SDTM CT from quarterly to semi-annual
(March/September) releases starting in 2024, and ADaM CT has never been
reliably quarterly (gaps of 266-448 days recur from 2017 on, long before
2024). There was nothing to backfill. Added
`tests/testthat/test-data_integrity.R` gap tests calibrated to each
type's actual verified history (200 days for SDTM, 500 days for ADaM;
not a single ~120-day bound, which would flag the genuine 2024-on
semi-annual gaps as failures) plus a freshness check, so a release that
is *actually* skipped or a fetch workflow that silently stops running
would still fail loudly.
* Verified `PKPARMCD` (C85839), `PKPARM` (C85493), and `PKUNIT` (C85494)
are present in the 2026-09-25 release and returned by `get_ct("sdtm")`.

## Attach-only bug fix

* Fixed `get_ct()`, `get_dataset()`, `list_datasets()`,
`cdiscdata_versions()`, and `available_ct_versions()` to work via
`cdiscdata::` alone, without `library(cdiscdata)` first. Lazy-loaded
package data (`ct_sdtm`, `ct_adam`, `ig_sdtm`, `ig_adam`,
`datasets_catalogue`) is only wired into the search path when the package
is *attached*; loading just the namespace (as `::` does) left an
unqualified reference to that data unresolved, even from inside the
package's own functions - e.g. `cdiscdata::get_ct("sdtm")` without a
prior `library()` call errored with `object 'datasets_catalogue' not
found`. Fixed via an internal `.pkg_data()` helper that loads a dataset
explicitly with `utils::data()`, independent of attach state. Verified by
a dedicated test (`tests/testthat/test-attach.R`) that builds and
installs the package into a fresh library and calls it via `callr::r()`
in a brand-new subprocess with no `library()` call.

## New: implementation-guide (IG) variable metadata

* Added `ig_sdtm`: SDTM Model Findings general-observation-class variables
(versions 1.4-1.7), plus the SDTMIG PP domain and the generic SUPP--
qualifier structure (version 3.2 only - see
`data-raw/ig_source/README.md` for why no newer SDTMIG version's PP/SUPP--
tables were available to source).
* Added `ig_adam`: ADaMIG ADSL (subject-level) and BDS (Basic Data
Structure, used by ADPP) variable tables, for versions 1.0, 1.1, and 1.2.
* Added `get_ig(standard, version)`, mirroring `get_ct()`.
* Added `build_domain_spec(domain, ig_version, ct_version, adsl)`, returning
a ready-to-use variable spec (name, label, type, length, core, order,
source, codelist id) for `"PP"`, `"SUPPPP"` (via the generic SUPP--
structure), or `"ADPP"` (via the generic BDS structure, joined against CT
for codelist ids). For `"ADPP"`, `adsl = TRUE` (the default) also unions
in the ADaMIG ADSL variables - a real ADPP carries ADSL's subject-level
variables alongside its own BDS variables - marked `source = "ADSL"` and
`core = "Perm"` (ADSL's own Core reflects requirements for ADSL itself,
not for merging a variable into ADPP, which is always optional); a
variable defined by both tables (e.g. `STUDYID`, `USUBJID`) keeps its BDS
version rather than being duplicated. `adsl = FALSE` returns the BDS
variables alone, as before. Neither table carries PP's own variables
(`PPTESTCD` and the rest) forward into ADPP; that remains a downstream
derivation choice, not IG metadata this function sources.
* `get_ig()`/`ig_sdtm`'s SDTMIG-3.2-only coverage of the PP domain and
SUPP-- structure is now called out explicitly in `get_ig()`'s
documentation, including the specific known gaps versus a newer SDTMIG
(`PPANMETH`, `EPOCH`, `PTAETORD` not yet added; SDTMIG 3.2 still has
`PPDTC`, not the later `PPPDTC` rename).
* IG metadata is transcribed from CDISC's own published SDTM
Model/SDTMIG/ADaMIG specification tables, copied read-only (with
attribution) from Bill Denney's private Rsdtm package - see
`data-raw/ig_source/README.md`. This supersedes the package's original
plan to leave all IG metadata to a separate `cdiscapi` package; the
README has been updated accordingly.

# cdiscdata 0.1.0

* First release of `cdiscdata`.
Expand Down
2 changes: 1 addition & 1 deletion R/available_versions.R
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
#' available_ct_versions("adam")
available_ct_versions <- function(type = c("sdtm", "adam")) {
type <- match.arg(type)
tbl <- switch(type, sdtm = ct_sdtm, adam = ct_adam)
tbl <- switch(type, sdtm = .pkg_data("ct_sdtm"), adam = .pkg_data("ct_adam"))
# valid_from must never be NA — enforced at ingestion time in data-raw/
dates <- sort(unique(tbl$valid_from), decreasing = TRUE)
as.character(dates)
Expand Down
154 changes: 154 additions & 0 deletions R/build_domain_spec.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
#' Build a variable specification for PP, SUPPPP, or ADPP
#'
#' Combines \code{\link{get_ig}} (variable name, label, type, core, order)
#' with \code{\link{get_ct}} (codelist id lookup) into a single, ready-to-use
#' variable specification for one of the three PK datasets. \code{PP} and
#' \code{SUPPPP} draw on the SDTMIG (\code{SUPPPP} from the generic SUPP--
#' structure, since CDISC has no domain-specific SUPPPP table); \code{ADPP}
#' draws on the ADaMIG BDS (Basic Data Structure) table, since ADPP is a BDS
#' dataset and CDISC has no ADPP-specific table either, optionally unioned
#' with the ADaMIG ADSL table (see \code{adsl} below), since a real ADPP
#' also carries ADSL's subject-level variables.
#'
#' This does not add PP-inherited traceability variables (\code{PPTESTCD},
#' \code{PPTEST}, and the rest of PP's variables that a BDS dataset built
#' from PP typically carries forward). Neither the ADaMIG BDS nor ADSL
#' tables define those; which of PP's variables to carry into ADPP, and
#' under what names, is a downstream derivation choice (e.g. admiral's own
#' conventions), not IG metadata this function can source.
#'
#' @param domain One of \code{"PP"}, \code{"SUPPPP"}, or \code{"ADPP"}.
#' @param ig_version IG version to use: an SDTMIG version (for \code{PP}/
#' \code{SUPPPP}) or an ADaMIG version (for \code{ADPP}).
#' \code{NULL} uses the newest available for that IG.
#' @param ct_version CT version to resolve codelist ids against; passed to
#' \code{\link{get_ct}}. \code{NULL} uses the newest available.
#' @param adsl For \code{domain = "ADPP"} only: whether to union in the
#' ADaMIG ADSL variables (default \code{TRUE}), since a real ADPP dataset
#' carries ADSL's subject-level variables (treatment, demographics, ...)
#' alongside its own BDS variables. \code{FALSE} returns the BDS variables
#' alone. Ignored (with a warning) for \code{domain != "ADPP"}.
#' @return A data frame with columns \code{variable}, \code{label},
#' \code{type}, \code{length}, \code{core}, \code{order}, \code{source}
#' (\code{"SDTMIG"} for \code{PP}/\code{SUPPPP}; \code{"BDS"} or
#' \code{"ADSL"} for \code{ADPP}), and \code{codelist_id} (the codelist's
#' CT C-code, e.g. \code{"C85839"} for PPTESTCD's PKPARMCD codelist;
#' \code{NA} when the variable has no codelist or the referenced codelist
#' name is not found in the CT version used). ADSL-sourced \code{ADPP} rows
#' have \code{core} forced to \code{"Perm"} regardless of their Core
#' designation in ADSL itself: from ADPP's perspective, merging in an
#' ADSL variable is a common but optional choice, not a requirement ADSL's
#' own Core reflects.
#' @export
#' @examples
#' build_domain_spec("PP")
#' build_domain_spec("SUPPPP")
#' build_domain_spec("ADPP") # BDS + ADSL (default)
#' build_domain_spec("ADPP", adsl = FALSE) # BDS only
build_domain_spec <- function(domain = c("PP", "SUPPPP", "ADPP"),
ig_version = NULL, ct_version = NULL,
adsl = TRUE) {
domain <- match.arg(domain)
if (domain != "ADPP" && !isTRUE(adsl)) {
.cdiscdata_warn(
"`adsl` is ignored for domain != \"ADPP\".",
"adsl_ignored"
)
}

if (domain %in% c("PP", "SUPPPP")) {
ig_domain <- if (domain == "PP") "PP" else "SUPPQUAL"
sdtmig_all <- get_ig("sdtm")
sdtmig_all <- sdtmig_all[sdtmig_all$source == "SDTMIG", ]
version <- .resolve_ig_version(ig_version, sdtmig_all$version, "SDTMIG")
sub <- sdtmig_all[sdtmig_all$domain == ig_domain & sdtmig_all$version == version, ]
if (nrow(sub) == 0L) {
.cdiscdata_abort(
paste0("No SDTMIG '", ig_domain, "' variables found for version '", version, "'."),
"no_ig_variables"
)
}
ct <- get_ct("sdtm", version = ct_version)
sub$source <- "SDTMIG"
codelist_id <- .lookup_codelist_id(sub$codelist, ct)
} else {
adamig_all <- get_ig("adam")
version <- .resolve_ig_version(ig_version, adamig_all$version, "ADaMIG")
bds <- adamig_all[adamig_all$dataset == "BDS" & adamig_all$version == version, ]
if (nrow(bds) == 0L) {
.cdiscdata_abort(
paste0("No ADaMIG BDS variables found for version '", version, "'."),
"no_ig_variables"
)
}
bds$source <- "BDS"
ct_adam <- get_ct("adam", version = ct_version)

if (isTRUE(adsl)) {
adsl_tbl <- adamig_all[adamig_all$dataset == "ADSL" & adamig_all$version == version, ]
# ADSL and BDS both define several shared base variables (STUDYID,
# USUBJID, ...); a real ADPP carries each once, so ADSL contributes
# only the variables BDS does not already define, keeping BDS's
# version (and its own Core) for anything in both.
adsl_tbl <- adsl_tbl[!adsl_tbl$variable %in% bds$variable, ]
adsl_tbl$source <- "ADSL"
adsl_tbl$core <- "Perm"
adsl_tbl$order <- max(bds$order) + seq_len(nrow(adsl_tbl))
sub <- rbind(bds, adsl_tbl)
# ADSL variables carried from SDTM DM (SEX, RACE, AGEU, ...) reuse
# SDTM's codelists, not ADaM's much smaller CT; ADaM CT is checked
# first (matching the BDS-only behaviour), SDTM CT second.
ct_sdtm <- get_ct("sdtm", version = ct_version)
codelist_id <- .lookup_codelist_id(sub$codelist, ct_adam)
fallback <- is.na(codelist_id) & !is.na(sub$codelist)
codelist_id[fallback] <- .lookup_codelist_id(sub$codelist[fallback], ct_sdtm)
} else {
sub <- bds
codelist_id <- .lookup_codelist_id(sub$codelist, ct_adam)
}
}

data.frame(
variable = sub$variable,
label = sub$label,
type = sub$type,
length = sub$length,
core = sub$core,
order = sub$order,
source = sub$source,
codelist_id = codelist_id,
stringsAsFactors = FALSE
)
}

# Resolve ig_version = NULL to the newest version present in `versions`
# (numeric comparison via package_version, so "1.10" would correctly rank
# above "1.9"); abort with the available list on an unknown explicit version.
.resolve_ig_version <- function(ig_version, versions, label) {
avail <- sort(unique(versions))
if (is.null(ig_version)) {
return(as.character(max(package_version(avail))))
}
if (!ig_version %in% avail) {
.cdiscdata_abort(
paste0(
"Version '", ig_version, "' is not available for ", label, ". ",
"Available versions: ", paste(avail, collapse = ", "), "."
),
"ig_version_unavailable"
)
}
ig_version
}

# Map codelist submission values (e.g. "PKPARMCD") to their CT C-code (e.g.
# "C85839") via the codelist header rows (term_code is NA for those).
# NA in `codelist_names` (no codelist referenced) and names not found in
# this CT version (e.g. a codelist introduced in a CT release older/newer
# than the one used) both resolve to NA, not an error: the caller decides
# whether an unresolved codelist id is acceptable.
.lookup_codelist_id <- function(codelist_names, ct) {
headers <- ct[is.na(ct$term_code), ]
headers <- headers[!duplicated(headers$codelist_name), ]
headers$codelist_code[match(codelist_names, headers$codelist_name)]
}
2 changes: 1 addition & 1 deletion R/cdiscdata_versions.R
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@
cdiscdata_versions <- function() {
cat_cols <- c("dataset", "type", "description", "latest", "n_versions",
"last_updated")
datasets_catalogue[, cat_cols]
.pkg_data("datasets_catalogue")[, cat_cols]
}
Loading
Loading