Skip to content

Load all CDISC Library IG, model, CDASH, and QRS exports; add get_cdash and ADPP extensions - #5

Merged
billdenney merged 10 commits into
mainfrom
fix/ig-labels-and-sdtmig-pp
Oct 1, 2026
Merged

billdenney merged 10 commits into
mainfrom
fix/ig-labels-and-sdtmig-pp

Conversation

@billdenney

Copy link
Copy Markdown
Member

Replaces the Rsdtm-derived IG tables with 52 CDISC Library CSV exports (read from CDISC_SOURCES_DIR, never committed): ig_sdtm 10,064 rows, model_sdtm 3,604, ig_adam 1,804, cdash_model 1,138, ig_cdash 4,483, qrs_supplement 75, plus ig_sources provenance. Breaking: new columns on ig_sdtm and ig_adam, and get_ig(standard, version, domain) now defaults to the newest SDTMIG. New get_cdash(); build_domain_spec gains standard and extension (ADaMIG-NCA, popPK, BDS-TTE); get_dataset() now returns the IG tables (it returned NULL before). Integrity gates cover row counts, unique keys, label length with exhaustive published-exception lists, no prose columns, and no CSV under data-raw. Corrects the earlier SDTMIG 3.2 PP count (21, not 24). 100% coverage, lint clean, check clean. Merge with a merge commit, not a squash.

🤖 Generated with Claude Code

billdenney and others added 10 commits September 30, 2026 20:59
Bumps Version to 0.2.0.9000 and opens a development-version NEWS header
after the 0.2.0 CRAN submission (release/cran-0.2.0).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
nca.reporter hit label-length failures writing XPT v5 from ig_sdtm/ig_adam.
Auditing the tables found a wider transcription problem than the labels
reported: hard-wrapped Word table cells left literal newlines in 62 ig_sdtm
and 271 ig_adam labels, 112 ig_sdtm and 7 ig_adam variable names had
trailing spaces ("--TESTCD " != "--TESTCD"), type values were padded
("Char "), and five section sub-headings in the ADaMIG 1.0 combined ADSL
table were transcribed as variables with no label.

read_ig_csv() now collapses whitespace in every identifying field, the
sub-heading rows are dropped, and the four labels still over 40 characters
use the abbreviated form. Checked against published documents, not from
memory:

- SDTMIG v3.3 (CDISC wiki, sdtmig_v3.3.pdf): section 4.2.1 states labels are
  "up to 40 characters"; section 6.3.11.2 (Pharmacokinetics Parameters), PP
  specification table, labels PPSTRESC "Character Result/Finding in Std
  Format"; section 6.3.10.1 (Generic Morphology/Physiology Specification),
  generic table, labels --TESTCD "Short Name of Measurement, Test or Exam".
  Applied to SDTMIG 3.2 PPSTRESC and SDTM Model 1.4-1.6 --TESTCD.
- ADaMIG v1.2 draft (CDISC wiki, ADaMIG-compiled), section 3.3.4.1, Table
  3.3.4.1.1: PBCHGCyN is published as "Percent Change to Baseline Category y
  (N)" (41 characters), so the IG itself exceeds the limit; the sibling
  PCHGCAyN is abbreviated "Percent Chg from Baseline Category y (N)". The
  new "Percent Chg to Baseline Category y (N)" follows that convention and
  is a derived label. The wiki copy is a draft, not the final v1.2.

Adds gates in test-data_integrity.R: no label is NA, contains a newline, or
exceeds 40 characters; no variable is NA, blank, or contains whitespace; no
type/core/codelist/role value has stray whitespace. Fixes live in data-raw
(utils_ig.R, build_ig_sdtm.R, build_ig_adam.R) and data/ is regenerated.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ig_sdtm now carries the SDTMIG PP domain and generic SUPP-- structure at
3.3 as well as 3.2, and get_ig() gains a domain argument so
get_ig("sdtm", version = "3.3", domain = "PP") works (for "adam" it matches
the dataset column). build_domain_spec("PP"/"SUPPPP") default to 3.3.

Checked against the published SDTMIG v3.3 (CDISC wiki PDF
wiki.cdisc.org/download/attachments/66274516/sdtmig_v3.3.pdf):
- section 6.3.11.2, PP specification table (printed pp.542-547/796): 24
  variables in the order STUDYID .. PPSPEC, TAETORD, EPOCH, PPDTC, PPDY,
  PPRFTDTC, PPSTINT, PPENINT. The table heading is stamped "Version 3.2" and
  the revision history lists no PP change, so the table is unchanged between
  3.2 and 3.3 and 3.3 reuses the 3.2 tables.
- section 8.4.1, SUPP-- specification (printed pp.763-765/796): the same ten
  variables, labels, and types as 3.2.
- PPANMETH is NOT in the 3.3 PP table, and the published name is PPDTC, not
  PPPDTC; TAETORD, not PTAETORD.

The Rsdtm 3.2 transcription had only 21 of the 24 PP rows (TAETORD, EPOCH
and PPDY dropped); data-raw/ig_source/sdtmig_3.2/PP-additions.csv adds them
in their published positions without touching the copied Rsdtm files. The
PDF is a cropped browser print, so it shows no Core column: Core "Perm" for
the three added rows is from the Certara PKSubmit SDTM_3.2 appendix
(secondary), and existing rows keep their Rsdtm Core.

NOT included: SDTMIG 3.4. CDISC's own copy is members-only and the only
direct PDF is a third-party mirror, so 3.4 is left until a source is
approved; PPANMETH and PPTPTREF are reported by the Certara appendix, not
the IG, as 3.4 additions.

Also corrects the get_ig() and NEWS claims that PPANMETH, EPOCH, PTAETORD
and a PPPDTC rename were missing only because of the 3.2 cap.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An ADPP built from PP carries PP's variables, which neither the ADaMIG BDS
nor ADSL tables define. build_domain_spec("ADPP", sdtm_domain = "PP") now
unions the SDTMIG PP domain's variables into the spec the same way
adsl = TRUE unions ADSL's: marked source = "SDTMIG", core forced to "Perm",
a variable BDS or ADSL already defines (STUDYID, USUBJID) keeping that
version, ordered after the existing rows. 22 variables are added. Their
codelist ids resolve against the SDTM CT because the ADaM CT has none of
PP's codelists (PPTESTCD C85839, PPTEST C85493, PPORRESU/PPSTRESU C85494,
PPSTAT C66789, PPSPEC C78734, EPOCH C99079). sdtmig_version picks the
SDTMIG version, default newest. The default is NULL, which leaves the
output identical (tested with expect_identical).

A value other than "PP" is a classed cdiscdata_error_sdtm_domain_unavailable;
sdtm_domain or sdtmig_version passed where they have no effect is a classed
cdiscdata_warning_sdtm_domain_ignored.

build_domain_spec() is split into small helpers (.spec_adpp,
.sdtmig_domain_rows, .append_unique), reusing the existing ADSL-union
logic rather than duplicating it, to stay under the lintr complexity limit.

Also corrects the NEWS whitespace bullet from the label-fix commit: the
default ADPP variable set is unchanged by the whitespace fix, and the
visible effect was seven padded ADSL names in the ADaMIG 1.1 output.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…xception

Replaces the derived "Percent Chg to Baseline Category y (N)" override with
the label the guide publishes, "Percent Change to Baseline Category y (N)"
(41 characters; ADaMIG v1.2 draft, CDISC wiki, section 3.3.4.1, Table
3.3.4.1.1). Decision (Bill): shortening a label to fit an XPT file is the
dataset writer's job, not cdiscdata's.

The 40-character gate in test-data_integrity.R now has an explicit
exact-match allow-list naming the table, dataset, version, variable, label
and reason, so any other over-length label, or this one changing, still
fails. A second test pins that exactly one ig_adam label and no ig_sdtm label
is over 40 characters, so the exception cannot go stale unnoticed.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
R's NEWS.md parser (tools:::.build_news_db_from_package_NEWS_md) drops
headings until the first one containing a version number, and the
development header "(development version)" has none, so a level-2 heading
"SDTMIG 3.3 ..." became the version level and R CMD check reported "Cannot
extract version info from the following section titles". Renames that
heading (and one ending in a parenthesis) so the parser reads the
development section normally.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ig_sdtm now carries all 63 domains of SDTMIG 3.4 (1917 variables), so
get_ig("sdtm", version = "3.4", domain = "PP") works. build_domain_spec
("PP"/"SUPPPP") and build_domain_spec("ADPP", sdtm_domain = "PP") default to
3.4. The 3.4 PP table has 26 variables: the 24 of 3.2/3.3 plus PPANMETH
(Analysis Method, codelist PKANMET) and PPTPTREF (Time Point Reference), both
Permissible, confirming the earlier vendor-page report against the primary
source. The default PP spec grows from 24 to 26 variables; the sdtm_domain
union adds 24 variables (22 with sdtmig_version = "3.3").

Source: SDTMIG_v3.4.csv, a CDISC Library export downloaded by Bill under
CDISC's own terms. It is NOT in this repository and is never committed:
data-raw/build_ig_sdtm.R reads it from CDISC_SOURCES_DIR (default
../cdisc-sources) and stops with a message naming the expected file when it
is absent, so the build is reproducible by anyone who obtains the export under
their own CDISC terms. Only variable metadata is transcribed (order, class,
domain, name, label, type, role, Core, codelist). The CDISC Notes column is not
carried into the package data (notes is NA for the 3.4 rows) and is read only
to recover a stated maximum length as an integer; a test pins that notes is
empty.

The export gives codelists as CDISC CT C-codes only (Codelist Submission
Values is empty throughout), so each first code is mapped to its submission
value from the stored CT, using the most recent header row that carries it
because 4 of the 135 distinct first codes are for codelists since retired from
the CT; a code with no header anywhere stops the build.

The existing gates (labels non-NA, newline-free, at most 40 characters; names
non-blank and whitespace-free; no stray whitespace in type/core/codelist/
role) pass on all 1917 rows with no problems found. New tests pin the 3.4 PP
table in published order, the domain count, per-domain order 1..n, no repeated
variable per domain, and the 3.3/3.4 ADPP unions.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ables

ig_sdtm, model_sdtm and ig_adam now carry every standard and version in the
34 CDISC Library CSV exports (SDTM 1.2-2.1, SDTMIG 3.1.2-3.4, SDTMIG-AP/MD,
SENDIG, SENDIG-AR/DART/GeneTox, ADaMIG 1.0-1.3, and the ADaMIG-MD, NCA, ADAE,
BDS-TTE, OCCDS and popPK guides), replacing every Rsdtm-derived row. Only
metadata fields are carried; the guides' prose columns are dropped.

get_ig() takes a standard, a version (newest by default) and a domain.
build_domain_spec() takes a standard and an extension (ADaMIG-NCA, popPK or
BDS-TTE) that is unioned onto BDS like adsl = TRUE. get_dataset() now reaches
the IG and model tables, which previously returned NULL.

The exports are read from CDISC_SOURCES_DIR and never committed;
data-raw/ig_source and its parsers are removed. Integrity gates cover row
counts, unique keys, label length (22 published exceptions, listed), types,
cores, whitespace, and that no CSV is tracked under data-raw.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
cdash_model (CDASH 1.0-1.3, 1,138 rows), ig_cdash (CDASHIG 1.1 and 2.0-2.3,
4,483 rows) and qrs_supplement (nine instruments, 75 items) are built from
18 more CDISC Library exports read from CDISC_SOURCES_DIR and never
committed. Only metadata and the CDASH collection wording are carried; the
definitions, mapping and CRF completion instructions, implementation notes,
and QRS item text are not. The QRS exports have no Version column, so the
instrument and version are parsed from the file name, and build_ig.R stops on
any *_Supplement_v*.csv it does not list.

get_cdash() mirrors get_ig() for the two CDASH tables; get_dataset() and
list_datasets() reach all three. ig_sources now lists all 52 exports.

Gates: key uniqueness, row counts per file, label length with an exhaustive
exception list (6 in the model, 71 in CDASHIG 2.0; CDASHIG 1.1 has no labels),
prose-length caps, and tests of the data-raw helpers on synthetic exports
including that no prose column of any layout is carried. build_domain_spec's
ADPP path is split into helpers to stay under the complexity limit.

Docs: NEWS rewritten for the whole redesign (including the corrected SDTMIG
3.2 PP count), README tables, and data-raw/README.md on obtaining the exports.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Oct 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
R/build_domain_spec.R 100.00% <100.00%> (ø)
R/get_cdash.R 100.00% <100.00%> (ø)
R/get_dataset.R 100.00% <100.00%> (ø)
R/get_ig.R 100.00% <100.00%> (ø)
R/ig_versions.R 100.00% <100.00%> (ø)
R/utils.R 100.00% <ø> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@billdenney
billdenney merged commit 1dc56a3 into main Oct 1, 2026
9 checks passed
@billdenney
billdenney deleted the fix/ig-labels-and-sdtmig-pp branch October 1, 2026 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant