Repository navigation
Load all CDISC Library IG, model, CDASH, and QRS exports; add get_cdash and ADPP extensions - #5
Merged
Merged
Conversation
Bumps Version to 0.2.0.9000 and opens a development-version NEWS header after the 0.2.0 CRAN submission (release/cran-0.2.0). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
nca.reporter hit label-length failures writing XPT v5 from ig_sdtm/ig_adam.
Auditing the tables found a wider transcription problem than the labels
reported: hard-wrapped Word table cells left literal newlines in 62 ig_sdtm
and 271 ig_adam labels, 112 ig_sdtm and 7 ig_adam variable names had
trailing spaces ("--TESTCD " != "--TESTCD"), type values were padded
("Char "), and five section sub-headings in the ADaMIG 1.0 combined ADSL
table were transcribed as variables with no label.
read_ig_csv() now collapses whitespace in every identifying field, the
sub-heading rows are dropped, and the four labels still over 40 characters
use the abbreviated form. Checked against published documents, not from
memory:
- SDTMIG v3.3 (CDISC wiki, sdtmig_v3.3.pdf): section 4.2.1 states labels are
"up to 40 characters"; section 6.3.11.2 (Pharmacokinetics Parameters), PP
specification table, labels PPSTRESC "Character Result/Finding in Std
Format"; section 6.3.10.1 (Generic Morphology/Physiology Specification),
generic table, labels --TESTCD "Short Name of Measurement, Test or Exam".
Applied to SDTMIG 3.2 PPSTRESC and SDTM Model 1.4-1.6 --TESTCD.
- ADaMIG v1.2 draft (CDISC wiki, ADaMIG-compiled), section 3.3.4.1, Table
3.3.4.1.1: PBCHGCyN is published as "Percent Change to Baseline Category y
(N)" (41 characters), so the IG itself exceeds the limit; the sibling
PCHGCAyN is abbreviated "Percent Chg from Baseline Category y (N)". The
new "Percent Chg to Baseline Category y (N)" follows that convention and
is a derived label. The wiki copy is a draft, not the final v1.2.
Adds gates in test-data_integrity.R: no label is NA, contains a newline, or
exceeds 40 characters; no variable is NA, blank, or contains whitespace; no
type/core/codelist/role value has stray whitespace. Fixes live in data-raw
(utils_ig.R, build_ig_sdtm.R, build_ig_adam.R) and data/ is regenerated.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ig_sdtm now carries the SDTMIG PP domain and generic SUPP-- structure at
3.3 as well as 3.2, and get_ig() gains a domain argument so
get_ig("sdtm", version = "3.3", domain = "PP") works (for "adam" it matches
the dataset column). build_domain_spec("PP"/"SUPPPP") default to 3.3.
Checked against the published SDTMIG v3.3 (CDISC wiki PDF
wiki.cdisc.org/download/attachments/66274516/sdtmig_v3.3.pdf):
- section 6.3.11.2, PP specification table (printed pp.542-547/796): 24
variables in the order STUDYID .. PPSPEC, TAETORD, EPOCH, PPDTC, PPDY,
PPRFTDTC, PPSTINT, PPENINT. The table heading is stamped "Version 3.2" and
the revision history lists no PP change, so the table is unchanged between
3.2 and 3.3 and 3.3 reuses the 3.2 tables.
- section 8.4.1, SUPP-- specification (printed pp.763-765/796): the same ten
variables, labels, and types as 3.2.
- PPANMETH is NOT in the 3.3 PP table, and the published name is PPDTC, not
PPPDTC; TAETORD, not PTAETORD.
The Rsdtm 3.2 transcription had only 21 of the 24 PP rows (TAETORD, EPOCH
and PPDY dropped); data-raw/ig_source/sdtmig_3.2/PP-additions.csv adds them
in their published positions without touching the copied Rsdtm files. The
PDF is a cropped browser print, so it shows no Core column: Core "Perm" for
the three added rows is from the Certara PKSubmit SDTM_3.2 appendix
(secondary), and existing rows keep their Rsdtm Core.
NOT included: SDTMIG 3.4. CDISC's own copy is members-only and the only
direct PDF is a third-party mirror, so 3.4 is left until a source is
approved; PPANMETH and PPTPTREF are reported by the Certara appendix, not
the IG, as 3.4 additions.
Also corrects the get_ig() and NEWS claims that PPANMETH, EPOCH, PTAETORD
and a PPPDTC rename were missing only because of the 3.2 cap.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An ADPP built from PP carries PP's variables, which neither the ADaMIG BDS
nor ADSL tables define. build_domain_spec("ADPP", sdtm_domain = "PP") now
unions the SDTMIG PP domain's variables into the spec the same way
adsl = TRUE unions ADSL's: marked source = "SDTMIG", core forced to "Perm",
a variable BDS or ADSL already defines (STUDYID, USUBJID) keeping that
version, ordered after the existing rows. 22 variables are added. Their
codelist ids resolve against the SDTM CT because the ADaM CT has none of
PP's codelists (PPTESTCD C85839, PPTEST C85493, PPORRESU/PPSTRESU C85494,
PPSTAT C66789, PPSPEC C78734, EPOCH C99079). sdtmig_version picks the
SDTMIG version, default newest. The default is NULL, which leaves the
output identical (tested with expect_identical).
A value other than "PP" is a classed cdiscdata_error_sdtm_domain_unavailable;
sdtm_domain or sdtmig_version passed where they have no effect is a classed
cdiscdata_warning_sdtm_domain_ignored.
build_domain_spec() is split into small helpers (.spec_adpp,
.sdtmig_domain_rows, .append_unique), reusing the existing ADSL-union
logic rather than duplicating it, to stay under the lintr complexity limit.
Also corrects the NEWS whitespace bullet from the label-fix commit: the
default ADPP variable set is unchanged by the whitespace fix, and the
visible effect was seven padded ADSL names in the ADaMIG 1.1 output.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…xception Replaces the derived "Percent Chg to Baseline Category y (N)" override with the label the guide publishes, "Percent Change to Baseline Category y (N)" (41 characters; ADaMIG v1.2 draft, CDISC wiki, section 3.3.4.1, Table 3.3.4.1.1). Decision (Bill): shortening a label to fit an XPT file is the dataset writer's job, not cdiscdata's. The 40-character gate in test-data_integrity.R now has an explicit exact-match allow-list naming the table, dataset, version, variable, label and reason, so any other over-length label, or this one changing, still fails. A second test pins that exactly one ig_adam label and no ig_sdtm label is over 40 characters, so the exception cannot go stale unnoticed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
R's NEWS.md parser (tools:::.build_news_db_from_package_NEWS_md) drops headings until the first one containing a version number, and the development header "(development version)" has none, so a level-2 heading "SDTMIG 3.3 ..." became the version level and R CMD check reported "Cannot extract version info from the following section titles". Renames that heading (and one ending in a parenthesis) so the parser reads the development section normally. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ig_sdtm now carries all 63 domains of SDTMIG 3.4 (1917 variables), so
get_ig("sdtm", version = "3.4", domain = "PP") works. build_domain_spec
("PP"/"SUPPPP") and build_domain_spec("ADPP", sdtm_domain = "PP") default to
3.4. The 3.4 PP table has 26 variables: the 24 of 3.2/3.3 plus PPANMETH
(Analysis Method, codelist PKANMET) and PPTPTREF (Time Point Reference), both
Permissible, confirming the earlier vendor-page report against the primary
source. The default PP spec grows from 24 to 26 variables; the sdtm_domain
union adds 24 variables (22 with sdtmig_version = "3.3").
Source: SDTMIG_v3.4.csv, a CDISC Library export downloaded by Bill under
CDISC's own terms. It is NOT in this repository and is never committed:
data-raw/build_ig_sdtm.R reads it from CDISC_SOURCES_DIR (default
../cdisc-sources) and stops with a message naming the expected file when it
is absent, so the build is reproducible by anyone who obtains the export under
their own CDISC terms. Only variable metadata is transcribed (order, class,
domain, name, label, type, role, Core, codelist). The CDISC Notes column is not
carried into the package data (notes is NA for the 3.4 rows) and is read only
to recover a stated maximum length as an integer; a test pins that notes is
empty.
The export gives codelists as CDISC CT C-codes only (Codelist Submission
Values is empty throughout), so each first code is mapped to its submission
value from the stored CT, using the most recent header row that carries it
because 4 of the 135 distinct first codes are for codelists since retired from
the CT; a code with no header anywhere stops the build.
The existing gates (labels non-NA, newline-free, at most 40 characters; names
non-blank and whitespace-free; no stray whitespace in type/core/codelist/
role) pass on all 1917 rows with no problems found. New tests pin the 3.4 PP
table in published order, the domain count, per-domain order 1..n, no repeated
variable per domain, and the 3.3/3.4 ADPP unions.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ables ig_sdtm, model_sdtm and ig_adam now carry every standard and version in the 34 CDISC Library CSV exports (SDTM 1.2-2.1, SDTMIG 3.1.2-3.4, SDTMIG-AP/MD, SENDIG, SENDIG-AR/DART/GeneTox, ADaMIG 1.0-1.3, and the ADaMIG-MD, NCA, ADAE, BDS-TTE, OCCDS and popPK guides), replacing every Rsdtm-derived row. Only metadata fields are carried; the guides' prose columns are dropped. get_ig() takes a standard, a version (newest by default) and a domain. build_domain_spec() takes a standard and an extension (ADaMIG-NCA, popPK or BDS-TTE) that is unioned onto BDS like adsl = TRUE. get_dataset() now reaches the IG and model tables, which previously returned NULL. The exports are read from CDISC_SOURCES_DIR and never committed; data-raw/ig_source and its parsers are removed. Integrity gates cover row counts, unique keys, label length (22 published exceptions, listed), types, cores, whitespace, and that no CSV is tracked under data-raw. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
cdash_model (CDASH 1.0-1.3, 1,138 rows), ig_cdash (CDASHIG 1.1 and 2.0-2.3, 4,483 rows) and qrs_supplement (nine instruments, 75 items) are built from 18 more CDISC Library exports read from CDISC_SOURCES_DIR and never committed. Only metadata and the CDASH collection wording are carried; the definitions, mapping and CRF completion instructions, implementation notes, and QRS item text are not. The QRS exports have no Version column, so the instrument and version are parsed from the file name, and build_ig.R stops on any *_Supplement_v*.csv it does not list. get_cdash() mirrors get_ig() for the two CDASH tables; get_dataset() and list_datasets() reach all three. ig_sources now lists all 52 exports. Gates: key uniqueness, row counts per file, label length with an exhaustive exception list (6 in the model, 71 in CDASHIG 2.0; CDASHIG 1.1 has no labels), prose-length caps, and tests of the data-raw helpers on synthetic exports including that no prose column of any layout is carried. build_domain_spec's ADPP path is split into helpers to stay under the complexity limit. Docs: NEWS rewritten for the whole redesign (including the corrected SDTMIG 3.2 PP count), README tables, and data-raw/README.md on obtaining the exports. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the Rsdtm-derived IG tables with 52 CDISC Library CSV exports (read from CDISC_SOURCES_DIR, never committed): ig_sdtm 10,064 rows, model_sdtm 3,604, ig_adam 1,804, cdash_model 1,138, ig_cdash 4,483, qrs_supplement 75, plus ig_sources provenance. Breaking: new columns on ig_sdtm and ig_adam, and get_ig(standard, version, domain) now defaults to the newest SDTMIG. New get_cdash(); build_domain_spec gains standard and extension (ADaMIG-NCA, popPK, BDS-TTE); get_dataset() now returns the IG tables (it returned NULL before). Integrity gates cover row counts, unique keys, label length with exhaustive published-exception lists, no prose columns, and no CSV under data-raw. Corrects the earlier SDTMIG 3.2 PP count (21, not 24). 100% coverage, lint clean, check clean. Merge with a merge commit, not a squash.
🤖 Generated with Claude Code