From c3ed65501c8518583dd45e6e3be920ba990e06dd Mon Sep 17 00:00:00 2001 From: Alex Kholodniak Date: Sun, 11 Oct 2026 07:40:37 +0300 Subject: [PATCH 1/3] feat: add ItemBank for storing and reusing calibrated items Fit results are positional arrays tied to one in-memory model, so an app that wants to work with new respondents has to recalibrate or invent its own storage. ItemBank is an immutable, id-keyed set of item parameters with a versioned JSON format, per-model validation, recorded calibration provenance, and response probabilities, so a bank calibrated once (here or in R's mirt) can be loaded anywhere. It is the base for scoring new respondents against fixed items. --- CHANGELOG.md | 5 + README.md | 55 +++++++++ lib/irt_ruby.rb | 1 + lib/irt_ruby/item_bank.rb | 195 ++++++++++++++++++++++++++++++ spec/irt_ruby/item_bank_spec.rb | 204 ++++++++++++++++++++++++++++++++ spec/readme_examples_spec.rb | 13 ++ 6 files changed, 473 insertions(+) create mode 100644 lib/irt_ruby/item_bank.rb create mode 100644 spec/irt_ruby/item_bank_spec.rb diff --git a/CHANGELOG.md b/CHANGELOG.md index 57dd08a..4d1f345 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ All notable changes to this project are documented in this file. +## [Unreleased] + +### Added +- Added `IrtRuby::ItemBank`, an immutable set of calibrated items with stable string ids. Banks are built from fit results with `ItemBank.from_fit` (recording calibration provenance), saved and loaded as versioned JSON (`to_json`, `ItemBank.from_json`), validated per model, and expose item lookup and response probabilities. The README documents the format and an export recipe for R's mirt. + ## [0.4.0] - 2026-10-11 ### Fixed diff --git a/README.md b/README.md index 3a7c7e3..6818a67 100644 --- a/README.md +++ b/README.md @@ -166,6 +166,61 @@ All models estimate person and item parameters jointly by maximizing the likelih - **Scale identification.** Results are reported on a fixed scale: Rasch abilities are centered to mean 0; 2PL and 3PL abilities are standardized to mean 0 and standard deviation 1, with difficulties and discriminations transformed so that response probabilities are unchanged. Only estimated respondents enter these statistics. The model keeps its internal parameters, so calling `fit` again continues from where it stopped; convergence is checked on the reported scale. - **Known limitations.** JMLE item estimates are biased when tests are short, and the bias does not vanish as the number of respondents grows. 2PL and 3PL JMLE is also slow to converge with plain gradient ascent. For high-stakes calibration, compare against marginal maximum likelihood software such as R's `mirt`. +## Item Banks + +An `IrtRuby::ItemBank` is an immutable set of calibrated items with stable string ids. Calibrate once, save the bank as JSON, and load it wherever you need the item parameters, for example inside a Rails app, without refitting. + +```ruby +results = IrtRuby::TwoParameterModel.new(responses, seed: 1).fit +bank = IrtRuby::ItemBank.from_fit(results, item_ids: %w[q1 q2 q3 q4], metadata: { source: "spring pilot" }) + +File.write("item_bank.json", bank.to_json) +bank = IrtRuby::ItemBank.from_json(File.read("item_bank.json")) + +bank.fetch("q2") # IrtRuby::ItemBank::Item with id, a, b, c +bank.probability("q2", 0.5) # probability of a correct answer at theta = 0.5 +``` + +`from_fit` infers the model from the result keys and records provenance in `metadata`: the irt_ruby version, the scale convention, and the `converged` and `iterations` values of the fit. Items whose estimates are `nil` raise an error; pass `exclude_unestimated: true` to leave them out and list their ids under `metadata["unestimated_items"]`. + +The JSON format is: + +```json +{ + "format": "irt_ruby.item_bank", + "format_version": 1, + "model": "2pl", + "metadata": { "software": "irt_ruby 0.4.0", "converged": true }, + "items": [ + { "id": "q1", "a": 0.87, "b": 1.12, "c": 0.0 } + ] +} +``` + +Parameters use the logistic metric without a scaling constant (no `D = 1.7`): `P(correct | theta) = c + (1 - c) / (1 + exp(-a * (theta - b)))`. `model` is `"rasch"` (`a` = 1, `c` = 0), `"2pl"` (`c` = 0), or `"3pl"` (0 <= `c` < 1); for Rasch and 2PL the fixed parameters may be omitted. Banks with a different `format` or a newer `format_version` are rejected. + +### Importing from R's mirt + +mirt uses the same logistic metric. Its traditional IRT parameters can be written as an item bank with `jsonlite`: + +```r +library(mirt) +library(jsonlite) + +mod <- mirt(responses, 1, itemtype = "2PL") +pars <- coef(mod, IRTpars = TRUE, simplify = TRUE)$items # columns a, b, g, u +stopifnot(all(pars[, "u"] == 1)) + +bank <- list( + format = "irt_ruby.item_bank", format_version = 1, model = "2pl", + metadata = list(software = paste("mirt", packageVersion("mirt")), estimation = "marginal maximum likelihood"), + items = lapply(rownames(pars), function(id) list(id = id, a = pars[id, "a"], b = pars[id, "b"], c = pars[id, "g"])) +) +writeLines(toJSON(bank, auto_unbox = TRUE, digits = NA), "item_bank.json") +``` + +Use `model = "3pl"` for `itemtype = "3PL"`, and `model = "rasch"` for `itemtype = "Rasch"`. Note the scale conventions differ: mirt fixes the ability distribution to mean 0 and SD 1 in the population, while irt_ruby standardizes the estimated abilities of the calibration sample. + ## Performance Benchmarks IRT Ruby includes comprehensive performance benchmarks to help you understand the computational characteristics of different models: diff --git a/lib/irt_ruby.rb b/lib/irt_ruby.rb index 88ff502..ae41a57 100644 --- a/lib/irt_ruby.rb +++ b/lib/irt_ruby.rb @@ -9,6 +9,7 @@ require "irt_ruby/rasch_model" require "irt_ruby/two_parameter_model" require "irt_ruby/three_parameter_model" +require "irt_ruby/item_bank" module IrtRuby class Error < StandardError; end diff --git a/lib/irt_ruby/item_bank.rb b/lib/irt_ruby/item_bank.rb new file mode 100644 index 0000000..e8d265c --- /dev/null +++ b/lib/irt_ruby/item_bank.rb @@ -0,0 +1,195 @@ +# frozen_string_literal: true + +require "json" + +module IrtRuby + # An immutable set of calibrated items that can be stored as JSON and reused + # without recalibrating. Parameters use the logistic metric with no scaling + # constant: P(correct | theta) = c + (1 - c) / (1 + exp(-a * (theta - b))). + class ItemBank + include Enumerable + + FORMAT = "irt_ruby.item_bank" + FORMAT_VERSION = 1 + MODELS = %w[rasch 2pl 3pl].freeze + + Item = Struct.new(:id, :a, :b, :c) + + attr_reader :model, :metadata + + # Builds a bank from a model's fit results. Items whose parameters are nil + # raise unless exclude_unestimated is true, in which case their ids are + # recorded in the metadata under "unestimated_items". + def self.from_fit(results, item_ids:, metadata: {}, exclude_unestimated: false) + model = if results.key?(:guessings) + "3pl" + elsif results.key?(:discriminations) + "2pl" + else + "rasch" + end + difficulties = results.fetch(:difficulties) + raise ArgumentError, "item_ids must be an Array with one id per item (#{difficulties.size})" unless item_ids.is_a?(Array) && item_ids.size == difficulties.size + + unestimated = item_ids.select.with_index { |_, j| difficulties[j].nil? } + raise ArgumentError, "items without estimates: #{unestimated.join(", ")}; pass exclude_unestimated: true to leave them out" unless unestimated.empty? || exclude_unestimated + + items = item_ids.each_index.reject { |j| difficulties[j].nil? }.map do |j| + { id: item_ids[j], a: results.fetch(:discriminations, nil)&.at(j) || 1.0, + b: difficulties[j], c: results.fetch(:guessings, nil)&.at(j) || 0.0 } + end + + provenance = { + "software" => "irt_ruby #{VERSION}", + "estimation" => "joint maximum likelihood", + "scale" => model == "rasch" ? "abilities centered at 0" : "abilities standardized to mean 0, SD 1", + "converged" => results[:converged], + "iterations" => results[:iterations] + } + provenance["unestimated_items"] = unestimated unless unestimated.empty? + + new(model: model, items: items, metadata: provenance.merge(metadata.transform_keys(&:to_s))) + end + + def self.from_json(json) + from_h(JSON.parse(json)) + end + + def self.from_h(hash) + raise ArgumentError, "item bank must be a Hash" unless hash.is_a?(Hash) + raise ArgumentError, "not an item bank: format must be #{FORMAT.inspect}" unless hash["format"] == FORMAT + + version = hash["format_version"] + raise ArgumentError, "unsupported item bank format_version #{version.inspect}; this irt_ruby reads #{FORMAT_VERSION}" unless version == FORMAT_VERSION + + new(model: hash["model"], items: hash["items"], metadata: hash.fetch("metadata", {})) + end + + # items: an Array of Hashes with an "id" (String) and "b", plus "a" (2PL/3PL) + # and "c" (3PL); string or symbol keys. metadata must be JSON-serializable. + def initialize(model:, items:, metadata: {}) + raise ArgumentError, "model must be one of #{MODELS}" unless MODELS.include?(model) + raise ArgumentError, "items must be a non-empty Array" unless items.is_a?(Array) && !items.empty? + raise ArgumentError, "metadata must be a Hash" unless metadata.is_a?(Hash) + + @model = model + @items = items.each_with_index.map { |item, index| build_item(item, index) }.freeze + @index = @items.each_with_index.to_h { |item, k| [item.id, k] }.freeze + raise ArgumentError, "item ids must be unique" unless @index.size == @items.size + + @metadata = deep_freeze(normalize_metadata(metadata)) + freeze + end + + def each(&block) + return enum_for(:each) unless block + + @items.each(&block) + self + end + + def size + @items.size + end + + def ids + @items.map(&:id) + end + + def [](id) + position = @index[id] + position && @items[position] + end + + def fetch(id) + self[id] || raise(KeyError, "unknown item id #{id.inspect}") + end + + def probability(id, theta) + item = fetch(id) + item.c + ((1.0 - item.c) / (1.0 + Math.exp(-item.a * (theta - item.b)))) + end + + def to_h + { + "format" => FORMAT, + "format_version" => FORMAT_VERSION, + "model" => @model, + "metadata" => @metadata, + "items" => @items.map { |item| { "id" => item.id, "a" => item.a, "b" => item.b, "c" => item.c } } + } + end + + def to_json(*args) + to_h.to_json(*args) + end + + def ==(other) + other.is_a?(ItemBank) && to_h == other.to_h + end + alias eql? == + + def hash + to_h.hash + end + + private + + def build_item(item, index) + raise ArgumentError, "item #{index + 1} must be a Hash" unless item.is_a?(Hash) + + id = value(item, :id) + raise ArgumentError, "item #{index + 1} id must be a non-empty String" unless id.is_a?(String) && !id.empty? + + b = number(item, :b, id, required: true) + a = number(item, :a, id, required: @model != "rasch") + c = number(item, :c, id, required: @model == "3pl") + a = validate_discrimination(a, id) + c = validate_guessing(c, id) + + Item.new(id.dup.freeze, a, b, c).freeze + end + + def validate_discrimination(a, id) + return 1.0 if a.nil? + raise ArgumentError, "item #{id.inspect}: rasch items must have a = 1" if @model == "rasch" && (a - 1.0).abs > 1e-9 + raise ArgumentError, "item #{id.inspect}: a must be positive" unless a.positive? + + a + end + + def validate_guessing(c, id) + return 0.0 if c.nil? + raise ArgumentError, "item #{id.inspect}: #{@model} items must have c = 0" if @model != "3pl" && !c.zero? + raise ArgumentError, "item #{id.inspect}: c must be in [0, 1)" unless c >= 0 && c < 1 + + c + end + + def number(item, key, id, required:) + raw = value(item, key) + return nil if raw.nil? && !required + raise ArgumentError, "item #{id.inspect}: #{key} must be a finite number" unless raw.is_a?(Numeric) && !raw.is_a?(Complex) && raw.finite? + + raw.to_f + end + + def value(item, key) + item.key?(key) ? item[key] : item[key.to_s] + end + + def normalize_metadata(metadata) + JSON.parse(JSON.generate(metadata)) + rescue JSON::GeneratorError => e + raise ArgumentError, "metadata must be JSON-serializable: #{e.message}" + end + + def deep_freeze(object) + case object + when Hash then object.each_value { |v| deep_freeze(v) } + when Array then object.each { |v| deep_freeze(v) } + end + object.freeze + end + end +end diff --git a/spec/irt_ruby/item_bank_spec.rb b/spec/irt_ruby/item_bank_spec.rb new file mode 100644 index 0000000..1532e1b --- /dev/null +++ b/spec/irt_ruby/item_bank_spec.rb @@ -0,0 +1,204 @@ +# frozen_string_literal: true + +require "spec_helper" + +RSpec.describe IrtRuby::ItemBank do + let(:data) do + rng = Random.new(3) + Array.new(60) { Array.new(4) { rng.rand < 0.55 ? 1 : 0 } } + end + let(:item_ids) { %w[q1 q2 q3 q4] } + + let(:items_3pl) do + [ + { "id" => "q1", "a" => 1.2, "b" => -0.5, "c" => 0.2 }, + { "id" => "q2", "a" => 0.8, "b" => 0.75, "c" => 0.0 } + ] + end + + describe ".from_fit" do + { + IrtRuby::RaschModel => "rasch", + IrtRuby::TwoParameterModel => "2pl", + IrtRuby::ThreeParameterModel => "3pl" + }.each do |model_class, model_name| + it "builds a #{model_name} bank keyed by item id" do + results = model_class.new(data, max_iter: 50, seed: 1).fit + bank = described_class.from_fit(results, item_ids: item_ids) + + expect(bank.model).to eq(model_name) + expect(bank.ids).to eq(item_ids) + item_ids.each_with_index do |id, j| + item = bank.fetch(id) + expect(item.b).to eq(results[:difficulties][j]) + expect(item.a).to eq(results.fetch(:discriminations, Array.new(4, 1.0))[j]) + expect(item.c).to eq(results.fetch(:guessings, Array.new(4, 0.0))[j]) + end + end + end + + it "records calibration provenance and merges caller metadata" do + results = IrtRuby::RaschModel.new(data, max_iter: 5, seed: 1).fit + bank = described_class.from_fit(results, item_ids: item_ids, metadata: { source: "pilot 2026", software: "custom" }) + + expect(bank.metadata).to include( + "software" => "custom", + "estimation" => "joint maximum likelihood", + "scale" => "abilities centered at 0", + "converged" => results[:converged], + "iterations" => 5, + "source" => "pilot 2026" + ) + end + + it "rejects items without estimates unless asked to exclude them" do + results = IrtRuby::RaschModel.new([[1, 0, 1], [0, 1, 1]], seed: 1).fit + + expect { described_class.from_fit(results, item_ids: %w[a b c]) } + .to raise_error(ArgumentError, /items without estimates: c/) + + bank = described_class.from_fit(results, item_ids: %w[a b c], exclude_unestimated: true) + expect(bank.ids).to eq(%w[a b]) + expect(bank.metadata["unestimated_items"]).to eq(%w[c]) + end + + it "requires one id per item" do + results = IrtRuby::RaschModel.new(data, max_iter: 5, seed: 1).fit + + expect { described_class.from_fit(results, item_ids: %w[q1 q2]) }.to raise_error(ArgumentError, /one id per item \(4\)/) + end + end + + describe "JSON" do + it "round-trips every model exactly" do + [IrtRuby::RaschModel, IrtRuby::TwoParameterModel, IrtRuby::ThreeParameterModel].each do |model_class| + bank = described_class.from_fit(model_class.new(data, max_iter: 50, seed: 1).fit, item_ids: item_ids) + restored = described_class.from_json(bank.to_json) + + expect(restored).to eq(bank) + expect(restored.map(&:b)).to eq(bank.map(&:b)) + end + end + + it "writes the format header, model, metadata, and items" do + bank = described_class.new(model: "3pl", items: items_3pl, metadata: { "source" => "test" }) + + expect(JSON.parse(JSON.generate(bank))).to eq( + "format" => "irt_ruby.item_bank", + "format_version" => 1, + "model" => "3pl", + "metadata" => { "source" => "test" }, + "items" => items_3pl + ) + end + + it "fills in fixed parameters for rasch and 2pl items" do + bank = described_class.from_json( + '{"format":"irt_ruby.item_bank","format_version":1,"model":"rasch","items":[{"id":"x","b":0.5}]}' + ) + + expect(bank.fetch("x").to_a).to eq(["x", 1.0, 0.5, 0.0]) + expect(bank.metadata).to eq({}) + end + + it "rejects documents that are not item banks" do + expect { described_class.from_json('{"format":"other","format_version":1}') } + .to raise_error(ArgumentError, /not an item bank/) + expect { described_class.from_json("[1]") }.to raise_error(ArgumentError, /must be a Hash/) + end + + it "rejects unsupported format versions" do + json = { "format" => "irt_ruby.item_bank", "format_version" => 2, "model" => "rasch", "items" => [] }.to_json + + expect { described_class.from_json(json) }.to raise_error(ArgumentError, /unsupported item bank format_version 2/) + end + end + + describe "validation" do + def build(model, items) + described_class.new(model: model, items: items) + end + + it "accepts symbol keys" do + expect(build("2pl", [{ id: "x", a: 1, b: 0 }]).fetch("x").to_a).to eq(["x", 1.0, 0.0, 0.0]) + end + + { + "an unknown model" => ["4pl", [{ "id" => "x", "b" => 0 }], /model must be one of/], + "an empty item list" => ["rasch", [], /non-empty Array/], + "a missing id" => ["rasch", [{ "b" => 0 }], /item 1 id must be a non-empty String/], + "a non-string id" => ["rasch", [{ "id" => 7, "b" => 0 }], /item 1 id must be a non-empty String/], + "duplicate ids" => ["rasch", [{ "id" => "x", "b" => 0 }, { "id" => "x", "b" => 1 }], /ids must be unique/], + "a missing difficulty" => ["rasch", [{ "id" => "x" }], /"x": b must be a finite number/], + "a non-numeric difficulty" => ["rasch", [{ "id" => "x", "b" => "0" }], /b must be a finite number/], + "an infinite difficulty" => ["rasch", [{ "id" => "x", "b" => Float::INFINITY }], /b must be a finite number/], + "a rasch discrimination other than 1" => ["rasch", [{ "id" => "x", "a" => 1.5, "b" => 0 }], /rasch items must have a = 1/], + "a missing 2pl discrimination" => ["2pl", [{ "id" => "x", "b" => 0 }], /a must be a finite number/], + "a non-positive discrimination" => ["2pl", [{ "id" => "x", "a" => 0, "b" => 0 }], /a must be positive/], + "a 2pl guessing parameter" => ["2pl", [{ "id" => "x", "a" => 1, "b" => 0, "c" => 0.1 }], /2pl items must have c = 0/], + "a missing 3pl guessing parameter" => ["3pl", [{ "id" => "x", "a" => 1, "b" => 0 }], /c must be a finite number/], + "a guessing parameter of 1" => ["3pl", [{ "id" => "x", "a" => 1, "b" => 0, "c" => 1 }], /c must be in \[0, 1\)/], + "a negative guessing parameter" => ["3pl", [{ "id" => "x", "a" => 1, "b" => 0, "c" => -0.1 }], /c must be in \[0, 1\)/] + }.each do |description, (model, items, message)| + it "rejects #{description}" do + expect { build(model, items) }.to raise_error(ArgumentError, message) + end + end + + it "rejects metadata that cannot be written as JSON" do + expect { described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], metadata: { "x" => Float::NAN }) } + .to raise_error(ArgumentError, /metadata must be JSON-serializable/) + end + end + + describe "immutability" do + it "is not affected by later changes to its inputs" do + items = [{ "id" => +"x", "a" => 1.0, "b" => 0.0, "c" => 0.1 }] + metadata = { "notes" => [+"first"] } + bank = described_class.new(model: "3pl", items: items, metadata: metadata) + items[0]["b"] = 5.0 + items[0]["id"] << "y" + metadata["notes"] << "second" + + expect(bank.fetch("x").b).to eq(0.0) + expect(bank.metadata).to eq("notes" => ["first"]) + end + + it "freezes the bank, its items, and its metadata" do + bank = described_class.new(model: "3pl", items: items_3pl, metadata: { "notes" => ["first"] }) + + expect(bank).to be_frozen + expect(bank.first).to be_frozen + expect(bank.first.id).to be_frozen + expect(bank.metadata["notes"]).to be_frozen + end + end + + describe "lookup and probabilities" do + let(:bank) { described_class.new(model: "3pl", items: items_3pl) } + + it "looks items up by id" do + expect(bank.size).to eq(2) + expect(bank["q2"].a).to eq(0.8) + expect(bank["missing"]).to be_nil + expect { bank.fetch("missing") }.to raise_error(KeyError, /unknown item id "missing"/) + end + + it "computes the logistic response probability without a scaling constant" do + expect(bank.probability("q1", 0.3)).to be_within(1e-15).of(0.2 + (0.8 / (1 + Math.exp(-1.2 * (0.3 - -0.5))))) + expect(bank.probability("q2", 0.75)).to eq(0.5) + end + + it "matches the probabilities of the model it was calibrated with" do + model = IrtRuby::ThreeParameterModel.new(data, max_iter: 50, seed: 1) + results = model.fit + bank = described_class.from_fit(results, item_ids: item_ids) + theta = results[:abilities].compact.first + + item_ids.each_with_index do |id, j| + expected = model.probability(theta, results[:discriminations][j], results[:difficulties][j], results[:guessings][j]) + expect(bank.probability(id, theta)).to be_within(1e-15).of(expected) + end + end + end +end diff --git a/spec/readme_examples_spec.rb b/spec/readme_examples_spec.rb index 0ab246a..379c6d6 100644 --- a/spec/readme_examples_spec.rb +++ b/spec/readme_examples_spec.rb @@ -52,4 +52,17 @@ expect(result[:abilities].size).to eq(data_with_missing.size) expect(result[:difficulties].size).to eq(data_with_missing.first.size) end + + it "runs the item bank example" do + rng = Random.new(1) + responses = Array.new(80) { Array.new(4) { rng.rand < 0.6 ? 1 : 0 } } + results = IrtRuby::TwoParameterModel.new(responses, seed: 1).fit + bank = IrtRuby::ItemBank.from_fit(results, item_ids: %w[q1 q2 q3 q4], metadata: { source: "spring pilot" }) + + restored = IrtRuby::ItemBank.from_json(bank.to_json) + + expect(restored).to eq(bank) + expect(restored.fetch("q2").c).to eq(0.0) + expect(restored.probability("q2", 0.5)).to be_between(0, 1) + end end From 862fbff0d146d46444316e488eaeb731e0c5bee4 Mon Sep 17 00:00:00 2001 From: Alex Kholodniak Date: Sun, 11 Oct 2026 07:46:41 +0300 Subject: [PATCH 2/3] fix: harden ItemBank as a long-lived contract Review found that callers could mutate the model name, from_fit invented a = 1 / c = 0 for missing estimates, eql? disagreed with hash, huge integers became Infinity after validation, excluded items skipped id checks, and Rasch slopes near 1 were stored unnormalized. Scoring new respondents will also need the calibration ability distribution, which the format could not carry, so add an optional ability_distribution now rather than bump format_version later. The mirt recipe now writes 17 significant digits and exports the latent mean and variance. --- CHANGELOG.md | 2 +- README.md | 15 ++-- lib/irt_ruby/item_bank.rb | 141 ++++++++++++++++++++++---------- spec/irt_ruby/item_bank_spec.rb | 85 +++++++++++++++++++ 4 files changed, 195 insertions(+), 48 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4d1f345..e31cd49 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ All notable changes to this project are documented in this file. ## [Unreleased] ### Added -- Added `IrtRuby::ItemBank`, an immutable set of calibrated items with stable string ids. Banks are built from fit results with `ItemBank.from_fit` (recording calibration provenance), saved and loaded as versioned JSON (`to_json`, `ItemBank.from_json`), validated per model, and expose item lookup and response probabilities. The README documents the format and an export recipe for R's mirt. +- Added `IrtRuby::ItemBank`, an immutable set of calibrated items with stable string ids. Banks are built from fit results with `ItemBank.from_fit` (recording calibration provenance), saved and loaded as versioned JSON (`to_json`, `ItemBank.from_json`), validated per model, carry the calibration ability distribution, and expose item lookup and response probabilities. The README documents the format and an export recipe for R's mirt. ## [0.4.0] - 2026-10-11 diff --git a/README.md b/README.md index 6818a67..0d954d0 100644 --- a/README.md +++ b/README.md @@ -181,7 +181,7 @@ bank.fetch("q2") # IrtRuby::ItemBank::Item with id, a, b, c bank.probability("q2", 0.5) # probability of a correct answer at theta = 0.5 ``` -`from_fit` infers the model from the result keys and records provenance in `metadata`: the irt_ruby version, the scale convention, and the `converged` and `iterations` values of the fit. Items whose estimates are `nil` raise an error; pass `exclude_unestimated: true` to leave them out and list their ids under `metadata["unestimated_items"]`. +`from_fit` infers the model from the result keys and records provenance in `metadata`: the irt_ruby version, the scale convention, and the `converged` and `iterations` values of the fit. It also stores the mean and SD of the estimated abilities as `ability_distribution`, describing the calibration sample on the bank's scale. Items with a `nil` estimate raise an error; pass `exclude_unestimated: true` to leave them out and list their ids under `metadata["unestimated_items"]`. The JSON format is: @@ -190,6 +190,7 @@ The JSON format is: "format": "irt_ruby.item_bank", "format_version": 1, "model": "2pl", + "ability_distribution": { "mean": 0.0, "sd": 1.0 }, "metadata": { "software": "irt_ruby 0.4.0", "converged": true }, "items": [ { "id": "q1", "a": 0.87, "b": 1.12, "c": 0.0 } @@ -197,7 +198,7 @@ The JSON format is: } ``` -Parameters use the logistic metric without a scaling constant (no `D = 1.7`): `P(correct | theta) = c + (1 - c) / (1 + exp(-a * (theta - b)))`. `model` is `"rasch"` (`a` = 1, `c` = 0), `"2pl"` (`c` = 0), or `"3pl"` (0 <= `c` < 1); for Rasch and 2PL the fixed parameters may be omitted. Banks with a different `format` or a newer `format_version` are rejected. +Parameters use the logistic metric without a scaling constant (no `D = 1.7`): `P(correct | theta) = c + (1 - c) / (1 + exp(-a * (theta - b)))`. `model` is `"rasch"` (`a` = 1, `c` = 0), `"2pl"` (`c` = 0), or `"3pl"` (0 <= `c` < 1); for Rasch and 2PL the fixed parameters may be omitted. `ability_distribution` is optional: the mean and SD of abilities in the calibration population, on the bank's scale. `metadata` is free-form JSON. Banks with a different `format` or a newer `format_version` are rejected. ### Importing from R's mirt @@ -208,18 +209,22 @@ library(mirt) library(jsonlite) mod <- mirt(responses, 1, itemtype = "2PL") -pars <- coef(mod, IRTpars = TRUE, simplify = TRUE)$items # columns a, b, g, u +cf <- coef(mod, IRTpars = TRUE, simplify = TRUE) +pars <- cf$items # columns a, b, g, u stopifnot(all(pars[, "u"] == 1)) bank <- list( format = "irt_ruby.item_bank", format_version = 1, model = "2pl", + ability_distribution = list(mean = unname(cf$means[1]), sd = sqrt(unname(cf$cov[1, 1]))), metadata = list(software = paste("mirt", packageVersion("mirt")), estimation = "marginal maximum likelihood"), items = lapply(rownames(pars), function(id) list(id = id, a = pars[id, "a"], b = pars[id, "b"], c = pars[id, "g"])) ) -writeLines(toJSON(bank, auto_unbox = TRUE, digits = NA), "item_bank.json") +writeLines(toJSON(bank, auto_unbox = TRUE, digits = I(17)), "item_bank.json") ``` -Use `model = "3pl"` for `itemtype = "3PL"`, and `model = "rasch"` for `itemtype = "Rasch"`. Note the scale conventions differ: mirt fixes the ability distribution to mean 0 and SD 1 in the population, while irt_ruby standardizes the estimated abilities of the calibration sample. +`digits = I(17)` writes 17 significant digits, enough to reproduce every double exactly; `digits = NA` writes only 15. Use `model = "3pl"` for `itemtype = "3PL"` and `model = "rasch"` for `itemtype = "Rasch"`. + +The scale conventions differ. For 2PL and 3PL, mirt fixes the ability distribution to mean 0 and SD 1 in the population, while irt_ruby standardizes the estimated abilities of the calibration sample. For Rasch, mirt fixes the slopes to 1 and estimates the ability variance, while irt_ruby only centers the estimated abilities. Either way, `ability_distribution` records the distribution that goes with the item parameters. ## Performance Benchmarks diff --git a/lib/irt_ruby/item_bank.rb b/lib/irt_ruby/item_bank.rb index e8d265c..857b892 100644 --- a/lib/irt_ruby/item_bank.rb +++ b/lib/irt_ruby/item_bank.rb @@ -12,15 +12,24 @@ class ItemBank FORMAT = "irt_ruby.item_bank" FORMAT_VERSION = 1 MODELS = %w[rasch 2pl 3pl].freeze + ESTIMATED_PARAMETERS = { + "rasch" => %i[difficulties], + "2pl" => %i[difficulties discriminations], + "3pl" => %i[difficulties discriminations guessings] + }.freeze Item = Struct.new(:id, :a, :b, :c) - attr_reader :model, :metadata + # ability_distribution is the mean and SD of abilities in the calibration + # population on the bank's scale, or nil when unknown. + attr_reader :model, :ability_distribution, :metadata - # Builds a bank from a model's fit results. Items whose parameters are nil - # raise unless exclude_unestimated is true, in which case their ids are - # recorded in the metadata under "unestimated_items". + # Builds a bank from a model's fit results. Items with a nil parameter raise + # unless exclude_unestimated is true, in which case their ids are recorded in + # the metadata under "unestimated_items". def self.from_fit(results, item_ids:, metadata: {}, exclude_unestimated: false) + raise ArgumentError, "metadata must be a Hash" unless metadata.is_a?(Hash) + model = if results.key?(:guessings) "3pl" elsif results.key?(:discriminations) @@ -28,15 +37,18 @@ def self.from_fit(results, item_ids:, metadata: {}, exclude_unestimated: false) else "rasch" end - difficulties = results.fetch(:difficulties) - raise ArgumentError, "item_ids must be an Array with one id per item (#{difficulties.size})" unless item_ids.is_a?(Array) && item_ids.size == difficulties.size + parameters = ESTIMATED_PARAMETERS.fetch(model).to_h do |key| + [key, results.fetch(key) { raise ArgumentError, "#{model} fit results must include #{key}" }] + end + validate_fit_ids!(item_ids, parameters) - unestimated = item_ids.select.with_index { |_, j| difficulties[j].nil? } + unestimated = item_ids.select.with_index { |_, j| parameters.values.any? { |values| values[j].nil? } } raise ArgumentError, "items without estimates: #{unestimated.join(", ")}; pass exclude_unestimated: true to leave them out" unless unestimated.empty? || exclude_unestimated - items = item_ids.each_index.reject { |j| difficulties[j].nil? }.map do |j| - { id: item_ids[j], a: results.fetch(:discriminations, nil)&.at(j) || 1.0, - b: difficulties[j], c: results.fetch(:guessings, nil)&.at(j) || 0.0 } + items = item_ids.each_index.reject { |j| unestimated.include?(item_ids[j]) }.map do |j| + { id: item_ids[j], b: parameters[:difficulties][j], + a: parameters.key?(:discriminations) ? parameters[:discriminations][j] : 1.0, + c: parameters.key?(:guessings) ? parameters[:guessings][j] : 0.0 } end provenance = { @@ -48,8 +60,33 @@ def self.from_fit(results, item_ids:, metadata: {}, exclude_unestimated: false) } provenance["unestimated_items"] = unestimated unless unestimated.empty? - new(model: model, items: items, metadata: provenance.merge(metadata.transform_keys(&:to_s))) + new(model: model, items: items, ability_distribution: sample_distribution(results[:abilities]), + metadata: provenance.merge(metadata.transform_keys(&:to_s))) + end + + def self.validate_fit_ids!(item_ids, parameters) + count = parameters[:difficulties].size + raise ArgumentError, "item_ids must be an Array with one id per item (#{count})" unless item_ids.is_a?(Array) && item_ids.size == count + + parameters.each do |key, values| + raise ArgumentError, "#{key} must be an Array with one value per item (#{count})" unless values.is_a?(Array) && values.size == count + end + item_ids.each_with_index do |id, j| + raise ArgumentError, "item #{j + 1} id must be a non-empty String" unless id.is_a?(String) && !id.empty? + end + raise ArgumentError, "item ids must be unique" unless item_ids.uniq.size == count + end + private_class_method :validate_fit_ids! + + def self.sample_distribution(abilities) + values = Array(abilities).compact + return nil if values.size < 2 + + mean = values.sum / values.size + sd = Math.sqrt(values.sum { |v| (v - mean)**2 } / values.size) + sd.positive? ? { "mean" => mean, "sd" => sd } : nil end + private_class_method :sample_distribution def self.from_json(json) from_h(JSON.parse(json)) @@ -62,21 +99,24 @@ def self.from_h(hash) version = hash["format_version"] raise ArgumentError, "unsupported item bank format_version #{version.inspect}; this irt_ruby reads #{FORMAT_VERSION}" unless version == FORMAT_VERSION - new(model: hash["model"], items: hash["items"], metadata: hash.fetch("metadata", {})) + new(model: hash["model"], items: hash["items"], ability_distribution: hash["ability_distribution"], + metadata: hash.fetch("metadata", {})) end # items: an Array of Hashes with an "id" (String) and "b", plus "a" (2PL/3PL) - # and "c" (3PL); string or symbol keys. metadata must be JSON-serializable. - def initialize(model:, items:, metadata: {}) - raise ArgumentError, "model must be one of #{MODELS}" unless MODELS.include?(model) + # and "c" (3PL); string or symbol keys. ability_distribution: nil or a Hash + # with "mean" and "sd". metadata must be JSON-serializable. + def initialize(model:, items:, ability_distribution: nil, metadata: {}) + @model = MODELS.find { |name| name == model } + raise ArgumentError, "model must be one of #{MODELS}" unless @model raise ArgumentError, "items must be a non-empty Array" unless items.is_a?(Array) && !items.empty? raise ArgumentError, "metadata must be a Hash" unless metadata.is_a?(Hash) - @model = model @items = items.each_with_index.map { |item, index| build_item(item, index) }.freeze @index = @items.each_with_index.to_h { |item, k| [item.id, k] }.freeze raise ArgumentError, "item ids must be unique" unless @index.size == @items.size + @ability_distribution = build_distribution(ability_distribution) @metadata = deep_freeze(normalize_metadata(metadata)) freeze end @@ -111,13 +151,11 @@ def probability(id, theta) end def to_h - { - "format" => FORMAT, - "format_version" => FORMAT_VERSION, - "model" => @model, - "metadata" => @metadata, - "items" => @items.map { |item| { "id" => item.id, "a" => item.a, "b" => item.b, "c" => item.c } } - } + hash = { "format" => FORMAT, "format_version" => FORMAT_VERSION, "model" => @model } + hash["ability_distribution"] = @ability_distribution if @ability_distribution + hash["metadata"] = @metadata + hash["items"] = @items.map { |item| { "id" => item.id, "a" => item.a, "b" => item.b, "c" => item.c } } + hash end def to_json(*args) @@ -127,7 +165,10 @@ def to_json(*args) def ==(other) other.is_a?(ItemBank) && to_h == other.to_h end - alias eql? == + + def eql?(other) + other.is_a?(ItemBank) && to_h.eql?(other.to_h) + end def hash to_h.hash @@ -141,41 +182,57 @@ def build_item(item, index) id = value(item, :id) raise ArgumentError, "item #{index + 1} id must be a non-empty String" unless id.is_a?(String) && !id.empty? - b = number(item, :b, id, required: true) - a = number(item, :a, id, required: @model != "rasch") - c = number(item, :c, id, required: @model == "3pl") - a = validate_discrimination(a, id) - c = validate_guessing(c, id) + label = "item #{id.inspect}" + b = number(value(item, :b), "#{label}: b", required: true) + a = number(value(item, :a), "#{label}: a", required: @model != "rasch") + c = number(value(item, :c), "#{label}: c", required: @model == "3pl") - Item.new(id.dup.freeze, a, b, c).freeze + Item.new(id.dup.freeze, discrimination(a, label), b, guessing(c, label)).freeze end - def validate_discrimination(a, id) + def discrimination(a, label) return 1.0 if a.nil? - raise ArgumentError, "item #{id.inspect}: rasch items must have a = 1" if @model == "rasch" && (a - 1.0).abs > 1e-9 - raise ArgumentError, "item #{id.inspect}: a must be positive" unless a.positive? + + if @model == "rasch" + raise ArgumentError, "#{label}: rasch items must have a = 1" if (a - 1.0).abs > 1e-9 + + return 1.0 + end + raise ArgumentError, "#{label}: a must be positive" unless a.positive? a end - def validate_guessing(c, id) + def guessing(c, label) return 0.0 if c.nil? - raise ArgumentError, "item #{id.inspect}: #{@model} items must have c = 0" if @model != "3pl" && !c.zero? - raise ArgumentError, "item #{id.inspect}: c must be in [0, 1)" unless c >= 0 && c < 1 + raise ArgumentError, "#{label}: #{@model} items must have c = 0" if @model != "3pl" && !c.zero? + raise ArgumentError, "#{label}: c must be in [0, 1)" unless c >= 0 && c < 1 c end - def number(item, key, id, required:) - raw = value(item, key) + def build_distribution(distribution) + return nil if distribution.nil? + raise ArgumentError, "ability_distribution must be a Hash with mean and sd" unless distribution.is_a?(Hash) + + mean = number(value(distribution, :mean), "ability_distribution mean", required: true) + sd = number(value(distribution, :sd), "ability_distribution sd", required: true) + raise ArgumentError, "ability_distribution sd must be positive" unless sd.positive? + + { "mean" => mean, "sd" => sd }.freeze + end + + def number(raw, label, required:) return nil if raw.nil? && !required - raise ArgumentError, "item #{id.inspect}: #{key} must be a finite number" unless raw.is_a?(Numeric) && !raw.is_a?(Complex) && raw.finite? - raw.to_f + converted = raw.to_f if raw.is_a?(Numeric) && !raw.is_a?(Complex) + raise ArgumentError, "#{label} must be a finite number" unless converted&.finite? + + converted end - def value(item, key) - item.key?(key) ? item[key] : item[key.to_s] + def value(hash, key) + hash.key?(key) ? hash[key] : hash[key.to_s] end def normalize_metadata(metadata) diff --git a/spec/irt_ruby/item_bank_spec.rb b/spec/irt_ruby/item_bank_spec.rb index 1532e1b..2adfcc7 100644 --- a/spec/irt_ruby/item_bank_spec.rb +++ b/spec/irt_ruby/item_bank_spec.rb @@ -67,6 +67,43 @@ expect { described_class.from_fit(results, item_ids: %w[q1 q2]) }.to raise_error(ArgumentError, /one id per item \(4\)/) end + + it "records the ability distribution of the calibration sample" do + rasch = IrtRuby::RaschModel.new(data, max_iter: 50, seed: 1).fit + two_pl = IrtRuby::TwoParameterModel.new(data, max_iter: 50, seed: 1).fit + abilities = rasch[:abilities].compact + mean, sd = IrtRuby::Estimation.mean_and_sd(abilities) + + expect(described_class.from_fit(rasch, item_ids: item_ids).ability_distribution).to eq("mean" => mean, "sd" => sd) + distribution = described_class.from_fit(two_pl, item_ids: item_ids).ability_distribution + expect(distribution["mean"]).to be_within(1e-12).of(0.0) + expect(distribution["sd"]).to be_within(1e-12).of(1.0) + end + + it "never invents values for estimated parameters" do + expect { described_class.from_fit({ difficulties: [0.0, 1.0], discriminations: [2.0] }, item_ids: %w[x y]) } + .to raise_error(ArgumentError, /discriminations must be an Array with one value per item \(2\)/) + expect { described_class.from_fit({ difficulties: [0.0], guessings: [0.1] }, item_ids: %w[x]) } + .to raise_error(ArgumentError, /3pl fit results must include discriminations/) + expect { described_class.from_fit({ difficulties: [0.0], discriminations: [nil] }, item_ids: %w[x]) } + .to raise_error(ArgumentError, /items without estimates: x/) + end + + it "validates every id, including those of excluded items" do + results = { difficulties: [0.0, nil] } + + expect { described_class.from_fit(results, item_ids: %w[x x], exclude_unestimated: true) } + .to raise_error(ArgumentError, /ids must be unique/) + expect { described_class.from_fit(results, item_ids: ["x", 7], exclude_unestimated: true) } + .to raise_error(ArgumentError, /item 2 id must be a non-empty String/) + end + + it "rejects metadata that is not a Hash" do + results = IrtRuby::RaschModel.new(data, max_iter: 5, seed: 1).fit + + expect { described_class.from_fit(results, item_ids: item_ids, metadata: "x") } + .to raise_error(ArgumentError, /metadata must be a Hash/) + end end describe "JSON" do @@ -92,6 +129,15 @@ ) end + it "round-trips the ability distribution" do + bank = described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0.5 }], + ability_distribution: { mean: 0.1, sd: 1.3 }) + restored = described_class.from_json(bank.to_json) + + expect(restored.ability_distribution).to eq("mean" => 0.1, "sd" => 1.3) + expect(JSON.parse(bank.to_json)["ability_distribution"]).to eq("mean" => 0.1, "sd" => 1.3) + end + it "fills in fixed parameters for rasch and 2pl items" do bank = described_class.from_json( '{"format":"irt_ruby.item_bank","format_version":1,"model":"rasch","items":[{"id":"x","b":0.5}]}' @@ -132,6 +178,7 @@ def build(model, items) "a missing difficulty" => ["rasch", [{ "id" => "x" }], /"x": b must be a finite number/], "a non-numeric difficulty" => ["rasch", [{ "id" => "x", "b" => "0" }], /b must be a finite number/], "an infinite difficulty" => ["rasch", [{ "id" => "x", "b" => Float::INFINITY }], /b must be a finite number/], + "a parameter that overflows to infinity" => ["rasch", [{ "id" => "x", "b" => 10**400 }], /b must be a finite number/], "a rasch discrimination other than 1" => ["rasch", [{ "id" => "x", "a" => 1.5, "b" => 0 }], /rasch items must have a = 1/], "a missing 2pl discrimination" => ["2pl", [{ "id" => "x", "b" => 0 }], /a must be a finite number/], "a non-positive discrimination" => ["2pl", [{ "id" => "x", "a" => 0, "b" => 0 }], /a must be positive/], @@ -145,6 +192,21 @@ def build(model, items) end end + it "stores a rasch discrimination within tolerance as exactly 1" do + expect(build("rasch", [{ "id" => "x", "a" => 1.0000000005, "b" => 0 }]).fetch("x").a).to eq(1.0) + end + + { + "a non-Hash" => [[0.0, 1.0], /must be a Hash with mean and sd/], + "a missing sd" => [{ "mean" => 0.0 }, /ability_distribution sd must be a finite number/], + "a non-positive sd" => [{ "mean" => 0.0, "sd" => 0 }, /sd must be positive/] + }.each do |description, (distribution, message)| + it "rejects an ability distribution with #{description}" do + expect { described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], ability_distribution: distribution) } + .to raise_error(ArgumentError, message) + end + end + it "rejects metadata that cannot be written as JSON" do expect { described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], metadata: { "x" => Float::NAN }) } .to raise_error(ArgumentError, /metadata must be JSON-serializable/) @@ -164,6 +226,16 @@ def build(model, items) expect(bank.metadata).to eq("notes" => ["first"]) end + it "is not affected by later changes to the model name" do + model = +"3pl" + bank = described_class.new(model: model, items: items_3pl) + model.replace("rasch") + + expect(bank.model).to eq("3pl") + expect(bank.model).to be_frozen + expect { bank.to_h["model"].replace("4pl") }.to raise_error(FrozenError) + end + it "freezes the bank, its items, and its metadata" do bank = described_class.new(model: "3pl", items: items_3pl, metadata: { "notes" => ["first"] }) @@ -174,6 +246,19 @@ def build(model, items) end end + describe "equality" do + it "keeps eql? consistent with hash" do + bank1 = described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], metadata: { "n" => 1 }) + bank2 = described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], metadata: { "n" => 1.0 }) + bank3 = described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], metadata: { "n" => 1 }) + + expect(bank1).to eq(bank2) + expect(bank1.eql?(bank2)).to be(false) + expect(bank1.eql?(bank3)).to be(true) + expect({ bank1 => :present }[bank3]).to eq(:present) + end + end + describe "lookup and probabilities" do let(:bank) { described_class.new(model: "3pl", items: items_3pl) } From 8a5490db87ae7d523fd2bd34c98043b9a4ee1782 Mon Sep 17 00:00:00 2001 From: Alex Kholodniak Date: Sun, 11 Oct 2026 07:49:20 +0300 Subject: [PATCH 3/3] fix: validate abilities and difficulties before building a bank Integer abilities were averaged with integer division, non-finite ones silently dropped the ability distribution, and a non-Array difficulties value raised NoMethodError instead of ArgumentError. --- lib/irt_ruby/item_bank.rb | 5 +++++ spec/irt_ruby/item_bank_spec.rb | 18 ++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/lib/irt_ruby/item_bank.rb b/lib/irt_ruby/item_bank.rb index 857b892..e7216a0 100644 --- a/lib/irt_ruby/item_bank.rb +++ b/lib/irt_ruby/item_bank.rb @@ -65,6 +65,8 @@ def self.from_fit(results, item_ids:, metadata: {}, exclude_unestimated: false) end def self.validate_fit_ids!(item_ids, parameters) + raise ArgumentError, "difficulties must be an Array" unless parameters[:difficulties].is_a?(Array) + count = parameters[:difficulties].size raise ArgumentError, "item_ids must be an Array with one id per item (#{count})" unless item_ids.is_a?(Array) && item_ids.size == count @@ -80,6 +82,9 @@ def self.validate_fit_ids!(item_ids, parameters) def self.sample_distribution(abilities) values = Array(abilities).compact + raise ArgumentError, "abilities must be finite numbers or nil" unless values.all? { |v| v.is_a?(Numeric) && !v.is_a?(Complex) && v.to_f.finite? } + + values = values.map(&:to_f) return nil if values.size < 2 mean = values.sum / values.size diff --git a/spec/irt_ruby/item_bank_spec.rb b/spec/irt_ruby/item_bank_spec.rb index 2adfcc7..3dbb4e5 100644 --- a/spec/irt_ruby/item_bank_spec.rb +++ b/spec/irt_ruby/item_bank_spec.rb @@ -80,6 +80,24 @@ expect(distribution["sd"]).to be_within(1e-12).of(1.0) end + it "computes the ability distribution in floating point" do + bank = described_class.from_fit({ difficulties: [0], abilities: [0, 1, 2, 3] }, item_ids: %w[x]) + + expect(bank.ability_distribution).to eq("mean" => 1.5, "sd" => Math.sqrt(1.25)) + end + + it "rejects non-finite abilities instead of dropping the distribution" do + [Float::NAN, Float::INFINITY].each do |bad| + expect { described_class.from_fit({ difficulties: [0.0], abilities: [bad, 0.0] }, item_ids: %w[x]) } + .to raise_error(ArgumentError, /abilities must be finite numbers or nil/) + end + end + + it "requires difficulties to be an Array" do + expect { described_class.from_fit({ difficulties: nil }, item_ids: %w[x]) } + .to raise_error(ArgumentError, /difficulties must be an Array/) + end + it "never invents values for estimated parameters" do expect { described_class.from_fit({ difficulties: [0.0, 1.0], discriminations: [2.0] }, item_ids: %w[x y]) } .to raise_error(ArgumentError, /discriminations must be an Array with one value per item \(2\)/)