diff --git a/CHANGELOG.md b/CHANGELOG.md index 57dd08a..e31cd49 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,11 @@ All notable changes to this project are documented in this file. +## [Unreleased] + +### Added +- Added `IrtRuby::ItemBank`, an immutable set of calibrated items with stable string ids. Banks are built from fit results with `ItemBank.from_fit` (recording calibration provenance), saved and loaded as versioned JSON (`to_json`, `ItemBank.from_json`), validated per model, carry the calibration ability distribution, and expose item lookup and response probabilities. The README documents the format and an export recipe for R's mirt. + ## [0.4.0] - 2026-10-11 ### Fixed diff --git a/README.md b/README.md index 3a7c7e3..0d954d0 100644 --- a/README.md +++ b/README.md @@ -166,6 +166,66 @@ All models estimate person and item parameters jointly by maximizing the likelih - **Scale identification.** Results are reported on a fixed scale: Rasch abilities are centered to mean 0; 2PL and 3PL abilities are standardized to mean 0 and standard deviation 1, with difficulties and discriminations transformed so that response probabilities are unchanged. Only estimated respondents enter these statistics. The model keeps its internal parameters, so calling `fit` again continues from where it stopped; convergence is checked on the reported scale. - **Known limitations.** JMLE item estimates are biased when tests are short, and the bias does not vanish as the number of respondents grows. 2PL and 3PL JMLE is also slow to converge with plain gradient ascent. For high-stakes calibration, compare against marginal maximum likelihood software such as R's `mirt`. +## Item Banks + +An `IrtRuby::ItemBank` is an immutable set of calibrated items with stable string ids. Calibrate once, save the bank as JSON, and load it wherever you need the item parameters, for example inside a Rails app, without refitting. + +```ruby +results = IrtRuby::TwoParameterModel.new(responses, seed: 1).fit +bank = IrtRuby::ItemBank.from_fit(results, item_ids: %w[q1 q2 q3 q4], metadata: { source: "spring pilot" }) + +File.write("item_bank.json", bank.to_json) +bank = IrtRuby::ItemBank.from_json(File.read("item_bank.json")) + +bank.fetch("q2") # IrtRuby::ItemBank::Item with id, a, b, c +bank.probability("q2", 0.5) # probability of a correct answer at theta = 0.5 +``` + +`from_fit` infers the model from the result keys and records provenance in `metadata`: the irt_ruby version, the scale convention, and the `converged` and `iterations` values of the fit. It also stores the mean and SD of the estimated abilities as `ability_distribution`, describing the calibration sample on the bank's scale. Items with a `nil` estimate raise an error; pass `exclude_unestimated: true` to leave them out and list their ids under `metadata["unestimated_items"]`. + +The JSON format is: + +```json +{ + "format": "irt_ruby.item_bank", + "format_version": 1, + "model": "2pl", + "ability_distribution": { "mean": 0.0, "sd": 1.0 }, + "metadata": { "software": "irt_ruby 0.4.0", "converged": true }, + "items": [ + { "id": "q1", "a": 0.87, "b": 1.12, "c": 0.0 } + ] +} +``` + +Parameters use the logistic metric without a scaling constant (no `D = 1.7`): `P(correct | theta) = c + (1 - c) / (1 + exp(-a * (theta - b)))`. `model` is `"rasch"` (`a` = 1, `c` = 0), `"2pl"` (`c` = 0), or `"3pl"` (0 <= `c` < 1); for Rasch and 2PL the fixed parameters may be omitted. `ability_distribution` is optional: the mean and SD of abilities in the calibration population, on the bank's scale. `metadata` is free-form JSON. Banks with a different `format` or a newer `format_version` are rejected. + +### Importing from R's mirt + +mirt uses the same logistic metric. Its traditional IRT parameters can be written as an item bank with `jsonlite`: + +```r +library(mirt) +library(jsonlite) + +mod <- mirt(responses, 1, itemtype = "2PL") +cf <- coef(mod, IRTpars = TRUE, simplify = TRUE) +pars <- cf$items # columns a, b, g, u +stopifnot(all(pars[, "u"] == 1)) + +bank <- list( + format = "irt_ruby.item_bank", format_version = 1, model = "2pl", + ability_distribution = list(mean = unname(cf$means[1]), sd = sqrt(unname(cf$cov[1, 1]))), + metadata = list(software = paste("mirt", packageVersion("mirt")), estimation = "marginal maximum likelihood"), + items = lapply(rownames(pars), function(id) list(id = id, a = pars[id, "a"], b = pars[id, "b"], c = pars[id, "g"])) +) +writeLines(toJSON(bank, auto_unbox = TRUE, digits = I(17)), "item_bank.json") +``` + +`digits = I(17)` writes 17 significant digits, enough to reproduce every double exactly; `digits = NA` writes only 15. Use `model = "3pl"` for `itemtype = "3PL"` and `model = "rasch"` for `itemtype = "Rasch"`. + +The scale conventions differ. For 2PL and 3PL, mirt fixes the ability distribution to mean 0 and SD 1 in the population, while irt_ruby standardizes the estimated abilities of the calibration sample. For Rasch, mirt fixes the slopes to 1 and estimates the ability variance, while irt_ruby only centers the estimated abilities. Either way, `ability_distribution` records the distribution that goes with the item parameters. + ## Performance Benchmarks IRT Ruby includes comprehensive performance benchmarks to help you understand the computational characteristics of different models: diff --git a/lib/irt_ruby.rb b/lib/irt_ruby.rb index 88ff502..ae41a57 100644 --- a/lib/irt_ruby.rb +++ b/lib/irt_ruby.rb @@ -9,6 +9,7 @@ require "irt_ruby/rasch_model" require "irt_ruby/two_parameter_model" require "irt_ruby/three_parameter_model" +require "irt_ruby/item_bank" module IrtRuby class Error < StandardError; end diff --git a/lib/irt_ruby/item_bank.rb b/lib/irt_ruby/item_bank.rb new file mode 100644 index 0000000..e7216a0 --- /dev/null +++ b/lib/irt_ruby/item_bank.rb @@ -0,0 +1,257 @@ +# frozen_string_literal: true + +require "json" + +module IrtRuby + # An immutable set of calibrated items that can be stored as JSON and reused + # without recalibrating. Parameters use the logistic metric with no scaling + # constant: P(correct | theta) = c + (1 - c) / (1 + exp(-a * (theta - b))). + class ItemBank + include Enumerable + + FORMAT = "irt_ruby.item_bank" + FORMAT_VERSION = 1 + MODELS = %w[rasch 2pl 3pl].freeze + ESTIMATED_PARAMETERS = { + "rasch" => %i[difficulties], + "2pl" => %i[difficulties discriminations], + "3pl" => %i[difficulties discriminations guessings] + }.freeze + + Item = Struct.new(:id, :a, :b, :c) + + # ability_distribution is the mean and SD of abilities in the calibration + # population on the bank's scale, or nil when unknown. + attr_reader :model, :ability_distribution, :metadata + + # Builds a bank from a model's fit results. Items with a nil parameter raise + # unless exclude_unestimated is true, in which case their ids are recorded in + # the metadata under "unestimated_items". + def self.from_fit(results, item_ids:, metadata: {}, exclude_unestimated: false) + raise ArgumentError, "metadata must be a Hash" unless metadata.is_a?(Hash) + + model = if results.key?(:guessings) + "3pl" + elsif results.key?(:discriminations) + "2pl" + else + "rasch" + end + parameters = ESTIMATED_PARAMETERS.fetch(model).to_h do |key| + [key, results.fetch(key) { raise ArgumentError, "#{model} fit results must include #{key}" }] + end + validate_fit_ids!(item_ids, parameters) + + unestimated = item_ids.select.with_index { |_, j| parameters.values.any? { |values| values[j].nil? } } + raise ArgumentError, "items without estimates: #{unestimated.join(", ")}; pass exclude_unestimated: true to leave them out" unless unestimated.empty? || exclude_unestimated + + items = item_ids.each_index.reject { |j| unestimated.include?(item_ids[j]) }.map do |j| + { id: item_ids[j], b: parameters[:difficulties][j], + a: parameters.key?(:discriminations) ? parameters[:discriminations][j] : 1.0, + c: parameters.key?(:guessings) ? parameters[:guessings][j] : 0.0 } + end + + provenance = { + "software" => "irt_ruby #{VERSION}", + "estimation" => "joint maximum likelihood", + "scale" => model == "rasch" ? "abilities centered at 0" : "abilities standardized to mean 0, SD 1", + "converged" => results[:converged], + "iterations" => results[:iterations] + } + provenance["unestimated_items"] = unestimated unless unestimated.empty? + + new(model: model, items: items, ability_distribution: sample_distribution(results[:abilities]), + metadata: provenance.merge(metadata.transform_keys(&:to_s))) + end + + def self.validate_fit_ids!(item_ids, parameters) + raise ArgumentError, "difficulties must be an Array" unless parameters[:difficulties].is_a?(Array) + + count = parameters[:difficulties].size + raise ArgumentError, "item_ids must be an Array with one id per item (#{count})" unless item_ids.is_a?(Array) && item_ids.size == count + + parameters.each do |key, values| + raise ArgumentError, "#{key} must be an Array with one value per item (#{count})" unless values.is_a?(Array) && values.size == count + end + item_ids.each_with_index do |id, j| + raise ArgumentError, "item #{j + 1} id must be a non-empty String" unless id.is_a?(String) && !id.empty? + end + raise ArgumentError, "item ids must be unique" unless item_ids.uniq.size == count + end + private_class_method :validate_fit_ids! + + def self.sample_distribution(abilities) + values = Array(abilities).compact + raise ArgumentError, "abilities must be finite numbers or nil" unless values.all? { |v| v.is_a?(Numeric) && !v.is_a?(Complex) && v.to_f.finite? } + + values = values.map(&:to_f) + return nil if values.size < 2 + + mean = values.sum / values.size + sd = Math.sqrt(values.sum { |v| (v - mean)**2 } / values.size) + sd.positive? ? { "mean" => mean, "sd" => sd } : nil + end + private_class_method :sample_distribution + + def self.from_json(json) + from_h(JSON.parse(json)) + end + + def self.from_h(hash) + raise ArgumentError, "item bank must be a Hash" unless hash.is_a?(Hash) + raise ArgumentError, "not an item bank: format must be #{FORMAT.inspect}" unless hash["format"] == FORMAT + + version = hash["format_version"] + raise ArgumentError, "unsupported item bank format_version #{version.inspect}; this irt_ruby reads #{FORMAT_VERSION}" unless version == FORMAT_VERSION + + new(model: hash["model"], items: hash["items"], ability_distribution: hash["ability_distribution"], + metadata: hash.fetch("metadata", {})) + end + + # items: an Array of Hashes with an "id" (String) and "b", plus "a" (2PL/3PL) + # and "c" (3PL); string or symbol keys. ability_distribution: nil or a Hash + # with "mean" and "sd". metadata must be JSON-serializable. + def initialize(model:, items:, ability_distribution: nil, metadata: {}) + @model = MODELS.find { |name| name == model } + raise ArgumentError, "model must be one of #{MODELS}" unless @model + raise ArgumentError, "items must be a non-empty Array" unless items.is_a?(Array) && !items.empty? + raise ArgumentError, "metadata must be a Hash" unless metadata.is_a?(Hash) + + @items = items.each_with_index.map { |item, index| build_item(item, index) }.freeze + @index = @items.each_with_index.to_h { |item, k| [item.id, k] }.freeze + raise ArgumentError, "item ids must be unique" unless @index.size == @items.size + + @ability_distribution = build_distribution(ability_distribution) + @metadata = deep_freeze(normalize_metadata(metadata)) + freeze + end + + def each(&block) + return enum_for(:each) unless block + + @items.each(&block) + self + end + + def size + @items.size + end + + def ids + @items.map(&:id) + end + + def [](id) + position = @index[id] + position && @items[position] + end + + def fetch(id) + self[id] || raise(KeyError, "unknown item id #{id.inspect}") + end + + def probability(id, theta) + item = fetch(id) + item.c + ((1.0 - item.c) / (1.0 + Math.exp(-item.a * (theta - item.b)))) + end + + def to_h + hash = { "format" => FORMAT, "format_version" => FORMAT_VERSION, "model" => @model } + hash["ability_distribution"] = @ability_distribution if @ability_distribution + hash["metadata"] = @metadata + hash["items"] = @items.map { |item| { "id" => item.id, "a" => item.a, "b" => item.b, "c" => item.c } } + hash + end + + def to_json(*args) + to_h.to_json(*args) + end + + def ==(other) + other.is_a?(ItemBank) && to_h == other.to_h + end + + def eql?(other) + other.is_a?(ItemBank) && to_h.eql?(other.to_h) + end + + def hash + to_h.hash + end + + private + + def build_item(item, index) + raise ArgumentError, "item #{index + 1} must be a Hash" unless item.is_a?(Hash) + + id = value(item, :id) + raise ArgumentError, "item #{index + 1} id must be a non-empty String" unless id.is_a?(String) && !id.empty? + + label = "item #{id.inspect}" + b = number(value(item, :b), "#{label}: b", required: true) + a = number(value(item, :a), "#{label}: a", required: @model != "rasch") + c = number(value(item, :c), "#{label}: c", required: @model == "3pl") + + Item.new(id.dup.freeze, discrimination(a, label), b, guessing(c, label)).freeze + end + + def discrimination(a, label) + return 1.0 if a.nil? + + if @model == "rasch" + raise ArgumentError, "#{label}: rasch items must have a = 1" if (a - 1.0).abs > 1e-9 + + return 1.0 + end + raise ArgumentError, "#{label}: a must be positive" unless a.positive? + + a + end + + def guessing(c, label) + return 0.0 if c.nil? + raise ArgumentError, "#{label}: #{@model} items must have c = 0" if @model != "3pl" && !c.zero? + raise ArgumentError, "#{label}: c must be in [0, 1)" unless c >= 0 && c < 1 + + c + end + + def build_distribution(distribution) + return nil if distribution.nil? + raise ArgumentError, "ability_distribution must be a Hash with mean and sd" unless distribution.is_a?(Hash) + + mean = number(value(distribution, :mean), "ability_distribution mean", required: true) + sd = number(value(distribution, :sd), "ability_distribution sd", required: true) + raise ArgumentError, "ability_distribution sd must be positive" unless sd.positive? + + { "mean" => mean, "sd" => sd }.freeze + end + + def number(raw, label, required:) + return nil if raw.nil? && !required + + converted = raw.to_f if raw.is_a?(Numeric) && !raw.is_a?(Complex) + raise ArgumentError, "#{label} must be a finite number" unless converted&.finite? + + converted + end + + def value(hash, key) + hash.key?(key) ? hash[key] : hash[key.to_s] + end + + def normalize_metadata(metadata) + JSON.parse(JSON.generate(metadata)) + rescue JSON::GeneratorError => e + raise ArgumentError, "metadata must be JSON-serializable: #{e.message}" + end + + def deep_freeze(object) + case object + when Hash then object.each_value { |v| deep_freeze(v) } + when Array then object.each { |v| deep_freeze(v) } + end + object.freeze + end + end +end diff --git a/spec/irt_ruby/item_bank_spec.rb b/spec/irt_ruby/item_bank_spec.rb new file mode 100644 index 0000000..3dbb4e5 --- /dev/null +++ b/spec/irt_ruby/item_bank_spec.rb @@ -0,0 +1,307 @@ +# frozen_string_literal: true + +require "spec_helper" + +RSpec.describe IrtRuby::ItemBank do + let(:data) do + rng = Random.new(3) + Array.new(60) { Array.new(4) { rng.rand < 0.55 ? 1 : 0 } } + end + let(:item_ids) { %w[q1 q2 q3 q4] } + + let(:items_3pl) do + [ + { "id" => "q1", "a" => 1.2, "b" => -0.5, "c" => 0.2 }, + { "id" => "q2", "a" => 0.8, "b" => 0.75, "c" => 0.0 } + ] + end + + describe ".from_fit" do + { + IrtRuby::RaschModel => "rasch", + IrtRuby::TwoParameterModel => "2pl", + IrtRuby::ThreeParameterModel => "3pl" + }.each do |model_class, model_name| + it "builds a #{model_name} bank keyed by item id" do + results = model_class.new(data, max_iter: 50, seed: 1).fit + bank = described_class.from_fit(results, item_ids: item_ids) + + expect(bank.model).to eq(model_name) + expect(bank.ids).to eq(item_ids) + item_ids.each_with_index do |id, j| + item = bank.fetch(id) + expect(item.b).to eq(results[:difficulties][j]) + expect(item.a).to eq(results.fetch(:discriminations, Array.new(4, 1.0))[j]) + expect(item.c).to eq(results.fetch(:guessings, Array.new(4, 0.0))[j]) + end + end + end + + it "records calibration provenance and merges caller metadata" do + results = IrtRuby::RaschModel.new(data, max_iter: 5, seed: 1).fit + bank = described_class.from_fit(results, item_ids: item_ids, metadata: { source: "pilot 2026", software: "custom" }) + + expect(bank.metadata).to include( + "software" => "custom", + "estimation" => "joint maximum likelihood", + "scale" => "abilities centered at 0", + "converged" => results[:converged], + "iterations" => 5, + "source" => "pilot 2026" + ) + end + + it "rejects items without estimates unless asked to exclude them" do + results = IrtRuby::RaschModel.new([[1, 0, 1], [0, 1, 1]], seed: 1).fit + + expect { described_class.from_fit(results, item_ids: %w[a b c]) } + .to raise_error(ArgumentError, /items without estimates: c/) + + bank = described_class.from_fit(results, item_ids: %w[a b c], exclude_unestimated: true) + expect(bank.ids).to eq(%w[a b]) + expect(bank.metadata["unestimated_items"]).to eq(%w[c]) + end + + it "requires one id per item" do + results = IrtRuby::RaschModel.new(data, max_iter: 5, seed: 1).fit + + expect { described_class.from_fit(results, item_ids: %w[q1 q2]) }.to raise_error(ArgumentError, /one id per item \(4\)/) + end + + it "records the ability distribution of the calibration sample" do + rasch = IrtRuby::RaschModel.new(data, max_iter: 50, seed: 1).fit + two_pl = IrtRuby::TwoParameterModel.new(data, max_iter: 50, seed: 1).fit + abilities = rasch[:abilities].compact + mean, sd = IrtRuby::Estimation.mean_and_sd(abilities) + + expect(described_class.from_fit(rasch, item_ids: item_ids).ability_distribution).to eq("mean" => mean, "sd" => sd) + distribution = described_class.from_fit(two_pl, item_ids: item_ids).ability_distribution + expect(distribution["mean"]).to be_within(1e-12).of(0.0) + expect(distribution["sd"]).to be_within(1e-12).of(1.0) + end + + it "computes the ability distribution in floating point" do + bank = described_class.from_fit({ difficulties: [0], abilities: [0, 1, 2, 3] }, item_ids: %w[x]) + + expect(bank.ability_distribution).to eq("mean" => 1.5, "sd" => Math.sqrt(1.25)) + end + + it "rejects non-finite abilities instead of dropping the distribution" do + [Float::NAN, Float::INFINITY].each do |bad| + expect { described_class.from_fit({ difficulties: [0.0], abilities: [bad, 0.0] }, item_ids: %w[x]) } + .to raise_error(ArgumentError, /abilities must be finite numbers or nil/) + end + end + + it "requires difficulties to be an Array" do + expect { described_class.from_fit({ difficulties: nil }, item_ids: %w[x]) } + .to raise_error(ArgumentError, /difficulties must be an Array/) + end + + it "never invents values for estimated parameters" do + expect { described_class.from_fit({ difficulties: [0.0, 1.0], discriminations: [2.0] }, item_ids: %w[x y]) } + .to raise_error(ArgumentError, /discriminations must be an Array with one value per item \(2\)/) + expect { described_class.from_fit({ difficulties: [0.0], guessings: [0.1] }, item_ids: %w[x]) } + .to raise_error(ArgumentError, /3pl fit results must include discriminations/) + expect { described_class.from_fit({ difficulties: [0.0], discriminations: [nil] }, item_ids: %w[x]) } + .to raise_error(ArgumentError, /items without estimates: x/) + end + + it "validates every id, including those of excluded items" do + results = { difficulties: [0.0, nil] } + + expect { described_class.from_fit(results, item_ids: %w[x x], exclude_unestimated: true) } + .to raise_error(ArgumentError, /ids must be unique/) + expect { described_class.from_fit(results, item_ids: ["x", 7], exclude_unestimated: true) } + .to raise_error(ArgumentError, /item 2 id must be a non-empty String/) + end + + it "rejects metadata that is not a Hash" do + results = IrtRuby::RaschModel.new(data, max_iter: 5, seed: 1).fit + + expect { described_class.from_fit(results, item_ids: item_ids, metadata: "x") } + .to raise_error(ArgumentError, /metadata must be a Hash/) + end + end + + describe "JSON" do + it "round-trips every model exactly" do + [IrtRuby::RaschModel, IrtRuby::TwoParameterModel, IrtRuby::ThreeParameterModel].each do |model_class| + bank = described_class.from_fit(model_class.new(data, max_iter: 50, seed: 1).fit, item_ids: item_ids) + restored = described_class.from_json(bank.to_json) + + expect(restored).to eq(bank) + expect(restored.map(&:b)).to eq(bank.map(&:b)) + end + end + + it "writes the format header, model, metadata, and items" do + bank = described_class.new(model: "3pl", items: items_3pl, metadata: { "source" => "test" }) + + expect(JSON.parse(JSON.generate(bank))).to eq( + "format" => "irt_ruby.item_bank", + "format_version" => 1, + "model" => "3pl", + "metadata" => { "source" => "test" }, + "items" => items_3pl + ) + end + + it "round-trips the ability distribution" do + bank = described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0.5 }], + ability_distribution: { mean: 0.1, sd: 1.3 }) + restored = described_class.from_json(bank.to_json) + + expect(restored.ability_distribution).to eq("mean" => 0.1, "sd" => 1.3) + expect(JSON.parse(bank.to_json)["ability_distribution"]).to eq("mean" => 0.1, "sd" => 1.3) + end + + it "fills in fixed parameters for rasch and 2pl items" do + bank = described_class.from_json( + '{"format":"irt_ruby.item_bank","format_version":1,"model":"rasch","items":[{"id":"x","b":0.5}]}' + ) + + expect(bank.fetch("x").to_a).to eq(["x", 1.0, 0.5, 0.0]) + expect(bank.metadata).to eq({}) + end + + it "rejects documents that are not item banks" do + expect { described_class.from_json('{"format":"other","format_version":1}') } + .to raise_error(ArgumentError, /not an item bank/) + expect { described_class.from_json("[1]") }.to raise_error(ArgumentError, /must be a Hash/) + end + + it "rejects unsupported format versions" do + json = { "format" => "irt_ruby.item_bank", "format_version" => 2, "model" => "rasch", "items" => [] }.to_json + + expect { described_class.from_json(json) }.to raise_error(ArgumentError, /unsupported item bank format_version 2/) + end + end + + describe "validation" do + def build(model, items) + described_class.new(model: model, items: items) + end + + it "accepts symbol keys" do + expect(build("2pl", [{ id: "x", a: 1, b: 0 }]).fetch("x").to_a).to eq(["x", 1.0, 0.0, 0.0]) + end + + { + "an unknown model" => ["4pl", [{ "id" => "x", "b" => 0 }], /model must be one of/], + "an empty item list" => ["rasch", [], /non-empty Array/], + "a missing id" => ["rasch", [{ "b" => 0 }], /item 1 id must be a non-empty String/], + "a non-string id" => ["rasch", [{ "id" => 7, "b" => 0 }], /item 1 id must be a non-empty String/], + "duplicate ids" => ["rasch", [{ "id" => "x", "b" => 0 }, { "id" => "x", "b" => 1 }], /ids must be unique/], + "a missing difficulty" => ["rasch", [{ "id" => "x" }], /"x": b must be a finite number/], + "a non-numeric difficulty" => ["rasch", [{ "id" => "x", "b" => "0" }], /b must be a finite number/], + "an infinite difficulty" => ["rasch", [{ "id" => "x", "b" => Float::INFINITY }], /b must be a finite number/], + "a parameter that overflows to infinity" => ["rasch", [{ "id" => "x", "b" => 10**400 }], /b must be a finite number/], + "a rasch discrimination other than 1" => ["rasch", [{ "id" => "x", "a" => 1.5, "b" => 0 }], /rasch items must have a = 1/], + "a missing 2pl discrimination" => ["2pl", [{ "id" => "x", "b" => 0 }], /a must be a finite number/], + "a non-positive discrimination" => ["2pl", [{ "id" => "x", "a" => 0, "b" => 0 }], /a must be positive/], + "a 2pl guessing parameter" => ["2pl", [{ "id" => "x", "a" => 1, "b" => 0, "c" => 0.1 }], /2pl items must have c = 0/], + "a missing 3pl guessing parameter" => ["3pl", [{ "id" => "x", "a" => 1, "b" => 0 }], /c must be a finite number/], + "a guessing parameter of 1" => ["3pl", [{ "id" => "x", "a" => 1, "b" => 0, "c" => 1 }], /c must be in \[0, 1\)/], + "a negative guessing parameter" => ["3pl", [{ "id" => "x", "a" => 1, "b" => 0, "c" => -0.1 }], /c must be in \[0, 1\)/] + }.each do |description, (model, items, message)| + it "rejects #{description}" do + expect { build(model, items) }.to raise_error(ArgumentError, message) + end + end + + it "stores a rasch discrimination within tolerance as exactly 1" do + expect(build("rasch", [{ "id" => "x", "a" => 1.0000000005, "b" => 0 }]).fetch("x").a).to eq(1.0) + end + + { + "a non-Hash" => [[0.0, 1.0], /must be a Hash with mean and sd/], + "a missing sd" => [{ "mean" => 0.0 }, /ability_distribution sd must be a finite number/], + "a non-positive sd" => [{ "mean" => 0.0, "sd" => 0 }, /sd must be positive/] + }.each do |description, (distribution, message)| + it "rejects an ability distribution with #{description}" do + expect { described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], ability_distribution: distribution) } + .to raise_error(ArgumentError, message) + end + end + + it "rejects metadata that cannot be written as JSON" do + expect { described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], metadata: { "x" => Float::NAN }) } + .to raise_error(ArgumentError, /metadata must be JSON-serializable/) + end + end + + describe "immutability" do + it "is not affected by later changes to its inputs" do + items = [{ "id" => +"x", "a" => 1.0, "b" => 0.0, "c" => 0.1 }] + metadata = { "notes" => [+"first"] } + bank = described_class.new(model: "3pl", items: items, metadata: metadata) + items[0]["b"] = 5.0 + items[0]["id"] << "y" + metadata["notes"] << "second" + + expect(bank.fetch("x").b).to eq(0.0) + expect(bank.metadata).to eq("notes" => ["first"]) + end + + it "is not affected by later changes to the model name" do + model = +"3pl" + bank = described_class.new(model: model, items: items_3pl) + model.replace("rasch") + + expect(bank.model).to eq("3pl") + expect(bank.model).to be_frozen + expect { bank.to_h["model"].replace("4pl") }.to raise_error(FrozenError) + end + + it "freezes the bank, its items, and its metadata" do + bank = described_class.new(model: "3pl", items: items_3pl, metadata: { "notes" => ["first"] }) + + expect(bank).to be_frozen + expect(bank.first).to be_frozen + expect(bank.first.id).to be_frozen + expect(bank.metadata["notes"]).to be_frozen + end + end + + describe "equality" do + it "keeps eql? consistent with hash" do + bank1 = described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], metadata: { "n" => 1 }) + bank2 = described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], metadata: { "n" => 1.0 }) + bank3 = described_class.new(model: "rasch", items: [{ "id" => "x", "b" => 0 }], metadata: { "n" => 1 }) + + expect(bank1).to eq(bank2) + expect(bank1.eql?(bank2)).to be(false) + expect(bank1.eql?(bank3)).to be(true) + expect({ bank1 => :present }[bank3]).to eq(:present) + end + end + + describe "lookup and probabilities" do + let(:bank) { described_class.new(model: "3pl", items: items_3pl) } + + it "looks items up by id" do + expect(bank.size).to eq(2) + expect(bank["q2"].a).to eq(0.8) + expect(bank["missing"]).to be_nil + expect { bank.fetch("missing") }.to raise_error(KeyError, /unknown item id "missing"/) + end + + it "computes the logistic response probability without a scaling constant" do + expect(bank.probability("q1", 0.3)).to be_within(1e-15).of(0.2 + (0.8 / (1 + Math.exp(-1.2 * (0.3 - -0.5))))) + expect(bank.probability("q2", 0.75)).to eq(0.5) + end + + it "matches the probabilities of the model it was calibrated with" do + model = IrtRuby::ThreeParameterModel.new(data, max_iter: 50, seed: 1) + results = model.fit + bank = described_class.from_fit(results, item_ids: item_ids) + theta = results[:abilities].compact.first + + item_ids.each_with_index do |id, j| + expected = model.probability(theta, results[:discriminations][j], results[:difficulties][j], results[:guessings][j]) + expect(bank.probability(id, theta)).to be_within(1e-15).of(expected) + end + end + end +end diff --git a/spec/readme_examples_spec.rb b/spec/readme_examples_spec.rb index 0ab246a..379c6d6 100644 --- a/spec/readme_examples_spec.rb +++ b/spec/readme_examples_spec.rb @@ -52,4 +52,17 @@ expect(result[:abilities].size).to eq(data_with_missing.size) expect(result[:difficulties].size).to eq(data_with_missing.first.size) end + + it "runs the item bank example" do + rng = Random.new(1) + responses = Array.new(80) { Array.new(4) { rng.rand < 0.6 ? 1 : 0 } } + results = IrtRuby::TwoParameterModel.new(responses, seed: 1).fit + bank = IrtRuby::ItemBank.from_fit(results, item_ids: %w[q1 q2 q3 q4], metadata: { source: "spring pilot" }) + + restored = IrtRuby::ItemBank.from_json(bank.to_json) + + expect(restored).to eq(bank) + expect(restored.fetch("q2").c).to eq(0.0) + expect(restored.probability("q2", 0.5)).to be_between(0, 1) + end end