Skip to content

fix: make joint estimation correct and honest about convergence - #23

Merged
kholdrex merged 6 commits into
mainfrom
fix/estimation-correctness
Oct 11, 2026
Merged

kholdrex merged 6 commits into
mainfrom
fix/estimation-correctness

Conversation

@kholdrex

Copy link
Copy Markdown
Member

Why

An audit of the estimation code found that results could be wrong or misleading without any signal to the caller:

  • 3PL gradients omitted chain-rule factors (8–55% relative error against finite differences), so 3PL optimized the wrong objective. On 1000×20 simulated data, recovered guessing parameters correlated 0.09 with the true values.
  • Persons and items without a finite joint MLE (all-correct, all-incorrect, unobserved, perfectly separated or disconnected groups) were returned as ordinary numbers that kept growing with max_iter, or kept their random initial values.
  • The latent scale was not identified, and the stopping rule could fire after the learning rate collapsed, far from an optimum. It was still reported as a normal result.
  • 1e-15 inside Math.log flattened the objective for badly mispredicted responses. Input and output arrays aliased model state.

What changes

  • Estimability: only the largest strongly connected component of the response graph is estimated (correct answer: item → person, incorrect: person → item). Everything else is returned as nil. This guarantees finite estimates for Rasch and for 2PL with bounded discriminations. For 3PL it is necessary but not sufficient, and the README says so.
  • 3PL: correct gradients. Excluded all-incorrect respondents keep their limiting log(1 - c) contribution. Guessing-gradient terms are capped so they cannot overflow to Infinity/NaN.
  • Scale: results are reported with Rasch abilities centered at 0, and 2PL/3PL abilities standardized to mean 0 / SD 1 with likelihood-preserving item transforms. Internal state is left untouched, so repeated fit calls continue cleanly.
  • Convergence: fit returns converged: and iterations:. converged is true only at a stationary point on the reported scale (new gradient_tolerance: option, default 1e-3). The learning rate grows 10% after accepted steps, so it can recover from earlier decays.
  • Other: stable log-sigmoid likelihood; defensive copies of input and output arrays.
  • Docs: README gains an "Estimation Details" section. The gemspec no longer claims production readiness. The 0.3.0 date in the CHANGELOG is corrected.

Breaking changes (suggest releasing as 0.4.0)

  • Result arrays can contain nil for non-estimable persons and items.
  • Result hashes have two extra keys, and 2PL/3PL numbers change because of the gradient fix and scale identification.

Verification

  • 277 specs pass on Ruby 2.6.10, 3.3 and 4.0.6. RuboCop is clean and the gem builds.
  • New spec/irt_ruby/estimation_spec.rb covers:
    • finite-difference gradient checks for every model and missing-data strategy;
    • exclusion of extreme, separated and disconnected groups;
    • scale identification and likelihood preservation;
    • repeated fits and stationarity on the reported scale;
    • 3PL floor terms and overflow;
    • Rasch/2PL parameter recovery.
  • Each regression spec fails on the code before its fix.

Known limitations

2PL/3PL joint estimation with plain gradient ascent converges slowly and often reports converged: false on realistic data. That flag is accurate, not a bug. Marginal maximum likelihood is the proper follow-up.

The 3PL gradients omitted chain-rule factors, so 3PL fits optimized the
wrong objective and recovered guessing parameters at chance level. Extreme
response patterns have no finite estimate, yet they were returned as
ordinary numbers that kept growing with max_iter, and the latent scale was
left unidentified. The stopping rule could also fire after the learning
rate collapsed, far from any optimum, with no way for callers to tell.

Exclude extreme and unobserved persons/items (returned as nil), fix the
scale after fitting, use a stable log-sigmoid likelihood, let the learning
rate recover after accepted steps, and report converged/iterations with a
stationarity check (gradient_tolerance). Finite-difference and recovery
specs guard the math; docs no longer claim production readiness.
Mixed response rows and columns do not guarantee finite joint estimates:
perfectly separated or disconnected groups still diverged or were left
unidentified. Keep the largest strongly connected component of the
response graph instead, which also covers extreme patterns.

Standardizing the internal parameters pushed discriminations past their
bounds and stalled later fit calls, and stationarity was judged on an
arbitrary internal scale. Report rescaled copies and check stationarity
on the reported scale. Under 3PL, all-incorrect respondents still inform
guessing through log(1 - c), and the guessing gradient no longer drops
to zero when the response function underflows.
At c = 0 a badly mispredicted correct response contributes exp(-z) to the
guessing gradient, which overflowed to Infinity; backtracking could then
never shrink the step, and once the learning rate reached zero the update
became NaN and fit raised. Cap each term so the sum stays finite.

Guessing gradients also include the limiting terms of excluded
all-incorrect respondents, so average them over those responses too when
checking stationarity. Document that strong connectivity guarantees
finite estimates only for the Rasch model.
A correct response with a tiny but positive guessing parameter still
overflowed through (1 - s) / P, reproducing the NaN crash. Apply the cap
to both branches. Strong connectivity does guarantee finite estimates for
2PL with bounded discriminations, so limit the documented caveat to 3PL.
CI resolves the latest RuboCop because Gemfile.lock is not tracked, and
1.92 reports Layout/ExtraSpacing on these response matrices.
Gemfile.lock is not tracked, so CI resolved whatever RuboCop was newest
and, with NewCops enabled, a new release could fail the quality job on
unchanged code. RuboCop 1.92 needs Ruby >= 2.7, and Bundler resolves all
groups even with BUNDLE_WITHOUT, so the gem is only declared on 2.7+;
the Ruby 2.6 job runs just the specs.
@kholdrex
kholdrex merged commit 0a6c350 into main Oct 11, 2026
5 checks passed
@kholdrex
kholdrex deleted the fix/estimation-correctness branch October 11, 2026 04:30
@kholdrex kholdrex changed the title fix: make joint estimation correct and honest about convergence Release 0.4.0: correct joint estimation and honest convergence Oct 11, 2026
@kholdrex kholdrex changed the title Release 0.4.0: correct joint estimation and honest convergence fix: make joint estimation correct and honest about convergence Oct 11, 2026
@kholdrex kholdrex mentioned this pull request Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant