Repository navigation
fix: make joint estimation correct and honest about convergence - #23
Merged
Merged
Conversation
The 3PL gradients omitted chain-rule factors, so 3PL fits optimized the wrong objective and recovered guessing parameters at chance level. Extreme response patterns have no finite estimate, yet they were returned as ordinary numbers that kept growing with max_iter, and the latent scale was left unidentified. The stopping rule could also fire after the learning rate collapsed, far from any optimum, with no way for callers to tell. Exclude extreme and unobserved persons/items (returned as nil), fix the scale after fitting, use a stable log-sigmoid likelihood, let the learning rate recover after accepted steps, and report converged/iterations with a stationarity check (gradient_tolerance). Finite-difference and recovery specs guard the math; docs no longer claim production readiness.
Mixed response rows and columns do not guarantee finite joint estimates: perfectly separated or disconnected groups still diverged or were left unidentified. Keep the largest strongly connected component of the response graph instead, which also covers extreme patterns. Standardizing the internal parameters pushed discriminations past their bounds and stalled later fit calls, and stationarity was judged on an arbitrary internal scale. Report rescaled copies and check stationarity on the reported scale. Under 3PL, all-incorrect respondents still inform guessing through log(1 - c), and the guessing gradient no longer drops to zero when the response function underflows.
At c = 0 a badly mispredicted correct response contributes exp(-z) to the guessing gradient, which overflowed to Infinity; backtracking could then never shrink the step, and once the learning rate reached zero the update became NaN and fit raised. Cap each term so the sum stays finite. Guessing gradients also include the limiting terms of excluded all-incorrect respondents, so average them over those responses too when checking stationarity. Document that strong connectivity guarantees finite estimates only for the Rasch model.
A correct response with a tiny but positive guessing parameter still overflowed through (1 - s) / P, reproducing the NaN crash. Apply the cap to both branches. Strong connectivity does guarantee finite estimates for 2PL with bounded discriminations, so limit the documented caveat to 3PL.
CI resolves the latest RuboCop because Gemfile.lock is not tracked, and 1.92 reports Layout/ExtraSpacing on these response matrices.
Gemfile.lock is not tracked, so CI resolved whatever RuboCop was newest and, with NewCops enabled, a new release could fail the quality job on unchanged code. RuboCop 1.92 needs Ruby >= 2.7, and Bundler resolves all groups even with BUNDLE_WITHOUT, so the gem is only declared on 2.7+; the Ruby 2.6 job runs just the specs.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
An audit of the estimation code found that results could be wrong or misleading without any signal to the caller:
max_iter, or kept their random initial values.1e-15insideMath.logflattened the objective for badly mispredicted responses. Input and output arrays aliased model state.What changes
nil. This guarantees finite estimates for Rasch and for 2PL with bounded discriminations. For 3PL it is necessary but not sufficient, and the README says so.log(1 - c)contribution. Guessing-gradient terms are capped so they cannot overflow toInfinity/NaN.fitcalls continue cleanly.fitreturnsconverged:anditerations:.convergedis true only at a stationary point on the reported scale (newgradient_tolerance:option, default1e-3). The learning rate grows 10% after accepted steps, so it can recover from earlier decays.Breaking changes (suggest releasing as 0.4.0)
nilfor non-estimable persons and items.Verification
spec/irt_ruby/estimation_spec.rbcovers:Known limitations
2PL/3PL joint estimation with plain gradient ascent converges slowly and often reports
converged: falseon realistic data. That flag is accurate, not a bug. Marginal maximum likelihood is the proper follow-up.