Repository navigation
BUG: Do not treat badly scaled moment covariances as singular in cent… - #754
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #754 +/- ##
==========================================
- Coverage 99.60% 99.58% -0.02%
==========================================
Files 127 128 +1
Lines 23832 24188 +356
Branches 1783 1809 +26
==========================================
+ Hits 23737 24087 +350
- Misses 35 40 +5
- Partials 60 61 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…singular Stacked on lm71-bug3b-iv-features (BUG3b), which has to be merged first. That branch adds the scale-free rank check of the covariance of the moment conditions to IVGMMResults.centered_j_stat. This commit moves the check to a function and also applies it to j_stat and c_stat. IVGMMResults.c_stat and IVGMMResults.j_stat gave meaningless numbers if the estimate of the covariance of the moment conditions was singular, which is the case if there are too few clusters. The weight matrix is the inverse of that estimate, so it is numerical noise, and so are the statistics. c_stat uses the weight type of the model since #744, and so it uses the clustered covariance. Reproduction: n = 120, a constant and an exogenous regressor, one endogenous regressor and two instruments, which are 4 moment conditions, and clusters = arange(n) % G. The model is IVGMM(..., weight_type="clustered", clusters=clusters).fit(cov_type="clustered", clusters=clusters). G c_stat() before c_stat() now 3 -3.73e15 invalid 4 -7.42e16 invalid 5 0.8760 0.8760 The model of the C statistic treats the endogenous regressor as exogenous, so it has 5 moment conditions, and its covariance is singular unless there are at least 5 clusters. 7.0 gave -0.049 for every G, since it ignored the clusters, which is not valid either. c_stat also raised a LinAlgError when NumPy found that the covariance is exactly singular, e.g., with 2 centered clusters. j_stat is affected in the same way, in 7.0 and in main. J of the same model, for the default of IVGMM (center=False) and for center=True, which is the default of IVGMMCUE: G center=False center=True now 2 -3.19e15 1.41 invalid, invalid 3 -6.87e15 1.73e14 invalid, invalid 4 0.764 1.52e15 valid, invalid 5 0.285 0.303 valid, valid A covariance that is estimated from G clusters is a sum of G outer products, so its rank is at most G. The centered cluster sums add to zero, so the rank is at most G - 1 if the moment conditions are centered. With center=True and G = 2 the statistic looks plausible, 1.41, but it is as meaningless as the others. The rank of a kernel or a robust covariance is bounded by the number of observations in the same way, and so these have singular cases too, e.g., 4 observations and 4 moment conditions with center=True gave J = 4.2e6 with a kernel and 6.96e9 with the robust estimator. The fix: * The check of centered_j_stat is a function, _is_singular. Its rank is that of the correlation matrix, so it does not depend on the units of the data. * IVGMM.fit and IVGMMCUE.fit pass the covariance that they inverted to the post-estimation, and j_stat is an InvalidTestStatistic if it is singular. It is not checked if no covariance is estimated, which is the case if iter_limit=1, since the weight matrix is then the inverse of Z'Z / n. * c_stat is an InvalidTestStatistic with a reason if the J statistic of the model that treats the tested variables as exogenous is invalid, or if fitting it raises a LinAlgError. The C statistic is the difference of two J statistics that use the covariance of this model. The covariance is tested and not its inverse. In 48,000 simulated clustered covariances, with the units of the instruments between 1e-3 and 1e3, 14,809 of which are singular by construction, the test of the covariance found all of them and had no false alarms. The same test of the inverse missed 780 of them (5%). The estimates are not changed. A GMM model with a singular covariance still uses the inverse of noise, so its parameters and standard errors are not meaningful, or it raises a LinAlgError. The statistics of every model that is not singular are unchanged: the reference tests of the J, centered J and C statistics are not modified. The new tests compute the expected statistics from the definitions, without the weight matrix classes, for the cases above and for models with two endogenous regressors and tested subsets, weights, continuously updating GMM, kernel and robust covariances with 4 to 20 observations, and instruments in units that differ by 1e16. They also check that the statistics are valid with exactly as many clusters as moment conditions, that j_stat is not invalid if no covariance is estimated, and the LinAlgError of c_stat. 54 of the 118 tests fail without the change.
…clusters IVGMMCUE with weight_type="clustered" raised numpy.linalg.LinAlgError: Singular matrix, from inside the objective function that scipy minimizes, if there were not more clusters than moment conditions. The message does not say what the problem is or what to do. It happens since #746, which makes the moment conditions centered by default, as the documentation says they should be. 7.0 fit a model with as many clusters as moment conditions, since it did not center them. Reproduction (reg1/repro_leads.py): n = 120, a constant and an exogenous regressor, one endogenous regressor and two instruments, so 4 moment conditions, and clusters = arange(n) % G. G center=True (the default) center=False 2 LinAlgError LinAlgError 3 LinAlgError LinAlgError 4 LinAlgError fits 5 fits fits The objective is a function of the inverse of the covariance of the moment conditions. The covariance that is estimated from G clusters is a sum of G outer products, and so it has rank at most G. The cluster sums of centered moment conditions add to zero, so its rank is at most G - 1. With 4 moment conditions it is singular for G = 4 and less if they are centered, and for G = 3 and less if they are not. Robust and kernel covariances have the same problem if there are too few observations, e.g., 4 observations and 4 moment conditions. fit now tests the rank of the covariance before it does anything else, with the function that the J and C statistics use, and raises a ValueError. For example, for G = 4: The clustered covariance of the 4 moment conditions is singular because there are too few clusters (4 clusters). Its rank is at most 3, one less than the number of clusters, since the cluster sums of centered moment conditions add to zero. The objective function of the continuously updating GMM estimator uses the inverse of the covariance, so it is not defined. Use more clusters than moment conditions, or a weight_type that does not use clusters. The covariance is estimated at the first step estimates of two-step GMM, which are 2SLS. Whether it is singular does not depend on the parameters, except for special values such as residuals that are all 0. The check is before the starting values are estimated, since that also inverts the covariance, and before the optimization starts, so there is no work to do before the error. The LinAlgError was raised after the optimizer started. A model with enough clusters is not changed, and its extra cost is one estimate of the covariance. The class and fit documentation say that the covariance has to be nonsingular. Not changed: * IVGMM, the two-step model, does not fail in the same way, and its estimation is not changed. With too few clusters it uses the inverse of a singular covariance, which is noise, and returns meaningless parameters, or it raises a LinAlgError if the covariance is exactly singular. For the model above and G = 3 the coefficient on the endogenous regressor is 1.70, and the true value is 0.5 and the estimate with G = 5 is 0.48. The J and C statistics of the previous commit are invalid. * A model with center=False and as many clusters as moment conditions still fits, as the report says. The covariance is not singular. The objective is then constant, and equal to the number of clusters, for every value of the parameters, so the estimates are the starting values. The new tests check the exact type of the error, since LinAlgError is a subclass of ValueError, the numbers of clusters and moment conditions in the message, the error with starting values, that the optimizer is not started, that models with enough clusters fit and have the objective of the definition as their J statistic, and the kernel and robust covariances with 4, 5 and 8 observations. 10 of the 20 new tests fail without the change.
The two Python 3.10 Azure jobs, which use NumPy 1.x, failed in test_c_stat_with_too_few_clusters_is_invalid[3] with "'NoneType' object has no attribute 'c_stat'". With 3 clusters the covariance of the model's own 4 moments is singular as well, and there NumPy raised LinAlgError when the model was estimated, which the fit helper of the test turns into None. The other tests in the file already return early in that case; this one did not. Checked by replacing inv() with one that raises for an ill-conditioned matrix, as NumPy did on those jobs: the test failed with the same AttributeError before this change and passes after it. It also passes with the real inv(), and the whole file passes (138 tests).
bashtage
force-pushed
the
lm71-fix-gmm-singular
branch
from
October 12, 2026 08:44
b0331c6 to
f4f8a09
Compare
shaneraphel
pushed a commit
to shaneraphel/linearmodels
that referenced
this pull request
Oct 13, 2026
origin/main now has bashtage#754 (a ValueError for IVGMMCUE with too few clusters), which changed the Raises and Notes sections of IVGMMCUE.fit. The Raises section comes first, then See Also and the Notes of this branch.
shaneraphel
pushed a commit
to shaneraphel/linearmodels
that referenced
this pull request
Oct 13, 2026
The weight matrix of two-step GMM is the inverse of an estimate of the covariance of the moment conditions. If there are fewer clusters than moment conditions (or too few observations) the estimate is singular, the inverse is numerical noise, and so are the estimates. bashtage#754 made j_stat and c_stat invalid and made IVGMMCUE raise a clear ValueError in this case, and left the estimates of IVGMM as they were: * In the model of the bug report with 3 clusters, the coefficient on x is 1.697 when it is 0.5, and its standard error is NaN (and 1.95e6 for the constant); it is a LinAlgError instead if NumPy finds that the matrix is exactly singular, which depends on rounding and so on the platform. * An exactly identified model does not need its weight matrix, but the noise in it is not harmless: with 3 clusters and 3 moment conditions the estimates are 9.7 away from the IV estimates, and the standard errors are not numbers. The first-step moment conditions of an exactly identified model add to zero, so that the covariance of 3 moment conditions is singular with 3 clusters even if they are not centered. IVGMM.fit now raises the same ValueError as IVGMMCUE.fit, with the message built by the same function: it says that the covariance is singular, that there are too few clusters (with their number and the rank that they allow) or too few observations, and to use more clusters than moment conditions. It is checked at every step before the covariance is inverted. It is not checked with iter_limit=1, which does not use a covariance. IVGMMResults.c_stat estimates a model with a moment condition for each tested variable, and returns an invalid statistic instead of an error if that model cannot be estimated, as it did for a LinAlgError. j_stat of an IVGMM is never invalid now, since the model is not estimated. The fits that are valid are not changed. Params, cov, std_errors, j_stat, centered_j_stat and c_stat of 600 random models (IVGMM and IVGMMCUE, the four weight types, observation weights, three covariances, units of the instruments from 1e-2 to 1e2, and cluster counts around the number of moment conditions) are the same to the last bit on origin/main and here for the 561 that are estimated on both. The 39 that raise here are clustered models with no more clusters than moment conditions: 19 raised on origin/main too, and the 20 that did not had an invalid J statistic, and NaN or enormous standard errors. Tests: the tests of bashtage#754 for the J and C statistics with too few clusters (and the small samples) accepted a result or a LinAlgError from fit, and returned if there was none. They test the ValueError and its message now, and the values of the statistics for the models that can be estimated. New tests check that the message is that of IVGMMCUE except for the sentence that says why the inverse is needed; that exactly identified models raise with 2 and 3 clusters and are the IV estimates with 4 or more; that every step is checked, and the initial weight does not avoid the check; that iter_limit=1 is not changed; and that the params, std errors, J, centered J and C of seven valid models are those of origin/main (and the params and J are the definitions).
shaneraphel
pushed a commit
to shaneraphel/linearmodels
that referenced
this pull request
Oct 13, 2026
…nditions
IVGMMCUE with weight_type="clustered" and center=False fitted a model with as
many clusters as moment conditions without an error, but the objective
function is constant. The cluster sums of the moment conditions are the rows
of a square matrix Xi (G by K), the average of the moment conditions is
Xi'1 / n and the covariance is Xi'Xi / n, so that the objective function is
1'Xi(Xi'Xi)^{-1}Xi'1, which is G whenever Xi is nonsingular, for every value of
the parameters (divided by the small sample adjustment if the covariance is
debiased). In the model of the bug report with 4 clusters the objective is
3.99999999997 at every parameter value that was tried, the optimizer stops
after one iteration, and the "estimates" are the starting values, the
estimates of IVGMM, with a J statistic equal to the number of clusters.
bashtage#754 made IVGMMCUE raise if the covariance is singular, which needs more
clusters than moment conditions if they are centered (the default) and at
least as many if they are not. Now more clusters than moment conditions are
needed with every value of center. The check is made at the same place,
before the starting values are estimated and before the objective is
optimized, and also if starting values are given. The message says that there
are as many clusters as moment conditions, that the moment conditions are not
centered, that the objective function is the same for every value of the
parameters so that there is nothing to optimize, and to use more clusters
than moment conditions or a weight_type that does not use clusters.
With as many clusters as moment conditions a model with centered moment
conditions, and an exactly identified model, are singular and raise the
message of bashtage#754. Nothing else changes: of the 600 random models of the
previous commit, the only one that is affected is the one with 4 clusters and
4 uncentered moment conditions, and its J statistic was 3.9999999999999654.
Tests: the objective function of the model with 4 clusters is 4 at five
random parameter vectors (and 4 / (the adjustment) if debiased), computed by
IVGMMCUE.j and from the definition without the package's classes; fit raises
before it optimizes, with and without starting values; the message; and a test
of 2 to 8 clusters with both values of center, which raises for 4 or fewer
clusters in IVGMMCUE, and for IVGMM exactly when the covariance is singular.
The test of bashtage#754 that fitted a model with 4 uncentered clusters expects an
error now.
shaneraphel
pushed a commit
to shaneraphel/linearmodels
that referenced
this pull request
Oct 13, 2026
…sing IVGMM test_centered_j_cue_with_too_few_clusters fitted an IVGMMCUE with center=False and 5 clusters for 5 moment conditions to check that the centered J statistic is invalid when the uncentered fit is valid. That model cannot be estimated any more: the objective function of the CUE is constant with as many clusters as moment conditions, and the CUE needs 6 clusters, for which the centered covariance is not singular either. The two-step estimator can still be estimated with 5 uncentered clusters, and it is the case where the J statistic is valid and the centered J statistic is not, so the test checks it with IVGMM. It checks that the CUE raises with 5 clusters, and that the CUE with 6 clusters has a valid centered J statistic. The test of the centered J statistic of the CUE with a singular covariance of bashtage#754 is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…ered_j_stat
IVGMMResults.centered_j_stat returned an InvalidTestStatistic, saying that the
covariance of the moment conditions is singular, for models in which it is not.
It used matrix_rank on the covariance, and the tolerance of matrix_rank is
relative to the largest eigenvalue. When the variances of two moment conditions
differ by a factor of about 1e16, which is the case if two instruments are
measured in units that differ by a factor of 1e8, the smaller one is below the
tolerance and the covariance is reported as singular.
For a model with four instruments, one of which is multiplied by a constant,
the J statistic is 7.6196 at every scale. The centered statistic is 7.906 for
constants up to 1e7 and is invalid from 1e8 on, and the same for constants
down to 1e-8. This occurred for the unadjusted, robust, kernel and clustered
estimators, with and without weights, and when the scale was that of an
exogenous regressor, which is also an instrument.
The rank is now that of the correlation matrix of the moment conditions, which
does not depend on the units. A covariance whose variances are not positive or
that is not finite is also singular. The statistic is computed as before, and
all of the existing reference values are unchanged. A covariance that is
singular is still found, e.g., when there are too few clusters, with any scale.
The tests that are added check the definition of the statistic for IVGMM and
IVGMMCUE, without using the weight matrix classes, with weights, one-way
clusters, a kernel and the small sample adjustment. The reference values from R
do not cover continuously updating GMM with weights or a kernel, or weights
with clusters. They also check continuously updating GMM with and without
centering, and that the statistic is invalid when there are as many clusters as
moment conditions. 32 tests of the units of the data fail without the change.