Repository navigation
BUG: Raise a ValueError if IVGMM has a singular moment covariance - #785
Merged
Merged
Conversation
The weight matrix of two-step GMM is the inverse of an estimate of the covariance of the moment conditions. If there are fewer clusters than moment conditions (or too few observations) the estimate is singular, the inverse is numerical noise, and so are the estimates. #754 made j_stat and c_stat invalid and made IVGMMCUE raise a clear ValueError in this case, and left the estimates of IVGMM as they were: * In the model of the bug report with 3 clusters, the coefficient on x is 1.697 when it is 0.5, and its standard error is NaN (and 1.95e6 for the constant); it is a LinAlgError instead if NumPy finds that the matrix is exactly singular, which depends on rounding and so on the platform. * An exactly identified model does not need its weight matrix, but the noise in it is not harmless: with 3 clusters and 3 moment conditions the estimates are 9.7 away from the IV estimates, and the standard errors are not numbers. The first-step moment conditions of an exactly identified model add to zero, so that the covariance of 3 moment conditions is singular with 3 clusters even if they are not centered. IVGMM.fit now raises the same ValueError as IVGMMCUE.fit, with the message built by the same function: it says that the covariance is singular, that there are too few clusters (with their number and the rank that they allow) or too few observations, and to use more clusters than moment conditions. It is checked at every step before the covariance is inverted. It is not checked with iter_limit=1, which does not use a covariance. IVGMMResults.c_stat estimates a model with a moment condition for each tested variable, and returns an invalid statistic instead of an error if that model cannot be estimated, as it did for a LinAlgError. j_stat of an IVGMM is never invalid now, since the model is not estimated. The fits that are valid are not changed. Params, cov, std_errors, j_stat, centered_j_stat and c_stat of 600 random models (IVGMM and IVGMMCUE, the four weight types, observation weights, three covariances, units of the instruments from 1e-2 to 1e2, and cluster counts around the number of moment conditions) are the same to the last bit on origin/main and here for the 561 that are estimated on both. The 39 that raise here are clustered models with no more clusters than moment conditions: 19 raised on origin/main too, and the 20 that did not had an invalid J statistic, and NaN or enormous standard errors. Tests: the tests of #754 for the J and C statistics with too few clusters (and the small samples) accepted a result or a LinAlgError from fit, and returned if there was none. They test the ValueError and its message now, and the values of the statistics for the models that can be estimated. New tests check that the message is that of IVGMMCUE except for the sentence that says why the inverse is needed; that exactly identified models raise with 2 and 3 clusters and are the IV estimates with 4 or more; that every step is checked, and the initial weight does not avoid the check; that iter_limit=1 is not changed; and that the params, std errors, J, centered J and C of seven valid models are those of origin/main (and the params and J are the definitions).
…nditions
IVGMMCUE with weight_type="clustered" and center=False fitted a model with as
many clusters as moment conditions without an error, but the objective
function is constant. The cluster sums of the moment conditions are the rows
of a square matrix Xi (G by K), the average of the moment conditions is
Xi'1 / n and the covariance is Xi'Xi / n, so that the objective function is
1'Xi(Xi'Xi)^{-1}Xi'1, which is G whenever Xi is nonsingular, for every value of
the parameters (divided by the small sample adjustment if the covariance is
debiased). In the model of the bug report with 4 clusters the objective is
3.99999999997 at every parameter value that was tried, the optimizer stops
after one iteration, and the "estimates" are the starting values, the
estimates of IVGMM, with a J statistic equal to the number of clusters.
#754 made IVGMMCUE raise if the covariance is singular, which needs more
clusters than moment conditions if they are centered (the default) and at
least as many if they are not. Now more clusters than moment conditions are
needed with every value of center. The check is made at the same place,
before the starting values are estimated and before the objective is
optimized, and also if starting values are given. The message says that there
are as many clusters as moment conditions, that the moment conditions are not
centered, that the objective function is the same for every value of the
parameters so that there is nothing to optimize, and to use more clusters
than moment conditions or a weight_type that does not use clusters.
With as many clusters as moment conditions a model with centered moment
conditions, and an exactly identified model, are singular and raise the
message of #754. Nothing else changes: of the 600 random models of the
previous commit, the only one that is affected is the one with 4 clusters and
4 uncentered moment conditions, and its J statistic was 3.9999999999999654.
Tests: the objective function of the model with 4 clusters is 4 at five
random parameter vectors (and 4 / (the adjustment) if debiased), computed by
IVGMMCUE.j and from the definition without the package's classes; fit raises
before it optimizes, with and without starting values; the message; and a test
of 2 to 8 clusters with both values of center, which raises for 4 or fewer
clusters in IVGMMCUE, and for IVGMM exactly when the covariance is singular.
The test of #754 that fitted a model with 4 uncentered clusters expects an
error now.
j_stat said that an InvalidTestStatistic is returned if the covariance of the moment conditions that is inverted is singular, which can happen with too few clusters. IVGMM.fit and IVGMMCUE.fit raise a ValueError in that case instead. The statistic can only be invalid now at the final estimates of IVGMMCUE, if the covariance is nonsingular at the first step estimates, where it is tested, and singular at the final estimates.
…sing IVGMM test_centered_j_cue_with_too_few_clusters fitted an IVGMMCUE with center=False and 5 clusters for 5 moment conditions to check that the centered J statistic is invalid when the uncentered fit is valid. That model cannot be estimated any more: the objective function of the CUE is constant with as many clusters as moment conditions, and the CUE needs 6 clusters, for which the centered covariance is not singular either. The two-step estimator can still be estimated with 5 uncentered clusters, and it is the case where the J statistic is valid and the centered J statistic is not, so the test checks it with IVGMM. It checks that the CUE raises with 5 clusters, and that the CUE with 6 clusters has a valid centered J statistic. The test of the centered J statistic of the CUE with a singular covariance of #754 is unchanged.
The class and fit documentation of IVGMM said that there have to be more clusters than moment conditions. A clustered estimate from G clusters has rank at most G, and at most G - 1 if the moment conditions are centered, so that an overidentified model with center=False can be estimated with as many clusters as moment conditions, and one with center=True cannot. The first step moment conditions of an exactly identified model add to zero, so that it needs more clusters than moment conditions with every value of center.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #785 +/- ##
========================================
Coverage 99.58% 99.58%
========================================
Files 128 128
Lines 24188 24599 +411
Branches 1809 1825 +16
========================================
+ Hits 24087 24497 +410
Misses 40 40
- Partials 61 62 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The weight matrix of two-step GMM is the inverse of an estimate of the
covariance of the moment conditions. If there are fewer clusters than moment
conditions (or too few observations) the estimate is singular, the inverse is
numerical noise, and so are the estimates. #754 made j_stat and c_stat
invalid and made IVGMMCUE raise a clear ValueError in this case, and left the
estimates of IVGMM as they were:
1.697 when it is 0.5, and its standard error is NaN (and 1.95e6 for the
constant); it is a LinAlgError instead if NumPy finds that the matrix is
exactly singular, which depends on rounding and so on the platform.
in it is not harmless: with 3 clusters and 3 moment conditions the
estimates are 9.7 away from the IV estimates, and the standard errors are
not numbers. The first-step moment conditions of an exactly identified model
add to zero, so that the covariance of 3 moment conditions is singular with
3 clusters even if they are not centered.
IVGMM.fit now raises the same ValueError as IVGMMCUE.fit, with the message
built by the same function: it says that the covariance is singular, that
there are too few clusters (with their number and the rank that they allow) or
too few observations, and to use more clusters than moment conditions. It is
checked at every step before the covariance is inverted. It is not checked
with iter_limit=1, which does not use a covariance. IVGMMResults.c_stat
estimates a model with a moment condition for each tested variable, and
returns an invalid statistic instead of an error if that model cannot be
estimated, as it did for a LinAlgError. j_stat of an IVGMM is never invalid
now, since the model is not estimated.
The fits that are valid are not changed. Params, cov, std_errors, j_stat,
centered_j_stat and c_stat of 600 random models (IVGMM and IVGMMCUE, the four
weight types, observation weights, three covariances, units of the
instruments from 1e-2 to 1e2, and cluster counts around the number of moment
conditions) are the same to the last bit on origin/main and here for the 561
that are estimated on both. The 39 that raise here are clustered models with
no more clusters than moment conditions: 19 raised on origin/main too, and the
20 that did not had an invalid J statistic, and NaN or enormous standard
errors.
Tests: the tests of #754 for the J and C statistics with too few clusters
(and the small samples) accepted a result or a LinAlgError from fit, and
returned if there was none. They test the ValueError and its message now, and
the values of the statistics for the models that can be estimated. New tests
check that the message is that of IVGMMCUE except for the sentence that says
why the inverse is needed; that exactly identified models raise with 2 and 3
clusters and are the IV estimates with 4 or more; that every step is checked,
and the initial weight does not avoid the check; that iter_limit=1 is not
changed; and that the params, std errors, J, centered J and C of seven valid
models are those of origin/main (and the params and J are the definitions).