Repository navigation
BUG: Fix issues in IV - #763
Merged
Merged
Conversation
…ered_j_stat IVGMMResults.centered_j_stat returned an InvalidTestStatistic, saying that the covariance of the moment conditions is singular, for models in which it is not. It used matrix_rank on the covariance, and the tolerance of matrix_rank is relative to the largest eigenvalue. When the variances of two moment conditions differ by a factor of about 1e16, which is the case if two instruments are measured in units that differ by a factor of 1e8, the smaller one is below the tolerance and the covariance is reported as singular. For a model with four instruments, one of which is multiplied by a constant, the J statistic is 7.6196 at every scale. The centered statistic is 7.906 for constants up to 1e7 and is invalid from 1e8 on, and the same for constants down to 1e-8. This occurred for the unadjusted, robust, kernel and clustered estimators, with and without weights, and when the scale was that of an exogenous regressor, which is also an instrument. The rank is now that of the correlation matrix of the moment conditions, which does not depend on the units. A covariance whose variances are not positive or that is not finite is also singular. The statistic is computed as before, and all of the existing reference values are unchanged. A covariance that is singular is still found, e.g., when there are too few clusters, with any scale. The tests that are added check the definition of the statistic for IVGMM and IVGMMCUE, without using the weight matrix classes, with weights, one-way clusters, a kernel and the small sample adjustment. The reference values from R do not cover continuously updating GMM with weights or a kernel, or weights with clusters. They also check continuously updating GMM with and without centering, and that the statistic is invalid when there are as many clusters as moment conditions. 32 tests of the units of the data fail without the change.
The covariance of IVJIVE is computed with the weight matrix classes of the GMM models. With cov_type="kernel" and no bandwidth they use a bandwidth of nobs - 2, which is not a consistent estimator, and the standard errors are too small. IV2SLS and IVLIML select the bandwidth using the data, and the documentation of IVJIVE says that its covariance estimators are the same as in the other IV models. For one simulated data set with 500 observations, 4 instruments and independent errors, the standard error of the coefficient on the endogenous variable is 0.050 with the default, 0.065 for IV2SLS and 0.066 for the robust covariance estimator of IVJIVE. Coverage of nominal 95% confidence intervals for the same coefficient, using 400 simulated data sets with 4 instruments, so that the standard error of the frequencies is 0.012: nobs errors IV2SLS, kernel IVJIVE, nobs - 2 IVJIVE, selected 250 iid 0.915 0.693 0.927 500 iid 0.925 0.667 0.943 500 AR(1), 0.5 0.932 0.703 0.943 1000 AR(1), 0.5 0.938 0.620 0.943 The bandwidth is now selected with the option of the kernel weight matrix that selects it using the scores, unless it is given. The bandwidth that is used is in cov_config of the results, as before, and a bandwidth of nobs - 2 can be used by passing it. IVGMM and IVGMMCUE are not changed. They document the default of nobs - 2, but it has the same effect there: the coverage of IVGMM is about 0.66 in the simulation of the second row. The first-stage tests of the rank of their results use that bandwidth when the covariance is a kernel. Two other options of the IVJIVE covariance estimators gave errors that were poor, and now give the errors of the other IV models: * An unknown kernel raised a KeyError. It now raises the ValueError of IV2SLS and IVLIML that the kernel is unknown, whether or not the bandwidth is given. * Two-way clusters, which IV2SLS and IVLIML support, raised a ValueError from NumPy about the dimensions of arrays. IVJIVE supports one-way clustering, and it now raises a ValueError that says so. A 2-d array with one column is accepted, as in the other models. The tests of the default bandwidth of IVJIVE in test_jive.py required nobs - 2, and are updated. The new tests check that the bandwidth is selected, with weights and for each kernel, that the first-stage tests of the rank use it, that it depends on the data, the coverage of the intervals, and the errors above.
The docstrings of cragg_donald, cragg_donald_f, kleibergen_paap and kleibergen_paap_f list their references as footnotes, ".. [1] ...", that the text never cites. Sphinx warns "Footnote [n] is not referenced" for each of them: 3, 1, 2 and 3 warnings for the four functions, which are 9 of the warnings of the documentation build in CI. A build of these docstrings alone with Sphinx 9.1 gives the same 9 warnings at the same lines before the change and none after. The references are a plain list now, as in the docstrings of centered_j_stat and IVJIVE, and the text cites the sources that it uses by author and year: Cragg and Donald (1993) for the statistic, Anderson (1951) for the test of the rank that uses the canonical correlations, and Stock and Yogo (2005) for the critical values. The test finds the footnotes that a docstring defines and does not cite, and the citations that it does not define, for these functions and for the same properties of FirstStageResults. It fails for the four functions without the change, and it keeps the references of each function.
… is requested kleibergen_paap computed the rk Wald statistic as well as the LM statistic, and discarded it. When the covariance of the Wald statistic was rank deficient, it warned that "the covariance of the Kleibergen-Paap statistic is rank deficient", about a statistic that was not requested. The scores of the Wald statistic are first-stage residuals, which sum to zero, and so their cluster sums have a covariance whose rank is one less than the number of clusters. The LM statistic needs ninstr - nendog + 1 clusters, and the Wald statistic needs one more. With exactly ninstr - nendog + 1 clusters the LM statistic is well defined, and kleibergen_paap and FirstStageResults.kleibergen_paap warned anyway. The rk Wald covariance is now only computed for the F statistic, by kleibergen_paap_f and FirstStageResults.kleibergen_paap_f, which warn if it is rank deficient as before. The values of all of the statistics are unchanged, and the LM statistic avoids about half of the work. The tests check, for several numbers of instruments and endogenous regressors, that the LM statistic does not warn with ninstr - nendog + 1 clusters and warns with one fewer, that the F statistic warns with ninstr - nendog + 1 clusters and not with one more, and the same through the results of a weighted model. They fail without the change.
Document what the first-stage tests of the rank do with weights and clusters, in the docstrings of cragg_donald, cragg_donald_f, kleibergen_paap, kleibergen_paap_f and the corresponding properties of FirstStageResults. The behavior has not changed, and each statement is checked by a test. * Weights. The functions use the weighted data: each array has to be multiplied by the square root of the observation weights, including the constant column of exog, which is then the root of the weights and not a column of ones. The sample size is the number of rows. The statistics do not change if all of the weights are multiplied by a constant, and the results of the IV models use weights that are scaled to have mean 1. A test checks the scale of the weights for the functions, using a scale from 1e-6 to 1e6, and for the models, that the constant has to be weighted, and that the statistics of a weighted model are the ones of the functions. * Clusters. cragg_donald does not use a covariance estimator, and so does not depend on the clusters of the model, which is checked by comparing the four covariance estimators. kleibergen_paap sums the weighted scores within the clusters in cov_config, with one element for each row, and does not support two-way clusters. The LM statistic needs at least ninstr - nendog + 1 clusters and the Wald F statistic needs one more, and they warn if there are fewer. The previous commit makes the count correct for the LM statistic. * The first-stage tests of the rank of the results are the same for IV2SLS, IVLIML, IVGMM, IVGMMCUE and IVJIVE, with and without weights, and for the unadjusted, robust and clustered covariance estimators. The warnings of kleibergen_paap now also say that the covariance is formed as a dense matrix with (ninstr * nendog)**2 elements, so that the memory grows with the square of the number of instruments, which matters for the many-instrument models that JIVE is used for. The Raises section includes clusters with the wrong number of elements. The sources of kleibergen_paap_f are cited in the text. The new tests pass without this change, since they check the behavior.
The example in the docstring of IVJIVE.from_formula is
np.log(wage) ~ 1 + exper + exper ** 2 + brthord + [educ ~ sibs]
In a formula, a ** 2 is the power of a set of terms, which is a again, and
not its square. The model that the example creates has no squared term. Its
parameters are Intercept, exper, brthord and educ. The example now uses
I(exper ** 2), and the notes say why. They also say that the formula
expressions have to be valid and that a constant has to be included with
"1 +", as the other from_formula docstrings do.
The examples of IVLIML, IV2SLS, IVGMM and IVGMMCUE have the same formula, and
are not changed here.
IVJIVE is also not in __all__ of linearmodels.iv.model, which lists the other
estimators, and is now.
The tests run the examples in the docstrings of IVJIVE and IVJIVE.from_formula,
check that the formula in the example has the squared term, as a parameter of
the model, and that IVJIVE is in __all__.
…ickling Add tests of the parts of IVJIVE and of the first-stage tests of the rank that the tests of the features did not check. They pass without any change to the code, since they found no defect in the behavior that they test. * from_formula with weights, when an exogenous variable has missing values, is the model of the data without those rows, with the weights of those rows also dropped, and the weights are used. Only the exogenous variables of a formula can have missing values. * predict with data, which does not need the dependent variable or the instruments, with exog and endog, and in sample, for a model with weights. * compare of IVJIVE with IV2SLS, IVLIML and IVGMM, with weights, from a dict or a list, with each precision and stars, and for models from formulas. * The summary with weights and the small sample adjustment, for each of the covariance estimators. * The first-stage tests of the rank of IVJIVE equal those of IV2SLS for each covariance estimator with and without weights, and the first stage, including its summary, works for each estimator and covariance estimator, with weights. * The results of IV2SLS, IVGMM, IVGMMCUE and IVJIVE can be pickled, including the first stage and the centered J statistic.
NormalTestStatistic, which pesaran_cd returns, had tests of a p-value, the critical values and the str and repr of a two-sided statistic. Add tests of the rest, which pass without a change to the code, since they found no defect: * The critical values are the quantiles of the normal distribution for tests of size 10%, 5% and 1%, two-sided and one-sided, and the p-value at a critical value is the size of the test. The one-sided critical values are smaller. The two-sided p-value is the p-value of a chi-squared statistic with 1 degree of freedom, and is symmetric. The one-sided p-values of a statistic and its negative add to 1. * The str and repr of a one-sided statistic, with and without a name, of a small p-value, and the p-values and critical values for infinite and NaN statistics, and for statistics that are NumPy scalars, 0-d arrays or ints. * NormalTestStatistic, WaldTestStatistic with and without a denominator degree of freedom, InvalidTestStatistic and InapplicableTestStatistic can be pickled with each protocol, copied and deep copied, and give the same strings, p-values and critical values, and the frozen distribution still works. This is needed to send results to another process. The docstring said that p-values are two-sided if two_sided is True, and did not say what the one-sided p-value and critical values are. They are for a test that rejects when the statistic is large: the p-value is the upper tail probability, and the critical values are the 0.9, 0.95 and 0.99 quantiles, which means that the p-value of a large negative statistic is close to 1. The str of a one-sided statistic does not say that it is one-sided, as before.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #763 +/- ##
==========================================
- Coverage 99.58% 99.58% -0.01%
==========================================
Files 110 112 +2
Lines 20328 20984 +656
Branches 1600 1640 +40
==========================================
+ Hits 20244 20897 +653
Misses 31 31
- Partials 53 56 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…ered_j_stat
IVGMMResults.centered_j_stat returned an InvalidTestStatistic, saying that the
covariance of the moment conditions is singular, for models in which it is not.
It used matrix_rank on the covariance, and the tolerance of matrix_rank is
relative to the largest eigenvalue. When the variances of two moment conditions
differ by a factor of about 1e16, which is the case if two instruments are
measured in units that differ by a factor of 1e8, the smaller one is below the
tolerance and the covariance is reported as singular.
For a model with four instruments, one of which is multiplied by a constant,
the J statistic is 7.6196 at every scale. The centered statistic is 7.906 for
constants up to 1e7 and is invalid from 1e8 on, and the same for constants
down to 1e-8. This occurred for the unadjusted, robust, kernel and clustered
estimators, with and without weights, and when the scale was that of an
exogenous regressor, which is also an instrument.
The rank is now that of the correlation matrix of the moment conditions, which
does not depend on the units. A covariance whose variances are not positive or
that is not finite is also singular. The statistic is computed as before, and
all of the existing reference values are unchanged. A covariance that is
singular is still found, e.g., when there are too few clusters, with any scale.
The tests that are added check the definition of the statistic for IVGMM and
IVGMMCUE, without using the weight matrix classes, with weights, one-way
clusters, a kernel and the small sample adjustment. The reference values from R
do not cover continuously updating GMM with weights or a kernel, or weights
with clusters. They also check continuously updating GMM with and without
centering, and that the statistic is invalid when there are as many clusters as
moment conditions. 32 tests of the units of the data fail without the change.