Repository navigation
Conversation
The comparison of the row labels of an input with the labels of dependent, introduced for panel models in #748, is needed by every other estimator. Move it from linearmodels/panel/model.py to linearmodels/shared/alignment.py so that it can be used everywhere: * is_default_index, is_labelled, labels_1d, differs, misaligned and check_row_alignment are the panel helpers, without the leading underscore. Panel models keep the rule that a DataFrame with a default index and named columns is labelled (the columns of a time by entity frame are the entities), through a thin wrapper, so their behavior is unchanged. * is_labelled and labels_1d also understand IVData (through the object it was built from, so an IVData made from an array or from a pandas object that has a default index has no labels, even after rows have been dropped). * row_labels returns the labels of the rows of an input, or None if it does not carry any, and check_alignment compares a set of named inputs with the labels of the reference. Both are used by the IV, absorbing, system and asset pricing models in the following commits. * The error names the input and says how to fix it: reindex the input to the index of dependent, or call .to_numpy() to pair the values by position. This extends the message of the panel models. The reference can be named, which is used for the factor models, where portfolios plays the role of dependent, and for the equations of a system.
Interaction.isnull combined the missing flags of cat and cont as pandas Series. The Series are aligned on their labels, so an Interaction in which only one of cat and cont carries labels (the other being an array, or None) had a union of the two sets of labels, 2 * nobs rows for nobs observations, and AbsorbingLS raised "operands could not be broadcast together" when it was used in interactions. Inputs without labels are paired by position, so combine the flags by position.
IV2SLS, IVLIML, IVGMM, IVGMMCUE, IVJIVE, OLS and AbsorbingLS pair the rows of their inputs by position. A pandas object that holds the same labels as dependent in a different order was silently attached to the wrong observations: reversing exog, instruments, weights, absorb or the clusters changed the results without an error. Apply the rule of #748 to these models. An input that carries labels (a pandas Series or DataFrame with an index other than the default 0, 1, ..., n-1, an xarray DataArray with coordinates, or IVData built from either) must use the same index, in the same order, as dependent, and a ValueError that names the input is raised if it does not. NumPy arrays and pandas objects with a default index carry no labels and are still paired by position. Nothing is compared if dependent has no labels. Only the labels of the rows are compared, the columns are variable names. The check covers * exog, endog, instruments and weights of the IV models; * clusters passed to fit, and in weight_config for IVGMM and IVGMMCUE (weight_type="clustered"). The clusters are compared with the observations that are used, so they must be given after observations with missing values have been dropped, as before. Clusters of another length still raise the "wrong nobs" error; * exog, absorb, interactions, weights and the clusters of AbsorbingLS. The cat and cont of an Interaction have to use the same labels if both have them. The labels are compared after the sizes of the inputs are validated, so the errors for inputs of the wrong length are unchanged, and before observations with missing values are dropped. from_formula takes all variables from the same DataFrame and is not affected, unless weights are given with labels. IVGMM.__init__ repeated the creation of the weight estimator of the base class (after creating a default one there). It now passes weight_type and weight_config to the base class, so that the clusters are checked in one place. Each model docstring has a note about the rule.
SUR, IV3SLS and IVSystemGMM pair the rows of the inputs of an equation by position, and the rows of the equations by position: the residuals of all of the equations in a row are treated as the errors of one observation. Inputs that hold the same labels as dependent in a different order were silently attached to the wrong observations, both within an equation and across the equations. Reordering exog, endog, instruments or weights of one equation changed the results without an error. Apply the rule of #748. In each equation, an input that carries labels (a pandas Series or DataFrame with an index other than the default 0, 1, ..., n-1, an xarray DataArray with coordinates, or IVData built from either) must use the same index, in the same order, as the dependent variable of the equation, and the dependent variables that carry labels must use the same index in all of the equations. The first dependent variable with labels is the reference across the equations. A ValueError that names the input and the equation is raised otherwise. NumPy arrays and pandas objects with a default index carry no labels and are still paired by position. Nothing is compared to a dependent variable that has no labels. The same rule applies to clusters passed to fit (SUR and IV3SLS), which are compared with the observations that are used in estimation. The labels are compared before observations with missing values are dropped. The formula interface takes all of its variables from the same DataFrame and is not affected, unless weights are given with labels. The multivariate_ls and multivariate_iv interfaces compare their dependent, exog, endog and instruments in the same way as the equations. Equations with different numbers of observations could be passed to the models. They could not be estimated, and failed in fit with an error from NumPy about the dimensions of the arrays to concatenate. Raise a clear ValueError that names the equations when the model is created. Each model docstring has a note about the rule.
TradedFactorModel, LinearFactorModel and LinearFactorModelGMM pair the rows of portfolios and factors by position: the return on the test portfolios in a period is paired with the factors in the same row. Factors that hold the same labels as portfolios in a different order were silently attached to the wrong periods, and the model was estimated with the wrong factors without an error. Apply the rule of #748. If portfolios and factors both carry labels (a pandas Series or DataFrame with an index other than the default 0, 1, ..., n-1, an xarray DataArray with coordinates, or IVData built from either), factors must use the same index, in the same order, as portfolios, and a ValueError that names factors is raised otherwise. NumPy arrays and pandas objects with a default index carry no labels and are still paired by position. from_formula takes the factors, and usually the portfolios, from the same DataFrame and is not affected, unless portfolios is passed with labels that differ from those of the DataFrame. Each model docstring has a note about the rule.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #756 +/- ##
==========================================
+ Coverage 99.58% 99.61% +0.02%
==========================================
Files 110 115 +5
Lines 20328 21697 +1369
Branches 1600 1657 +57
==========================================
+ Hits 20244 21613 +1369
Misses 31 31
Partials 53 53
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The comparison of the row labels of an input with the labels of dependent,
introduced for panel models in #748, is needed by every other estimator. Move
it from linearmodels/panel/model.py to linearmodels/shared/alignment.py so that
it can be used everywhere:
check_row_alignment are the panel helpers, without the leading underscore.
Panel models keep the rule that a DataFrame with a default index and named
columns is labelled (the columns of a time by entity frame are the entities),
through a thin wrapper, so their behavior is unchanged.
built from, so an IVData made from an array or from a pandas object that
has a default index has no labels, even after rows have been dropped).
carry any, and check_alignment compares a set of named inputs with the labels
of the reference. Both are used by the IV, absorbing, system and asset pricing
models in the following commits.
index of dependent, or call .to_numpy() to pair the values by position. This
extends the message of the panel models. The reference can be named, which
is used for the factor models, where portfolios plays the role of dependent,
and for the equations of a system.