BUG: Raise the missing index error from every group operation of Pane…

[bashtage]

…lData

#716 made PanelData.entity_ids and PanelData.time_ids raise a ValueError that says that the entity or time index contains missing values (NaN). The estimators that do not use those properties still failed with errors that do not say what is wrong, or did not fail.

Reproduction (reg1/repro_leads.py): a panel with 20 entities and 5 periods, and a NaN in the entity label of one row.

model                                  main                         now
PooledOLS, FirstDifferenceOLS          ValueError (#716)            same
PanelOLS(time_effects=True)            ValueError (#716)            same
PanelOLS(entity_effects=True)          LinAlgError: SVD did not     ValueError (#716)
                                       converge
PanelOLS(entity and time effects)      LinAlgError: SVD did not     ValueError (#716)
                                       converge in Linear Least
                                       Squares
RandomEffects                          ValueError: array must not   ValueError (#716)
                                       contain infs or NaNs

A NaN (or NaT) in the time label is covered for every model except PanelOLS(time_effects=True), which raised LinAlgError: SVD did not converge.

pandas encodes a missing label as -1 in the codes of the level. The effects demean by the groups of the index, which are found with groupby, and a groupby drops the rows with a missing label. The demeaned data have NaN in those rows, and the check of the rank of the regressors fails in the SVD. Other group operations of PanelData do not fail and are wrong. dummies uses the last label of the level for a row with a missing label, so that the row is in the last entity, and the entities and time periods of the 3-d view (values3d, and so first_difference) have the same problem.

The fix is a method, PanelData._check_index, with the message of #716. It is used by entity_ids and time_ids, as before, and now also by the operations that group by the index:

• demean, for the level of its group, and both for "both",
• mean, count and dummies, for the level of their group,
• the panel view that values3d and first_difference use, for both levels.

The levels that an operation does not use are not checked. An index with a missing label can still be used to construct a PanelData, as #716 tests.

I ran 52 combinations of the estimators with options (cov_type, clusters, weights, low_memory, use_lsdv, auto_df, count_effects, drop_absorbed, check_rank, other_effects) with a missing label in the entity index and in the time index. 28 of the 104 fits did not raise the ValueError on main, 26 with LinAlgError and 2 with RandomEffects' "array must not contain infs or NaNs". All of them do now. The estimates and the results of panels without a missing label are not changed: the panel tests pass (3,398 passed, including the new tests, 61 skipped and 47 xfailed).

The change is in panel/data.py only, so it does not touch the code that the input alignment branch changes in panel/model.py.

The new tests are in test_missing_index.py. They fit each estimator with the label of the entity and of the time missing, using integer labels and string and date labels, with weights, covariance estimators, low memory demeaning and formulas, and call each operation of PanelData. 52 of the 158 tests fail without the change.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.58%. Comparing base (208845b) to head (d902943).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #764   +/-   ##
=======================================
  Coverage   99.58%   99.58%           
=======================================
  Files         110      111    +1     
  Lines       20328    20414   +86     
  Branches     1600     1602    +2     
=======================================
+ Hits        20244    20330   +86     
  Misses         31       31           
  Partials       53       53           

Flag	Coverage Δ
adder	99.57% <100.00%> (+<0.01%)	⬆️
subtractor	99.57% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.