Repository navigation
Feature 567 scorecard - #595
Conversation
…lculating the scorecard statistics.
…t_init_beg, this causes problems with creating cartesian coordinates. Filtering requires a fcst_vaar, updated in YAML config.
… formatting and appearance.
…reated in the code based on settings in the filtering section. Renamed the ndays setting to sample_size, to make this more clear to users who are not using climate data.
…y scorecard.py from filter settings in the YAML config file and renamed the ndays config value to sample_size.
…feature_567_scorecard
… to METcalcpy agg_stat to calculate the aggregation statistics for the NOAA ECNT data.
…feature_567_scorecard
…ETplotpy into feature_567_scorecard
…feature_567_scorecard
… matplotlib. Also adding SAL1L2 and CNT example XMLs.
…feature_567_scorecard
…ETplotpy into feature_567_scorecard
There was a problem hiding this comment.
The scorecard_render.py file was missing so I copied the version on casper and committed it.
I ran this on seneca (using GEFS data), but got error message related to broadcasting:
File "/d1/projects/Python_Scorecard/METcalcpy/metcalcpy/scorecard.py", line 422, in _calculate_diff_sig_emc
val2_minus_val1 = np.subtract(values_2, values_1)
ValueError: operands could not be broadcast together with shapes (112,) (120,)
In Claude, requested the scorecard to analyze:
Why it fails: the two models don’t have the same set of cases. I counted the CNT rows (CONUS, ADPUPA, UGRD/VGRD) in the raw .stat files: every scorecard cell has 120 AIGEFS cases but only 116 GEFS cases. Your other machine reported 112 vs 120, so that copy probably has a few more GEFS files missing. scorecard_plot.py turns off event equalization (self.event_equalize = False), so METcalcpy’s _calculate_diff_sig_emc subtracts the two arrays as they are, and they’re different lengths. The NCAR method would hit the same mismatch.
The proposed fix is to implement event equalization, which I allowed in **my local ** METplotpy and now the code runs without error (but does take over an hour to complete).
Claude flags issues with MAE and ANOM_CORR: incorrect triangle directions:
- EMC treats MAE as “bigger is better” (scorecard.py:438, :460)
_calculate_diff_sig_emc picks the direction from the stat’s name. Only ME*/BIAS* and RMSE*/RMSVE* are handled. Every other stat goes to the else branch, ds = (mean1 - mean2) / intvl, which means “larger is better”.
MAE is lower-is-better, so its sign comes out flipped: if AIGEFS has the larger MAE, it’s drawn as the better model.
- Every cell uses the first stat’s rule (:396, :447)
Both EMC and NCAR take the direction from self.params['list_stat_1'][0]. That’s always the first entry in stats_list, not the stat of the cell being computed.
With [MAE, ANOM_CORR], every cell, ANOM_CORR included, is judged as if it were MAE.
| Method | MAE triangles | ANOM_CORR triangles |
|---|---|---|
| EMC (your current config) | wrong | correct (two errors cancel) |
| NCAR | correct | wrong |
Proposed work-around (Claude):
As a workaround, you can list only one stat in stats_list, or one lower-is-better and one higher-is-better stat run separately, so the first-stat rule is right for every cell. The real fix is to take the stat from each cell and give EMC a direction for MAE.
Prompting for location of the error, and the error is in METcalcpy.
No changes were made in my local copy of METcalcpy scorecard.py, but the following comments were added to the code:
In _calculate_diff_sig_ncar():
NOTE (wrong stat): list_stat_1[0] is always the FIRST stat in
stats_list, not this cell's stat. With stats_list [MAE, ANOM_CORR],
ANOM_CORR cells are judged by MAE's "closer to 0 is better" rule, so
the model with the HIGHER anomaly correlation is drawn as worse.
and
NOTE (direction bug): ds > 0 means series 1 (model A) is better.
The better/worse direction is chosen from the stat NAME below:
ME*/BIAS* -> smaller |value| is better
RMSE*/RMSVE* -> smaller value is better
anything else (else branch) -> LARGER value is better
MAE (and MSE, BCMSE, ESTDEV, CRPS, ...) are lower-is-better but are
not listed, so they fall into the else branch and the sign is
FLIPPED: a model with the larger MAE is drawn as the better one
(triangle points the wrong way).
NOTE (wrong stat): list_stat_1[0] is always the FIRST stat in
stats_list, not the stat of the cell being computed. With
stats_list [MAE, ANOM_CORR] every cell, ANOM_CORR included, uses
MAE's rule. The stat should come from the derived series instead.
In _calculate_diff_sig_emc():
NOTE (direction bug): ds > 0 means series 1 (model A) is better.
The better/worse direction is chosen from the stat NAME below:
ME*/BIAS* -> smaller |value| is better
RMSE*/RMSVE* -> smaller value is better
anything else (else branch) -> LARGER value is better
MAE (and MSE, BCMSE, ESTDEV, CRPS, ...) are lower-is-better but are
not listed, so they fall into the else branch and the sign is
FLIPPED: a model with the larger MAE is drawn as the better one
(triangle points the wrong way).
NOTE (wrong stat): list_stat_1[0] is always the FIRST stat in
stats_list, not the stat of the cell being computed. With
stats_list [MAE, ANOM_CORR] every cell, ANOM_CORR included, uses
MAE's rule. The stat should come from the derived series instead.
NOTE: wrong sign for MAE and other lower-is-better stats (see above)
Question:
Do you want to proceed with updates to METcalcpy (addressing bug with MAE and ANOM_CORR) and METplotpy (adding event equalization support) to address the triangle direction and mismatch in cases?
|
Summary: Bugs identified by Claude in METcalcpy with respect to the triangle directions in MAE and ANOM_CORR. Data has mismatched cases (Claude's terminology) which is remedied via event_equalization support in METplotpy scorecard_plot.py. This works on my local code, but I did not merge this, pending your assessment. Painful detail is in the comment of the review. |
bikegeek
left a comment
There was a problem hiding this comment.
created an issue in METcalcpy to address the EMC vs NCAR inconsistencies observed in MAE and Anom Correlation.

Related to #567 (Create Python-based Scorecard). Companion PRs: dtcenter/METcalcpy#[N] (SAL1L2 support, #480) and dtcenter/METdataio#[N] (#434).
Builds on the prototype scorecard work started by Minna on this branch.
Summary
scorecard_plot.pyExample configurations
metplotpy/plots/scorecard/gefs_cnt_scorecard.yaml(CNT line type).metplotpy/plots/scorecard/gefs_sal1l2_scorecard.yaml(SAL1L2 line type; requires the companion METcalcpy and METdataio SAL1L2 PRs).Pull Request Testing
Generated scorecards on Derecho/Casper using both example configs (CNT and SAL1L2) with GEFS data. Ran tests using GEFS and AIGEFS data (provided by NOAA/OMD) over March 2026. Example output:
With the companion METdataio and METcalcpy branches on
PYTHONPATH, runscorecard_plot.pywithgefs_cnt_scorecard.yaml, then withgefs_sal1l2_scorecard.yaml. Input data: @bikegeek -- you can use the test GEFS and AIGEFS data on seneca.Example of run commands on Casper:
module load conda
conda activate metplus-dev
ROOT=/glade/derecho/scratch/harrold/scorecard/minnas_work
export PYTHONPATH=$ROOT/METplotpy:$ROOT/METdataio:$ROOT/METcalcpy:$PYTHONPATH
cd $ROOT/METplotpy
python -m metplotpy.plots.scorecard.scorecard_plot
$ROOT/METplotpy/metplotpy/plots/scorecard/gefs_cnt_scorecard.yaml
Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? No, this needs extensive documentation of the YAML and functionality. This will be added on the coordinated release cycle.
If yes, describe the new output and/or changes to the existing output:
Please complete this pull request review by 11/10
Pull Request Checklist
See the METplus Workflow for details.
Select: Reviewer(s) and Development issue
Select: Milestone as the version that will include these changes
Select: Coordinated METplus-X.Y Support project for bugfix releases or METplotpy-X.Y.Z Development project for official releases