Skip to content

Feature 567 scorecard - #595

Merged
bikegeek merged 49 commits into
developfrom
feature_567_scorecard
Nov 13, 2026
Merged

bikegeek merged 49 commits into
developfrom
feature_567_scorecard

Conversation

@michelleharrold

Copy link
Copy Markdown
Contributor

Related to #567 (Create Python-based Scorecard). Companion PRs: dtcenter/METcalcpy#[N] (SAL1L2 support, #480) and dtcenter/METdataio#[N] (#434).

Builds on the prototype scorecard work started by Minna on this branch.

Summary

scorecard_plot.py

  • Retooled to increase functionality and general layout
  • Switched the plotting backend to matplotlib, consistent with the plotly removal in METplotpy 13.0.0 (Remove plotly dependency #555).

Example configurations

  • Adds metplotpy/plots/scorecard/gefs_cnt_scorecard.yaml (CNT line type).
  • Adds metplotpy/plots/scorecard/gefs_sal1l2_scorecard.yaml (SAL1L2 line type; requires the companion METcalcpy and METdataio SAL1L2 PRs).

Pull Request Testing

  • Describe testing already performed for these changes:

Generated scorecards on Derecho/Casper using both example configs (CNT and SAL1L2) with GEFS data. Ran tests using GEFS and AIGEFS data (provided by NOAA/OMD) over March 2026. Example output:

scorecard_example
  • Recommend testing for the reviewer(s) to perform, including the location of input datasets, and any additional instructions:

With the companion METdataio and METcalcpy branches on PYTHONPATH, run scorecard_plot.py with gefs_cnt_scorecard.yaml, then with gefs_sal1l2_scorecard.yaml. Input data: @bikegeek -- you can use the test GEFS and AIGEFS data on seneca.

Example of run commands on Casper:
module load conda
conda activate metplus-dev

ROOT=/glade/derecho/scratch/harrold/scorecard/minnas_work
export PYTHONPATH=$ROOT/METplotpy:$ROOT/METdataio:$ROOT/METcalcpy:$PYTHONPATH

cd $ROOT/METplotpy
python -m metplotpy.plots.scorecard.scorecard_plot
$ROOT/METplotpy/metplotpy/plots/scorecard/gefs_cnt_scorecard.yaml

  • Do these changes include sufficient documentation updates, ensuring that no errors or warnings exist in the build of the documentation? No, this needs extensive documentation of the YAML and functionality. This will be added on the coordinated release cycle.

    If yes, describe the new output and/or changes to the existing output:

  • Please complete this pull request review by 11/10

Pull Request Checklist

See the METplus Workflow for details.

  • Add any new Python packages to the METplus Components Python Requirements table.
  • Review the source issue metadata (required labels, projects, and milestone).
  • Complete the PR definition above.
  • Ensure the PR title matches the feature or bugfix branch name.
  • Define the PR metadata, as permissions allow.
    Select: Reviewer(s) and Development issue
    Select: Milestone as the version that will include these changes
    Select: Coordinated METplus-X.Y Support project for bugfix releases or METplotpy-X.Y.Z Development project for official releases
  • After submitting the PR, select the ⚙️ icon in the Development section of the right hand sidebar. Search for the issue that this PR will close and select it, if it is not already selected.
  • After the PR is approved, merge your changes. If permissions do not allow this, request that the reviewer do the merge.
  • Close the linked issue and delete your feature or bugfix branch from GitHub.

bikegeek and others added 30 commits April 1, 2026 12:24
…t_init_beg, this causes problems with creating cartesian coordinates. Filtering requires a fcst_vaar, updated in YAML config.
…reated in the code based on settings in the filtering section. Renamed the ndays setting to sample_size, to make this more clear to users who are not using climate data.
…y scorecard.py from filter settings in the YAML config file and renamed the ndays config value to sample_size.
… to METcalcpy agg_stat to calculate the aggregation statistics for the NOAA ECNT data.
@michelleharrold michelleharrold added this to the METplotpy-13.0.0 milestone Nov 4, 2026
@michelleharrold michelleharrold added reporting: DTC NOAA BASE NOAA Office of Atmospheric Research DTC Project required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project labels Nov 4, 2026
@github-project-automation github-project-automation Bot moved this to 🩺 Needs Triage in METplus-13.0 Development Nov 4, 2026
@bikegeek bikegeek moved this from 🩺 Needs Triage to 🔎 In review in METplus-13.0 Development Nov 7, 2026

@bikegeek bikegeek left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scorecard_render.py file was missing so I copied the version on casper and committed it.

I ran this on seneca (using GEFS data), but got error message related to broadcasting:

File "/d1/projects/Python_Scorecard/METcalcpy/metcalcpy/scorecard.py", line 422, in _calculate_diff_sig_emc
val2_minus_val1 = np.subtract(values_2, values_1)
ValueError: operands could not be broadcast together with shapes (112,) (120,)

In Claude, requested the scorecard to analyze:

Why it fails: the two models don’t have the same set of cases. I counted the CNT rows (CONUS, ADPUPA, UGRD/VGRD) in the raw .stat files: every scorecard cell has 120 AIGEFS cases but only 116 GEFS cases. Your other machine reported 112 vs 120, so that copy probably has a few more GEFS files missing. scorecard_plot.py turns off event equalization (self.event_equalize = False), so METcalcpy’s _calculate_diff_sig_emc subtracts the two arrays as they are, and they’re different lengths. The NCAR method would hit the same mismatch.

The proposed fix is to implement event equalization, which I allowed in **my local ** METplotpy and now the code runs without error (but does take over an hour to complete).

Claude flags issues with MAE and ANOM_CORR: incorrect triangle directions:

  1. EMC treats MAE as “bigger is better” (scorecard.py:438, :460)

_calculate_diff_sig_emc picks the direction from the stat’s name. Only ME*/BIAS* and RMSE*/RMSVE* are handled. Every other stat goes to the else branch, ds = (mean1 - mean2) / intvl, which means “larger is better”.
MAE is lower-is-better, so its sign comes out flipped: if AIGEFS has the larger MAE, it’s drawn as the better model.

  1. Every cell uses the first stat’s rule (:396, :447)

Both EMC and NCAR take the direction from self.params['list_stat_1'][0]. That’s always the first entry in stats_list, not the stat of the cell being computed.
With [MAE, ANOM_CORR], every cell, ANOM_CORR included, is judged as if it were MAE.

Method MAE triangles ANOM_CORR triangles
EMC (your current config) wrong correct (two errors cancel)
NCAR correct wrong

Proposed work-around (Claude):
As a workaround, you can list only one stat in stats_list, or one lower-is-better and one higher-is-better stat run separately, so the first-stat rule is right for every cell. The real fix is to take the stat from each cell and give EMC a direction for MAE.

Prompting for location of the error, and the error is in METcalcpy.

No changes were made in my local copy of METcalcpy scorecard.py, but the following comments were added to the code:

In _calculate_diff_sig_ncar():

NOTE (wrong stat): list_stat_1[0] is always the FIRST stat in
stats_list, not this cell's stat. With stats_list [MAE, ANOM_CORR],
ANOM_CORR cells are judged by MAE's "closer to 0 is better" rule, so
the model with the HIGHER anomaly correlation is drawn as worse.

and

NOTE (direction bug): ds > 0 means series 1 (model A) is better.
The better/worse direction is chosen from the stat NAME below:
ME*/BIAS* -> smaller |value| is better
RMSE*/RMSVE* -> smaller value is better
anything else (else branch) -> LARGER value is better
MAE (and MSE, BCMSE, ESTDEV, CRPS, ...) are lower-is-better but are
not listed, so they fall into the else branch and the sign is
FLIPPED: a model with the larger MAE is drawn as the better one
(triangle points the wrong way).
NOTE (wrong stat): list_stat_1[0] is always the FIRST stat in
stats_list, not the stat of the cell being computed. With
stats_list [MAE, ANOM_CORR] every cell, ANOM_CORR included, uses
MAE's rule. The stat should come from the derived series instead.

In _calculate_diff_sig_emc():

NOTE (direction bug): ds > 0 means series 1 (model A) is better.
The better/worse direction is chosen from the stat NAME below:
ME*/BIAS* -> smaller |value| is better
RMSE*/RMSVE* -> smaller value is better
anything else (else branch) -> LARGER value is better
MAE (and MSE, BCMSE, ESTDEV, CRPS, ...) are lower-is-better but are
not listed, so they fall into the else branch and the sign is
FLIPPED: a model with the larger MAE is drawn as the better one
(triangle points the wrong way).
NOTE (wrong stat): list_stat_1[0] is always the FIRST stat in
stats_list, not the stat of the cell being computed. With
stats_list [MAE, ANOM_CORR] every cell, ANOM_CORR included, uses
MAE's rule. The stat should come from the derived series instead.

NOTE: wrong sign for MAE and other lower-is-better stats (see above)

Question:

Do you want to proceed with updates to METcalcpy (addressing bug with MAE and ANOM_CORR) and METplotpy (adding event equalization support) to address the triangle direction and mismatch in cases?

@bikegeek

bikegeek commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

Summary:

Bugs identified by Claude in METcalcpy with respect to the triangle directions in MAE and ANOM_CORR.

Data has mismatched cases (Claude's terminology) which is remedied via event_equalization support in METplotpy scorecard_plot.py. This works on my local code, but I did not merge this, pending your assessment.

Painful detail is in the comment of the review.

@bikegeek

bikegeek commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Summary:

Bugs identified by Claude in METcalcpy with respect to the triangle directions in MAE and ANOM_CORR.

Data has mismatched cases (Claude's terminology) which is remedied via event_equalization support in METplotpy scorecard_plot.py. This works on my local code, but I did not merge this, pending your assessment.

Painful detail is in the comment of the review.

Running the scorecard plot with updates to METcalcpy to address the METcalcpy scorecard categorization results in a plot that looks like this (local changes, I did not merge these, pending further discussion):
fixed_scorecard

bikegeek
bikegeek previously approved these changes Nov 12, 2026

@bikegeek bikegeek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

created an issue in METcalcpy to address the EMC vs NCAR inconsistencies observed in MAE and Anom Correlation.

@bikegeek
bikegeek merged commit da1b816 into develop Nov 13, 2026
16 of 17 checks passed
@bikegeek
bikegeek deleted the feature_567_scorecard branch November 13, 2026 00:07
@github-project-automation github-project-automation Bot moved this from 🔎 In review to 🏁 Done in METplus-13.0 Development Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

reporting: DTC NOAA BASE NOAA Office of Atmospheric Research DTC Project required: FOR DEVELOPMENT RELEASE Required to be completed in the development release for the assigned project

Projects

Status: 🏁 Done

Development

Successfully merging this pull request may close these issues.

2 participants