Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
07a948e
Work in progress. Initial commit
bikegeek Apr 1, 2026
2988c61
dev version of config file
bikegeek Apr 1, 2026
d5b99c0
Removed plottable submodule
bikegeek Apr 1, 2026
9c57efb
All work-in-progress png images for scorecard
bikegeek Apr 1, 2026
831b2e3
Moved column checking to its own method
bikegeek Apr 1, 2026
21cd478
Added support for filtering data
bikegeek Apr 2, 2026
f748bd7
Cleaning up subset_data
bikegeek Apr 3, 2026
ef24fb8
More cleanup, move reformatting code into a method. Added code for c…
bikegeek Apr 3, 2026
15d08c6
Cleaned up params passed into METcalcpy scorecard, do NOT include fcs…
bikegeek Apr 10, 2026
93a33a2
Updated to remove the fcst_init_beg for this example and included the…
bikegeek Apr 10, 2026
42740d8
Added check if the fcst_var is missing in the config file for filtering
bikegeek Apr 10, 2026
2fbe171
Updated with information needed for locating the image files
bikegeek Apr 13, 2026
1389a4a
Added support for generating the table. Additional work needed on th…
bikegeek Apr 13, 2026
89dade7
Issue #567 Remove need for defining the derived series. This is now …
bikegeek Apr 22, 2026
06f1d39
Issue #567 Generate the derived series information needed by METcalcp…
bikegeek Apr 22, 2026
497f436
Merge branch 'develop' of https://github.com/dtcenter/METplotpy into …
bikegeek May 13, 2026
1df58a4
config file for development using subset of GEFS data from NOAA (only…
bikegeek Jul 29, 2026
e92245e
Some cleaning up, added support for generating config values to input…
bikegeek Jul 29, 2026
9b9f980
png for scorecard symbols
bikegeek Aug 15, 2026
a893d55
Merge branch 'develop' of https://github.com/dtcenter/METplotpy into …
bikegeek Aug 15, 2026
08fc9e4
Delete metplotpy/external/Screenshot 2026-03-09 at 09.48.24.png
bikegeek Aug 15, 2026
11f3628
Delete metplotpy/external/unhyphenated_HT_RT copy.png
bikegeek Aug 15, 2026
a8e30c3
Delete metplotpy/external/cheatsheets-1.png
bikegeek Aug 15, 2026
2ec2cae
Delete metplotpy/external/down_red.xcf
bikegeek Aug 15, 2026
c3e139b
Delete metplotpy/external/up_small_green.xcf
bikegeek Aug 15, 2026
e2b1409
Delete metplotpy/external/images/keep_pngs/blue_square.xcf
bikegeek Aug 15, 2026
439bddf
Delete metplotpy/external/images/keep_pngs/gray_square.xcf
bikegeek Aug 15, 2026
2dd9fe4
Delete metplotpy/external/images/keep_pngs/green_square.xcf
bikegeek Aug 15, 2026
43d692c
remove unecessary png file
bikegeek Aug 15, 2026
c9c3f0b
Merge branch 'feature_567_scorecard' of https://github.com/dtcenter/M…
bikegeek Aug 15, 2026
d20e8ac
Remove unused graphics
bikegeek Aug 16, 2026
17ee077
Plottable third-party library for creating tables
bikegeek Aug 16, 2026
b3f8f68
Merge branch 'feature_567_scorecard' of https://github.com/dtcenter/M…
bikegeek Aug 16, 2026
fb3a906
remove more unused images
bikegeek Aug 16, 2026
cbcdca2
Remove entire dir of unused images
bikegeek Aug 16, 2026
4a29661
Added plottable library as submodule for generating tables
bikegeek Aug 16, 2026
e888ec9
upside-down/downward pointing red triangles
bikegeek Aug 16, 2026
0664a27
update reading/writing of csv to not include index
bikegeek Aug 16, 2026
3af437b
modify table settings
bikegeek Aug 16, 2026
c9fbbe2
Add missing config params for calculating METcalcpy scorecard stats
bikegeek Aug 17, 2026
9076412
Clean up unused code, modify formatting of columns, added subtitle
bikegeek Aug 18, 2026
c138617
Merge branch 'develop' of https://github.com/dtcenter/METplotpy into …
bikegeek Oct 20, 2026
42efc53
do not ignore deprecation warning
bikegeek Oct 20, 2026
d1fb1fe
Retooled scorecard_plot.py to increase functionality and switching to…
michelleharrold Oct 28, 2026
0cf6c62
New file for viewing scorecard
bikegeek Nov 6, 2026
8aef882
Merge branch 'develop' of https://github.com/dtcenter/METplotpy into …
bikegeek Nov 7, 2026
2eb4c66
Merge branch 'feature_567_scorecard' of https://github.com/dtcenter/M…
bikegeek Nov 7, 2026
c4e90bd
Added event equalizer support
bikegeek Nov 11, 2026
edf11f4
Merge branch 'develop' into feature_567_scorecard
bikegeek Nov 12, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "metplotpy/external/plottable"]
path = metplotpy/external/plottable
url = https://github.com/znstrider/plottable/
Empty file added metplotpy/external/__init__.py
Empty file.
1 change: 1 addition & 0 deletions metplotpy/external/plottable
Submodule plottable added at 553f5b
204 changes: 204 additions & 0 deletions metplotpy/plots/scorecard/gefs_cnt_scorecard.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
#
# GEFS vs. AIGEFS SAL1L2 scorecard (command-line, no database)
#
# Run from the METplotpy repo root:
# python -m metplotpy.plots.scorecard.scorecard_plot <this file>
#
# Pipeline: reformat .stat files -> subset -> per-case statistics -> p-values
# -> draw scorecard (scorecard_render.py, matplotlib)
#

#
# OUTPUT and LOG DIRECTORIES
#

# All output will reside in this specified output directory, including the
# intermediate files and the finished scorecard (<output_dir>/scorecard.png).
output_dir: '/glade/derecho/scratch/harrold/scorecard/minnas_work/output'

# A single log file will be generated, capturing logging information from
# all intermediate steps
log_dir : '/glade/derecho/scratch/harrold/scorecard/minnas_work/logs/'
log_filename: noaa_gefs.log
log_level: 'DEBUG'

#
# INPUT FILES
#

# Where the MET .stat files live. One directory is read for every combination of
# the ${...} variables below: ${base} (top directory), ${model} (each entry of
# 'model') and ${init} (each date from ymd_start to ymd_end, formatted with 'init').
# NOTE: the words "base", "model" and "init" are substituted as plain text, so
# they must not appear anywhere else in the paths (e.g. a folder called "init_data").
#met_stat_input: '/Users/minnawin/Python_Scorecard_Dev/Data_Reformatter/point_stat_HRRR_RRFS/sample_data/all_dates'
met_stat_input: '${base}/${model}.${init}'
base: '/glade/derecho/scratch/harrold/scorecard/test_data/GEFS_data'

# Which files to read from each directory: one glob pattern or a list of them,
# matched against the file name. Optional: if left out, every *.stat file is read.
# Use this to keep grid-to-grid, grid-to-obs and precip files apart. The run stops
# with an error if a directory is missing or nothing matches.
stat_file_pattern:
- '*grid2obs*.stat'

# What to do when a date directory, or the files you asked for, are missing for
# some model/date. Optional, default 'skip'.
# skip : drop that date for ALL models (so every model is compared on the same
# days), carry on, and print a warning at the end listing what was skipped.
# The scorecard subtitle then shows e.g. "30 of 31 days".
# error : stop at the first missing directory or file.
on_missing_input: 'skip'

# Directory-name prefixes for ${model}, i.e. the folders gefs.YYYYMMDD and
# aigefs.YYYYMMDD. These are folder names (lower case) and are NOT the same
# thing as the MODEL column values listed under subset_params below.
model:
- gefs
- aigefs

# Date format of the date directories (Python strftime), the first and last date
# (inclusive, written in that same format) and the step between dates in days.
init: "%Y%m%d"
increment: 1
ymd_start: '20260301'
ymd_end: '20260331'

#
# REFORMAT
#
# if data has unlabelled columns (i.e. a MET .stat file from MET point-stat or
# MET grid-stat tool). Set to True to reformat, False otherwise.
# Or if re-running using the same data and reformatting has already been done,
# then set to False for better performance.
reformat_needed: True

# Set to 'True' if the .stat data was already aggregated with the MET stat_analysis
# tool, or if the line type already has confidence-interval columns (ncu/ncl and/or
# btcu/btcl), as CNT does. Set to 'False' for raw point-stat/grid-stat output;
# the statistic is then calculated for every case (valid time) with METcalcpy
# sum_stat, and the significance test compares those cases.
has_confidence_stats: True

# MET line type of the input data (SAL1L2, SL1L2, CNT, ...)
linetype: 'CNT'

#
# SUBSETTING
#
# Filtering data based on model, masking regions, stat names, levels, variable of
# interest (and its values), and fcst_lead
subset_params:

# Column names in the MET output file. Values are kept only if they match.

# Forecast lead times in HHMMSS ('60000' is 6 h, '1200000' is 120 h).
# These become the scorecard columns.
'fcst_lead':
- '0'
- '60000'
- '120000'
- '240000'
- '360000'
- '480000'
- '600000'
- '720000'
- '840000'
- '960000'
- '1080000'
- '1200000'

# Vertical levels, e.g. P250 = 250 hPa. Each becomes a scorecard row.
'fcst_lev':
- 'P850'
- 'P500'
- 'P250'

# Values of the MODEL column. ORDER MATTERS: the first model is "series 1"
# (model A) and the second is "series 2" (model B). A green up-triangle means
# model A is better than model B.
'model':
- 'AIGEFS'
- 'GEFS'

# Forecast variables. Each becomes a scorecard row group.
'fcst_var':
- 'UGRD'
- 'VGRD'

# Observation type. Use exactly one (e.g. ADPUPA = upper-air, ADPSFC = surface,
# GDAS for grid-to-grid). Mixing types in one scorecard is not supported.
'obtype':
- 'ADPUPA'

# Independent variable name and corresponding value(s): the verification region.
indep_variable: 'vx_mask'

indep_values:
- 'CONUS'

# A list of one or more statistics of interest (line type specific).
# SAL1L2: MAE works today. ANOM_CORR (and ANOM_CORR_RAW, RMSFA, RMSOA) are derived
# from partial sums and need the METcalcpy/METdataio derive-step fixes first; with
# fail_on_missing_stats: True the run stops with a clear error if one is listed
# before those fixes are in.
stats_list:
- 'MAE'
- 'ANOM_CORR'

# Stop with an error if a statistic in stats_list is missing from the results
# (it was filtered out or could not be computed). Set to False to log the error
# and still draw whatever is available. Optional, default True.
fail_on_missing_stats: True

#
# CONFIDENCE INTERVALS via agg_stat.py
#
# NOTE: this setting is not read by scorecard_plot.py at present; it is kept so
# existing configs stay valid. Which route is used is set by has_confidence_stats.
calculate_confidence_intervals: True

#
# SCORECARD STATISTICS (p-values)
#

# Append to the scorecard output file with successive runs
append_subsequent: False

# Method to calculate p-values: 'NCAR' (t-test on the per-case differences, same as
# METviewer) or 'EMC'. Use NCAR. The EMC method decides which model is better from
# the statistic's NAME, and MAE is not among the names it knows (ME, BIAS, RMSE,
# RMSVE), so its better/worse direction for MAE should be treated as unverified.
pval_method: 'EMC'

# Number of days used by the EMC method only (ignored with NCAR)
sample_size: 120


#
# SCORECARD PLOT
#
# The scorecard is drawn with matplotlib (scorecard_render.py) and written to
# <output_dir>/scorecard.<format>. The default colors follow METviewer: green =
# model A better, pink = model B better, gray = no significant difference,
# hatched = no data; darker shades are 95 / 99 / 99.9 % significance.

# Title for scorecard (generated automatically as "<model A> vs. <model B> Scorecard"
# if this is left unspecified)
scorecard_title: 'Scorecard GEFS and AIGEFS'

# Optional look-and-feel settings. Every key is optional; anything left out uses
# the default shown. Unknown keys give a warning.
#scorecard_style:
# title: 'AIGEFS vs. GEFS: Winds at 250 hPa' # overrides scorecard_title
# subtitle: 'March 2026, CONUS upper-air' # default: line type, region, obs type, dates
# palette: metviewer # metviewer | colorblind (blue/orange)
# show_values: false # print the signed confidence (1 - p) in each cell
# markers: true # triangles showing significance level
# formats: [png] # any of png, pdf, svg
# dpi: 200
# label_map: # display names for the row labels
# UGRD: 'U wind'
# VGRD: 'V wind'
# P250: '250 hPa'
# model_names: ['AIGEFS', 'GEFS'] # default: the two entries of subset_params model
118 changes: 118 additions & 0 deletions metplotpy/plots/scorecard/gefs_ecnt_scorecard.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@

#
# SOURCE CODE DIRECTORY
#

# Top directory where the source code for METplotpy resides.
# Used for locating the images used in the scorecard table
base_dir: '/Users/minnawin/Python_Scorecard_Dev'

#
# OUTPUT and LOG DIRECTORIES
#


# All output will reside in this specified output directory
output_dir: '/Users/minnawin/Python_Scorecard_Dev/output'

# A single log file will be generated, capturing logging information from
# all intermediate steps
log_dir : '/Users/minnawin/Python_Scorecard_Dev/logs/'
log_filename: noaa_gefs.log
log_level: 'DEBUG'

# MET .stat input files
#met_stat_input: '/Users/minnawin/Python_Scorecard_Dev/Data_Reformatter/point_stat_HRRR_RRFS/sample_data/all_dates'
met_stat_input: '${base}/${model}.${init}'
base: '/Users/minnawin/Python_Scorecard_Dev/NOAA_data'
model:
- gefs
- aigefs
init: "%Y%m%d"
increment: 1
ymd_start: '20260301'
ymd_end: '20260331'

#
# REFORMAT
#
# if data has unlabelled columns (i.e. a MET .stat file from MET point-stat or
# MET grid-stat tool). Set to True to reformat, False otherwise.
# Or if re-running using the same data and reformatting has already been done,
# then set to False for better performance.
reformat_needed: True

# Set to 'True' MET .stat file is aggregated using the MET stat-analysis tool,
# or if line type already has CI stats columns (ncu/ncl and/or btcu/btcl).
# Set to 'False' if data doesn't have CI stats.
has_confidence_stats: True
linetype: 'ECNT'

#
# SUBSETTING
#
# Filtering data based on model, masking regions, stat names, levels, variable of
# interest (and its values), and fcst_lead
subset_params:

# Column names in the MET output file
'fcst_lead':
- '0'
- '60000'
- '120000'
- '240000'
- '360000'
- '480000'

'fcst_lev':
- 'A6'

'model':
- 'AIGEFS'
- 'GEFS'

'fcst_var':
- 'APCP_06'

# independent variable name and corresponding value(s)
indep_variable: 'vx_mask'

indep_values:
- 'CONUS'

# a list of one or more statistics of interest (line type specific)
stats_list:
- 'ME'


#
# CONFIDENCE INTERVALS via agg_stat.py
#
# Python code will generate the necessary config information to pass to
# the METcalcpy agg_stat.py module. Some line types do not require this step.
# If the MET output does not have btcu/btcl or ncu/ncl columns, then set
# this value to True
calculate_confidence_intervals: True

#
# SCORECARD STATISTICS (p-values)
#

# Append to the scorecard output file with successive runs
append_subsequent: False

# Method to calculate p-values: 'EMC' or 'NCAR' (code currently uses NCAR)
pval_method: 'NCAR'

# Used in the EMC method of calculating p-values
sample_size: 10


#
# SCORECARD PLOT
#

# Title for scorecard (this can be automatically generated in the code if
# this is left unspecified)
scorecard_title: 'Scorecard HRRR and RRFS'

Loading
Loading