Skip to content

Verify HSP* and Update RegressTest to run under Pandas 3.0.0 #209

Open
Open
Verify HSP* and Update RegressTest to run under Pandas 3.0.0#209

Description

@rburghol
rburghol
opened on Jan 27, 2026

Relevant to PRs #216 which achieved preliminary successful model runs under pandas 3.0.0, and eliminates deprecated pandas methods and argument type restrictions.

  • tests/convert/regression_base.py under pandas 3.0.0 fails with an error pandas.errors.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match).
  • RegressTest class imports HDF5 from src/hsp2/hsp2tools/HDF5.py whereas main.py imports HDF5 from src/hsp2hsp2io/hdf.py
    • Need to explore whether the hsp2io:HDF5:read_ts() and hsp2tools:HDF5:get_time_series() behave the same way
    • Need to understand if these need to be converged (recall a previous planning group meeting indicated this as important). @timcera
  • Verify: Closing the hdf5 at the end of the run eliminates the pytest error: ValueError: The file '/opt/model/HSPsquared/tests/test10/HSPFresults/test10.h5' is already opened, but not in read-only mode (as requested). in pandas >= 3.0.0
  • Why does HSP2 VOL timeseries have the misaligned timestamp? (see Table 1)

Misaligned SEDTRN timeseries produces pandas 3.0.0 test error.

  • The error that is thrown in pandas-3.0.0 tests -- either: pandas-2.3.3 does not produce this misalignment, or the indices have shifted from a simple sequential integer to a timestamp.
  • The time index im the hsp2 series is really wacked out
  • The test code that fails looks to find hsp2 timesteps where the flow volume is < 1.0e-4, and then sets the corresponding SSED values to 0.0. However, then it goes on to also try to set the same timeslots to 0.0 in the HSPF SSED timeseries. This causes an error because the timestamp indices are misaligned.
            idx_low_vol = ts_vol_hsp2 < 1.0e-4
            ts_hsp2.loc[idx_low_vol] = ts_hsp2.loc[idx_low_vol] = 0
            ts_hspf.loc[idx_low_vol] = ts_hspf.loc[idx_low_vol] = 0
  • The fact that the test TRIES this is an error IMO because I think we should not be trying to zero the HSPF SSED on days when the hsp2 has miniscule flow since this could paper over actual differences in the SSED simulation. I think it would be correct to set HSPF SSED to 0.0 when HSPF volume is < 1.0e-4.

Table 1: Timeseries output from hsp2 versus hspf showing a timestamp disagreement.

>>> ts_hsp2
index
1970-01-03 04:35:06.000    29.916803
1970-01-03 04:35:09.600    29.834137
1970-01-03 04:35:13.200    29.751753
                             ...
1970-01-03 13:21:57.600     0.000000
1970-01-03 13:22:01.200     0.000000
1970-01-03 13:22:04.800     0.000000
Name: SSED3, Length: 8784, dtype: float32
>>> ts_hspf
1976-01-01 01:00:00    29.916805
1976-01-01 02:00:00    29.834139
1976-01-01 03:00:00    29.751757
                         ...
1976-12-31 22:00:00     0.000000
1976-12-31 23:00:00     0.000000
1977-01-01 00:00:00     0.000000
Name: SSEDCLAY, Length: 8784, dtype: float64

Table 2: Excerpt of error text from https://github.com/respec/HSPsquared/actions/runs/21381288537/job/61548511129?pr=207

tests/test_regression.py:49: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tests/convert/regression_base.py:225: in check_con
    ts_hsp2, ts_hspf = self.validate_time_series(
tests/convert/regression_base.py:329: in validate_time_series
    ts_hspf.loc[idx_low_vol] = ts_hspf.loc[idx_low_vol] = 0
    ^^^^^^^^^^^^^^^^^^^^^^^^

...
E               pandas.errors.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match).

/opt/hostedtoolcache/Python/3.12.12/x64/lib/python3.12/site-packages/pandas/core/indexing.py:2677: IndexingError

Activity

  1. timcera commented on Jan 28, 2026

    @timcera
    timcera
    on Jan 28, 2026
    Contributor

    You need to edit pyproject.toml to have "pandas<3.0" instead of just "pandas". The pyproject.toml is used by pip to install and also to prepare a package for upload to pypi.org. The environment*.yml files are used by conda and also need to be edited.

  2. rburghol commented on Jan 28, 2026

    @rburghol
    rburghol
    on Jan 28, 2026
    ContributorAuthor

    Thanks @timcera !!!

  3. rburghol
    changed the title [-]Adapt to Pandas 3.0.0 changes and emergent data disagreements reported by test_regression[/-] [+]Verify HSP* and Update RegressTest to run under Pandas 3.0.0[/+] on Jan 29, 2026

  4. rburghol commented on Jan 31, 2026

    @rburghol
    rburghol
    on Jan 31, 2026
    ContributorAuthor

    @timcera I think I recall you raising the issue about the 2 HDF5 classes and how that could be a problem, and I think this issue raises the possibility that this is in fact causing a divergence. I think this shows the difference in timeseries indices is the result of the 2 different HDF5 classes, one of which is used in regression_test.py and the other which is used by main.py.:

    # Note: This code must be run from the tests dir due to importing
# the `convert` directory where the regression_base file lives
# todo: make this convert path passable by argument 
import os

import numpy
from hsp2.hsp2.main import *
from hsp2.hsp2.om import *
from hsp2.hsp2io.hdf import HDF5
from hsp2.hsp2io.io import IOManager
from hsp2.state.state import *

import numpy as np
from convert.regression_base import RegressTest

case = "test10specl"
base_case = "test10"
#case = "testcbp"
tdir = "/opt/model/HSPsquared/tests"
#import_uci(str(hsp2_specl_uci), str(temp_specl_h5file))
#run(temp_specl_h5file, saveall=True, compress=False)
fpath = "./test10specl/HSP2results/test10specl.h5"
# try also:
# fpath = './tests/testcbp/HSP2results/JL1_6562_6560.h5'
# sometimes when testing you may need to close the file, so try:
# f = h5py.File(fpath,'a') # use mode 'a' which allows read, write, modify
# # f.close()
hdf5_instance = HDF5(fpath)
io_manager = IOManager(hdf5_instance)

############# RegressTest data loader
## TEST CASE
test = RegressTest(case, threads=1)

params = ('RCHRES', 'SEDTRN', '001', 'SSEDCLAY', '2')
test.check_con(params = params)
operation, activity, id, constituent, tcode = params
ts_hsp2 = test.hsp2_data.get_time_series(operation, id, constituent, activity)
ts_hspf = test.get_hspf_time_series(params)

# begin checking code for validate_time_series()
if len(ts_hsp2) > len(ts_hspf):
    ts_hsp2 = ts_hsp2[0 : len(ts_hspf)]

ts_hsp2 = test.fill_nan_and_null(ts_hsp2)
ts_hspf = test.fill_nan_and_null(ts_hspf)

### special cases
# SEDTRN
ts_vol_hsp2 = test.hsp2_data.get_time_series(operation, id, "VOL", "HYDR")
ts_vol_hsp2 = test.fill_nan_and_null(ts_vol_hsp2)

idx_low_vol = ts_vol_hsp2 < 1.0e-4
ts_hsp2.loc[idx_low_vol] = ts_hsp2.loc[idx_low_vol] = 0
ts_hspf.loc[idx_low_vol] = ts_hspf.loc[idx_low_vol] = 0


siminfo["tindex"] = date_range(siminfo["start"], siminfo["stop"], freq=Minute(siminfo["delt"]))[1:]
siminfo["steps"] = len(siminfo["tindex"])

params = ('RCHRES', 'SEDTRN', '001', 'SSEDCLAY', '2')
operation, activity, id, constituent, tcode = params

params = ('RCHRES', 'HYDR', '001', 'IVOL', '2')
segment = operation[0] + id 
operation, activity, id, constituent, tcode = params

tsivol_io = io_manager.read_ts(Category.RESULTS, operation, segment, activity)
tsivol_tools = test.hsp2_data.get_time_series(operation, id, "IVOL", activity)
np.quantile(tsivol_io.IVOL, [0,0.25,0.5,0.75,1.0])
np.quantile(tsivol_tools, [0,0.25,0.5,0.75,1.0])
tsivol_io.index
tsivol_tools.index

    Output:

    • quantiles match (as do values in order)
    • indices are misaligned -- hsp2.hsp2tools.HDF5 has decimal hours
    array([0.        , 0.        , 0.0644508 , 0.49963082, 2.85344005])
>>> np.quantile(tsivol_tools, [0,0.25,0.5,0.75,1.0])
array([0.        , 0.        , 0.0644508 , 0.49963082, 2.85344005])
>>> tsivol_io.index
DatetimeIndex(['1976-01-01 01:00:00', '1976-01-01 02:00:00',
               '1976-01-01 03:00:00', '1976-01-01 04:00:00',
               '1976-01-01 05:00:00', '1976-01-01 06:00:00',
               '1976-01-01 07:00:00', '1976-01-01 08:00:00',
               '1976-01-01 09:00:00', '1976-01-01 10:00:00',
               ...
               '1976-12-31 15:00:00', '1976-12-31 16:00:00',
               '1976-12-31 17:00:00', '1976-12-31 18:00:00',
               '1976-12-31 19:00:00', '1976-12-31 20:00:00',
               '1976-12-31 21:00:00', '1976-12-31 22:00:00',
               '1976-12-31 23:00:00', '1977-01-01 00:00:00'],
              dtype='datetime64[us]', length=8784, freq='60min')
>>> tsivol_tools.index
DatetimeIndex([       '1970-01-03 04:35:06', '1970-01-03 04:35:09.600000',
               '1970-01-03 04:35:13.200000', '1970-01-03 04:35:16.800000',
               '1970-01-03 04:35:20.400000',        '1970-01-03 04:35:24',
               '1970-01-03 04:35:27.600000', '1970-01-03 04:35:31.200000',
               '1970-01-03 04:35:34.800000', '1970-01-03 04:35:38.400000',
               ...
               '1970-01-03 13:21:32.400000',        '1970-01-03 13:21:36',
               '1970-01-03 13:21:39.600000', '1970-01-03 13:21:43.200000',
               '1970-01-03 13:21:46.800000', '1970-01-03 13:21:50.400000',
                      '1970-01-03 13:21:54', '1970-01-03 13:21:57.600000',
               '1970-01-03 13:22:01.200000', '1970-01-03 13:22:04.800000'],
              dtype='datetime64[ns]', name='index', length=8784, freq=None)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions