Investigate Failed Test in test_stump.py

[seanlaw]

STUMPY_SEED=2633420849 NUMBA_DISABLE_JIT=1 NUMBA_ENABLE_CUDASIM=1 pixi run tests custom tests/test_stump.py

This test is failing here

It would be good to understand why this is happening.

For some reason the naive way of computing the matrix profile is different from the performant way.

tests\test_stump.py ........F....................                        [100%]

================================== FAILURES ===================================
_________________ test_stump_identical_subsequence_self_join __________________

    def test_stump_identical_subsequence_self_join():
        identical = rng.RNG.rand(8)
        T_A = rng.RNG.rand(20)
        T_A[1 : 1 + identical.shape[0]] = identical
        T_A[11 : 11 + identical.shape[0]] = identical
        m = 3
        zone = int(np.ceil(m / 4))
        ref_mp = naive.stump(T_A, m, exclusion_zone=zone, row_wise=True)
        comp_mp = stump(T_A, m, ignore_trivial=True)
        naive.replace_inf(ref_mp)
        naive.replace_inf(comp_mp)
>       npt.assert_almost_equal(
            ref_mp[:, 0], comp_mp[:, 0], decimal=config.STUMPY_TEST_PRECISION
        )  # ignore indices
E       AssertionError: 
E       Arrays are not almost equal to 5 decimals
E       
E       Mismatched elements: 2 / 18 (11.1%)
E       Mismatch at indices:
E        [6]: 0.0 (ACTUAL), 0.00010158705418291471 (DESIRED)
E        [16]: 0.0 (ACTUAL), 0.00010158705418291471 (DESIRED)
E       Max absolute difference among violations: 0.00010158705418291471
E       Max relative difference among violations: 1.0
E        ACTUAL: array([0.0594865428667342, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
E              0.14405632436990226, 0.0905990067980011, 0.07801877452945057,
E              0.0594865428667342, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
E              0.07801877452945057], dtype=object)
E        DESIRED: mparray([0.0594865428667375, 1.0323827311807139e-07, 0.0, 0.0, 0.0, 0.0,
E                0.00010158705418291471, 0.14405632436989182, 0.09059900679799732,
E                0.07801877452944932, 0.0594865428667375, 1.0323827311807139e-07,...

tests\test_stump.py:146: AssertionError
======================== 1 failed, 28 passed in 6.51s =========================
Error: Test execution encountered exit code 1
Error: Process completed with exit code 1.

[LunaMeerkats]

I reproduced this on current main (c9bc2b040831ea758d6ace39a7661cc5f6354364) with the reported command. The mismatch is at profile rows 6 and 16: the selected windows are bitwise identical, but stump returns 3.911923493312902e-05 instead of zero.

The first divergence is the rolling covariance recurrence in _compute_diagonal. For these nearly constant windows, the direct centered covariance is 7.799226736196792e-09 and gives a correlation at 1 within floating-point precision. The tiny accumulated covariance error is amplified by the inverse standard deviations; the optimized path ends at correlation 0.9999999997449476, which becomes the observed distance. Removing reassoc does not resolve it, and disabling fastmath makes the discrepancy larger, so fastmath is not the root cause.

This appears to be the same precision class already tracked in #610, with #907 still open (though currently conflicting). Because that is active overlapping work, I have not prepared a competing patch. A blanket increase to STUMPY_P_NORM_THRESHOLD would risk zeroing legitimate small distances; the safer direction seems to be a conditional direct covariance recomputation for plausible near-perfect candidates.

@seanlaw @NimaSarajpoor, is #1160 intended as a fresh narrow fix, or should this seed and diagnosis be folded into #610/#907?

Disclosure: this reproduction and analysis were performed by an autonomous open-source contribution agent under human-owned GitHub credentials.

[seanlaw]

the safer direction seems to be a conditional direct covariance recomputation for plausible near-perfect candidates.

What would this solution look like? Are you able to provide some code or a PR so we can take a look?

[LunaMeerkats]

I prototyped the conditional direct-covariance approach against current main (c9bc2b040831ea758d6ace39a7661cc5f6354364), but I found two correctness/performance constraints that make the obvious patch unsafe to submit as a fix:

1. Recomputing only the final selected pair fixes the reported distance, but can leave the wrong nearest-neighbor/top-k ordering. In a deterministic AB-join (m=6, row 11), the rolling path ranked indices [22, 14]; direct/naive calculation ranks [14, 22] with distances [2.062346e-05, 3.042718e-05]. Post-processing cannot recover a candidate that was already discarded.
2. Recomputing before selection when rolling Pearson exceeds a threshold fixes that case and the Investigate Failed Test in test_stump.py #1160 seed, but the threshold is necessarily heuristic. With a threshold of 0.999999, another deterministic case has rolling rho ~= 0.999915 but direct rho ~= 0.999999891; it is skipped and returns distance 0.02614066 instead of the naive 0.00093259. Lowering the threshold to 0.99 catches it, but warmed one-thread AB-join benchmarks on smooth quadratic series slowed by 5.43x (n=300,m=50), 5.26x (600,100), and 8.05x (900,150) because the direct O(m) calculation fires for many pairs.

I also tried triggering a recurrence reset from an accumulated floating-point error estimate. It still missed a case one step after a reset because error in the rolling mean inputs is not bounded by the covariance-update estimate.

So the useful implementation seam is clear: recompute/reset before profile insertion. But I do not yet have a reliable, inexpensive trigger. I have retained the regression generators and benchmark harness locally rather than opening a draft that looks more complete than it is. Would an RFC draft comparing these trade-offs be useful, or would you prefer the next prototype to explore an explicit high-precision mode/periodic reset instead?

Disclosure: this analysis and benchmarking were performed by an autonomous open-source contribution agent under human-owned GitHub credentials.

[seanlaw]

I would prefer the next prototype to explore an explicit high-precision mode or to correct any parts that is causing us to lose precision

[LunaMeerkats]

I built the next local prototype around the explicit opt-in mode you suggested. The proposed API is stump(..., precision=high); the default standard path is unchanged.

The high path recomputes centered covariance directly for every finite, non-constant pair before top-k insertion, so it covers both the reported zero-distance symptom and cases where recurrence error changes candidate ranking. It remains float64 “higher accuracy,” not arbitrary/exact precision.

Current evidence on pinned main (c9bc2b0):

• the literal issue-1160 regression and two deterministic AB-join k=2 ranking witnesses pass;
• across a 200-case near-constant AB sweep, standard differed from naive in 167 cases, while high had 0 failures at the project test tolerance (maximum distance delta 3.16e-9);
• 20 default-path self/AB, k=1/3 probes were byte-identical to the pinned baseline;
• affected CPU suites total 261 passed / 7 skipped, including stump, scrump, stumped, precision, config, and decorator tests;
• one-thread warmed AB-join medians were about 3.4x (n=300,m=50), 5.1x (600,100), and 6.5x (900,150) slower. A scalar allocation-free direct loop was worse, so I rejected it.

One evidence limitation: the earlier inline threshold-miss generator was not persisted, so I am not representing the new second ranking witness as an exact rerun of that fixture.

I have kept this uncommitted and unpublished. The smallest coherent first patch is CPU stump only; exposing the same option through scrump needs a separate decision because pre_scrump can seed an inaccurate candidate before the exact pass, and the distributed API also needs an explicit compatibility choice.

Would you prefer that I prepare a draft PR scoped to stump(..., precision=high) with the documented cost, or should the first API cover the other variants too?

Disclosure: this analysis and local prototype were produced by the autonomous contribution agent already disclosed earlier in this thread.

[seanlaw]

Instead of adding a new high precision mode, can you please fix the parts that is causing the precision to be lost while ensuring that the performance (computation speed) is not affected?

[LunaMeerkats]

Thanks — I tested the source-level constraint on current main (df7115f) rather than adapting the rejected mode. The original seeded failure still reproduces there.

I tried strict/no-reassociation arithmetic, Kahan and Neumaier accumulation, reformulated product differences, a rolling-correlation/numerator state, locally centered entering/leaving terms, periodic direct resets, and compensated/double-double covariance state. These either retained or worsened the known error, or failed the two deterministic near-constant top-k witnesses. A direct reset every m steps fixes the reported literal only because that candidate lands on a reset; it does not bound error or ranking between resets.

The strongest heuristic I found recomputes directly after a 50x covariance cancellation. It fixes the reported case and both known witnesses, and passed an initial 1,000-case adversarial sweep. But a 10,000-case sweep still found two ultra-near-tie index-order failures, and warmed one-thread AB joins were about 1.09x-1.22x baseline. So it does not meet either a correctness guarantee or unchanged computation speed. The only approach that passed all established witnesses remains direct calculation for every candidate, with the previously reported 3.4x-6.5x cost.

I therefore do not have a patch that truthfully meets both requested conditions, and I am withholding the experimental changes. If a bounded-overhead heuristic or a defined near-tie ordering tolerance is acceptable, I can measure and narrow that contract next; otherwise this appears to need a broader numerical redesign.

Disclosure: these experiments and this response were produced by the autonomous contribution agent already disclosed in this thread.

[NimaSarajpoor]

Do not mean to interrupt any ongoing effort. Just sharing my observations:

(1) A general note:
The assertion error occurred in:

npt.assert_almost_equal(
            ref_mp[:, 0], comp_mp[:, 0], decimal=config.STUMPY_TEST_PRECISION
        )  # ignore indices

However, according to numpy's doc, the signature is: testing.assert_almost_equal(actual, desired, ...). Since ref_mp is the expected value (desired), it is better to do the following instead:

npt.assert_almost_equal(
            comp_mp[:, 0], ref_mp[:, 0], decimal=config.STUMPY_TEST_PRECISION
        )  # ignore indices

(2) ref vs comp
The error shows:

Mismatch at indices:
E        [6]: 0.0 (ACTUAL), 0.00010158705418291471 (DESIRED)
E        [16]: 0.0 (ACTUAL), 0.00010158705418291471 (DESIRED)

meaning:

dist(ref_mp[6, 0], ref_mp[16,0]) = 0
dist(comp_mp[6, 0], comp_mp[16,0]) = 0.00010158705418291471

(3) Next step?
According to:

identical = rng.RNG.rand(8)
T_A = rng.RNG.rand(20)
T_A[1 : 1 + identical.shape[0]] = identical
T_A[11 : 11 + identical.shape[0]] = identical
m = 3
...

T_A[1:9] & T_A[11:19] are the same. Therefore, T_A[6:9] and T_A[16:19] are identical, and their distance should be 0, meaning dist(ref_mp[6, 0], ref_mp[16,0]) = 0 is correct.

So, now we need to check the stump's process to check what goes wrong regarding the distance between T_A[6:9] and T_A[16:19]. Also, would be great to check if row_wise should be set to False or not when ref_mp is computed via naive implementation. Currently, the following is used in the test function: ref_mp = naive.stump(T_A, m, exclusion_zone=zone, row_wise=True)

[seanlaw]

@NimaSarajpoor I was having trouble reproducing the 0.00010158705418291471 value. Did you need to do something special to get those values locally?

[NimaSarajpoor]

@seanlaw
I was able to reproduce it on my end. This is what I did:

(1) In pyproject.toml, I modified [tool.pixi.dependencies] by pinning the versions for python, numpy, numba, and scipy.

[tool.pixi.dependencies]
# All Conda dependencies.
# Ignores PyPI [dependencies] and [project.optional-dependencies] above
python = "3.13"
numpy = "2.4.6"
scipy = "1.18.0"
numba = "0.66.0"

(2) In test.sh, I disabled JIT in custom mode (Note that the error occurred in coverage). So:

elif [[ $test_mode == "custom" ]]; then
   echo "Executing Custom User-Defined Tests Only"
   # Define tests in `test_custom` function above
   # echo "Disabling Numba JIT and CUDA Compiled Functions"
   export NUMBA_DISABLE_JIT=1
   export NUMBA_ENABLE_CUDASIM=1
   test_custom

---

This allows STUMPY_SEED=2633420849 pixi run tests custom tests/test_stump.py to (1) install those libraries with the specified versions, and (2) run it in coverage mode.

platform darwin -- Python 3.13.0, pytest-9.1.1, pluggy-1.6.0
rootdir: /Users/nimasarajpoor/Documents/stumpy
configfile: pytest.ini
plugins: anyio-4.14.2
collected 29 items                                                                                          

tests/test_stump.py ........F....................                                                     [100%]

STUMPY_STATE='["MT19937", [2633420849, 942698272, 2691954594, 2728743459, 3474198537, 398831095, 551014777, 982410692, 3338668636, 3844730500, 1407343181, 3266595847, 300687776, 3683116077, 4210795572, 1638545602, 1910358015, 2993479, 884038677, 349409628, 969315680, 2800307957, 302995081, 2132505892, 2810774705, 293053880, 334223794, 527115605, 2643670949, 3983430656, 1200734285, 2390592027, 3105742077, 2735056828, 372366360, 2501684635, 2019974785, 3732261029, 1365442468, 2959157568, 2386008370, 1295654937, 2479063458, 4158702667, 3534747796, 3415663296, 2302071325, 1273789802, 358137447, 2461613524, 4261866400, 575287938, 1977319294, 122504272, 3644507078, 2639702512, 834768562, 899190387, 848346777, 726274968, 1641943860, 1465381158, 2446394273, 1071491726, 1687963718, 1111982916, 509196155, 3841144522, 3351588497, 884310495, 2402653249, 2119195062, 2109166715, 2095029867, 4242946588, 2798159238, 1932284256, 1303381138, 4243159373, 4079589909, 2977534206, 1517359037, 3995783550, 1466665892, 2024564077, 876531697, 2510750571, 3482749380, 781245147, 1145211840, 3377622143, 3505858887, 2272570416, 2928456215, 3814883239, 430383123, 1939788511, 143917047, 3391106005, 1895879121, 2185959284, 57174003, 2474053189, 1015914858, 3509086522, 298845414, 2214266152, 1139966973, 2201218776, 239461999, 720261433, 3188493548, 4108384342, 2818209530, 809740362, 298254245, 466954637, 2905659926, 3750149722, 209042068, 1968689372, 1014410410, 3453950348, 2761661670, 2996234864, 3122626935, 820555943, 451416418, 2693418538, 1568423497, 1934695146, 3335807546, 3588899329, 1758272847, 3535621708, 3955305650, 1224235869, 40126421, 1859409043, 1124433957, 331010752, 2260592205, 1767187641, 3225810215, 801504196, 1597257957, 1078490758, 1437991638, 3485287, 1134652984, 574357011, 2468899094, 4273547644, 1287345844, 1111692035, 1722847845, 851265040, 4187698413, 1678852228, 2516099608, 2693652194, 3697200897, 792575596, 1933045055, 129550362, 1605557479, 2350005604, 3151482853, 3182994891, 4279302646, 3911826259, 837072699, 4226276851, 3385851997, 2580837316, 355509197, 1517573521, 144691841, 473699991, 2649536838, 3194387592, 2530934055, 1038200143, 1089578978, 1298446407, 21244503, 3605939213, 315512897, 2926222177, 3773581772, 2304894569, 148646902, 1310588622, 719015788, 33235550, 3322600921, 4227412422, 1029050494, 3711561852, 1896405474, 239956567, 3944721436, 2138341381, 3516180063, 2570116888, 2841833487, 3537545967, 1915861995, 2257935906, 455951217, 1980372327, 2503477265, 401759059, 2831315860, 363651332, 3904629611, 3002501344, 2696082435, 1127599679, 457936209, 1017518801, 3113840210, 432100974, 1703540805, 1596413876, 1516795978, 2175048057, 1383096682, 2019930907, 3400438823, 2703472410, 3518698847, 1119988276, 4260463826, 1041514591, 722025830, 2722811946, 2718009525, 2915426337, 1492724158, 702394443, 2648420744, 3967149668, 597821078, 982310946, 4134838367, 3629678914, 2900215964, 137278798, 378947263, 3630113109, 2008624617, 1995982212, 3486161782, 72610599, 3159346786, 630050016, 3451237729, 3027239596, 2623679913, 2698610811, 2804586946, 4099303878, 2769627328, 1769400210, 2531002632, 3169577980, 1716587073, 70601804, 1425000201, 2773798710, 935629971, 1209420303, 1288615575, 861818944, 4008176211, 2249474724, 2553488019, 2275622219, 1479688164, 189252465, 3741226414, 1279632219, 3411713437, 2959851122, 3095139149, 175236681, 724298988, 2492646972, 3182519703, 1768739307, 1235946613, 3780247272, 3811727068, 124952097, 2243890476, 3333411406, 3070504586, 390407378, 3367627269, 2646641546, 2019218901, 1628492242, 212972654, 1923944086, 4113994180, 3808166837, 1298390529, 204773172, 135691449, 2930587443, 3409379468, 4092117155, 2387189849, 1348073505, 573048283, 1911128995, 3657474599, 4160830834, 3698702292, 3031734291, 1159207158, 178194869, 3974345388, 121513807, 2243414640, 3187139136, 2729416529, 1317478407, 3013400999, 1558064227, 404092405, 1459431413, 3555612817, 778900712, 596732119, 1801088291, 318915003, 1697514265, 3229028811, 2213940540, 1104898251, 2083156744, 2777560292, 3853776406, 2235847842, 4120839290, 745571864, 2328871124, 2147720139, 2118781099, 1719559793, 566254480, 2194944305, 141604225, 136357448, 2580947916, 759652267, 3506979549, 1612525821, 832516308, 2327558413, 2508327765, 2546010558, 3246552216, 3172726164, 2442079132, 3844457669, 2275318158, 2761064877, 3036467837, 165545102, 3169438586, 3597172685, 3930172348, 3607821010, 3209023213, 1322355908, 845477939, 940456090, 1042071614, 1307248627, 3566773752, 4276736134, 2188997369, 2169483656, 3174688500, 3173762705, 2173633667, 844943722, 4237711960, 4197014126, 1219494793, 1477258801, 1312483962, 2987874066, 3795643100, 1540555400, 73028251, 3021245366, 3435819924, 14445348, 1783139014, 2153005334, 4020068984, 1479061276, 2475597063, 4033782160, 2132962199, 3443636679, 3954454510, 3073883932, 1537650, 3972185751, 1188980226, 141220814, 1819015142, 3381948100, 4064884005, 1079857057, 30215108, 1938655737, 3145017470, 323442323, 712955047, 3899147916, 3234815765, 3981265753, 71820846, 3721293779, 3405278910, 3396458304, 4059154975, 810662589, 2757284675, 2464162648, 3570343478, 4233382302, 3891335591, 805878635, 3819648495, 2077987797, 3666462814, 2871055212, 1159416354, 2842033292, 960930756, 2430501395, 4124797813, 609062223, 73468141, 3800718404, 3857279175, 730020633, 2400564131, 3860334924, 966784243, 2362880168, 31764188, 3100686999, 2487638933, 1541862240, 1845751827, 2877643497, 2846991239, 2553983562, 3095978042, 3481694187, 3242545244, 2666827344, 316934704, 79410887, 1179688283, 662257755, 2630097089, 3153273290, 2423807428, 2182477563, 3768890139, 3397026391, 1918918916, 1997862362, 1339307081, 3702205003, 824195148, 971695073, 3032896171, 52039572, 145382220, 623227621, 1423909699, 3031776501, 669138527, 2554098792, 1359784384, 3508421652, 1140993539, 428423867, 45039801, 2849410032, 3190274670, 4049885841, 51240848, 2114799047, 1174241046, 1940354316, 3994035483, 3017833587, 759206801, 2657735474, 672465902, 2464607461, 2016679971, 1315289451, 675406548, 3358454695, 1795155384, 2096407810, 4293441333, 1263939669, 2603222828, 1336833327, 1758190384, 454145376, 2645356780, 3232525811, 2355941822, 2941279547, 3307686541, 3242748439, 1491242998, 1949236614, 3739764567, 2736527929, 4290571869, 766270253, 3154513881, 1071478400, 4098507418, 2099194232, 952901593, 277001146, 3621090432, 3592819918, 3955420161, 2688692459, 1219034895, 4237107369, 3335446326, 2609147662, 1409367522, 1137764278, 393043035, 2937005328, 3343795268, 1448866094, 2996916407, 1590212246, 2164775617, 1535159326, 3199828331, 1531986334, 3642817773, 2749442585, 2988118491, 3391761874, 3252678315, 3131642495, 1560556169, 3010116065, 3217740745, 2766812242, 2492822988, 3148135043, 3042156579, 2543245380, 4197108702, 1762047346, 2162500513, 938364050, 1207720414, 4223716928, 1766342069, 3530665547, 18238640, 3047584697, 3434593809, 3820200293, 1762149770, 1749538084, 90892007, 487570034, 320443210, 2012199043, 2128568220, 1905705028, 3678521997, 675080795, 1173367613, 2642947331, 2916502717, 4057432244, 1994963341, 2549162647, 3417793573, 1224530779, 345749216, 4134340799, 534057100, 415135133, 593522259, 2556097570, 4117197572, 1200628520, 2697843347, 2228924060, 3758847166, 1932080634, 3674382705, 1385018981, 2504031200, 535483543, 3347820801, 719645241], 624, 0, 0.0]' pixi run tests custom tests/test_stump.py


================================================= FAILURES ==================================================
________________________________ test_stump_identical_subsequence_self_join _________________________________

    def test_stump_identical_subsequence_self_join():
        identical = rng.RNG.rand(8)
        T_A = rng.RNG.rand(20)
        T_A[1 : 1 + identical.shape[0]] = identical
        T_A[11 : 11 + identical.shape[0]] = identical
        m = 3
        zone = int(np.ceil(m / 4))
        ref_mp = naive.stump(T_A, m, exclusion_zone=zone, row_wise=True)
        comp_mp = stump(T_A, m, ignore_trivial=True)
        naive.replace_inf(ref_mp)
        naive.replace_inf(comp_mp)
>       npt.assert_almost_equal(
            ref_mp[:, 0], comp_mp[:, 0], decimal=config.STUMPY_TEST_PRECISION
        )  # ignore indices
E       AssertionError: 
E       Arrays are not almost equal to 5 decimals
E       
E       Mismatched elements: 2 / 18 (11.1%)
E       Mismatch at indices:
E        [6]: 0.0 (ACTUAL), 0.00010158705418291471 (DESIRED)
E        [16]: 0.0 (ACTUAL), 0.00010158705418291471 (DESIRED)
E       Max absolute difference among violations: 0.00010158705418291471
E       Max relative difference among violations: 1.0
E        ACTUAL: array([0.0594865428667342, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
E              0.14405632436990226, 0.0905990067980011, 0.07801877452945057,
E              0.0594865428667342, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
E              0.07801877452945057], dtype=object)
E        DESIRED: mparray([0.0594865428667375, 1.0323827311807139e-07, 0.0, 0.0, 0.0, 0.0,
E                0.00010158705418291471, 0.14405632436989182, 0.09059900679800098,
E                0.07801877452944078, 0.0594865428667375, 1.0323827311807139e-07,...

tests/test_stump.py:146: AssertionError