Skip to content

fix(analytics): exclude internal categories by group flag, not by name - #25

Open
DerekLangley wants to merge 1 commit into
pragprogrammer:mainfrom
DerekLangley:upstream-group-internal
Open

fix(analytics): exclude internal categories by group flag, not by name#25
DerekLangley wants to merge 1 commit into
pragprogrammer:mainfrom
DerekLangley:upstream-group-internal

Conversation

@DerekLangley

Copy link
Copy Markdown

I noticed that the analytics code currently identifies YNAB's internal category groups by their English display names. That works for the usual names, but it breaks if those groups are localized or renamed, and it doesn't catch Hidden Categories.

Since the YNAB API exposes an internal flag for category groups, this PR switches the exclusion logic to use that flag instead of relying on names.

What was happening

MONEY_FLOW_EXCLUDE_GROUPS in src/server/analytics.py matches the literal strings "Internal Master Category" and "Credit Card Payments". That is locale-dependent, silently stops working when a user renames a group, can be spoofed by a user-created group of the same name, and misses Hidden Categories entirely.

The API already provides the right signal: internal is required on both CategoryBase and CategoryGroup in spec 1.86.0. The models did not carry the field, and YNABBaseModel sets extra="ignore", so Pydantic was discarding it.

One important detail: the flag does not inherit

On a real budget, the Credit Card Payments group is internal: true while every category inside it is internal: false.

A category-level check therefore readmits all of them, and a payment category's activity is the movement for spending already counted in the real category. That causes credit-funded spending to be double-counted and pct_of_total to be computed against an inflated denominator.

The predicate therefore needs to use the category's group's internal flag.

group.internal is available from GET /plans/{id}/categories, but not from GET /plans/{id}/months/{month}. Analytics now resolve a category_id -> group.internal map from the delta-synced category list — one already-cached call, with no new endpoint required.

Behavior change

This deliberately changes output for some existing users in two cases:

  • If the relevant group names are localized or renamed, the old name-based check performed no exclusion. Internal categories will now be correctly removed.
  • Hidden Categories is internal: true, but was missed by the old name check. Users with hidden categories carrying activity may therefore see different get_money_flow and get_spending_by_category totals.

Both are the intended result of the fix rather than incidental behavior changes.

Compatibility and fallback behavior

A category can be absent from the current category map — for example, a historical month can reference a category that has since been deleted. In that case, the code treats the internal status as unknown and falls back to the old name check rather than assuming the category is not internal.

Two fallback situations are counted and logged separately:

  • A missing category is treated as routine historical churn and logged at INFO.
  • A resolved group with no internal flag is treated as a potential schema regression and logged at WARNING.

This keeps routine historical misses from obscuring an actual API/schema change.

internal is modeled as bool | None so an absent flag remains distinguishable from a real false value. An absent value must degrade to the name-based fallback; an explicit false must not.

Tests

Tests cover:

  • group-level vs. category-level internal
  • a renamed internal group
  • a user-created group that cannot spoof an internal group by name
  • Hidden Categories
  • both fallback reasons
  • routine missing-category cases emitting no warnings

The cache fixture in tests/test_server.py is changed to an AsyncMock so the added get_categories call is awaitable in the existing tests.

Test note: The full suite requires YNAB_API_KEY to be set due to the repository's existing import-time configuration behavior (#4); with a token set, 118 tests pass.

`MONEY_FLOW_EXCLUDE_GROUPS` matches the literal strings "Internal Master
Category" and "Credit Card Payments". That is locale-dependent, stops working
silently when a user renames a group, can be spoofed by a user-created group of
the same name, and misses the "Hidden Categories" group entirely.

The API already provides the signal: `internal` is required on both
`CategoryBase` and `CategoryGroup` in spec 1.86.0. The models did not carry the
field, so `extra="ignore"` was discarding it.

The flag does not inherit, and that is the trap. On a real budget the
"Credit Card Payments" group is `internal: true` while every category inside it
is `internal: false`. A category-level check therefore readmits all of them, and
a payment category's `activity` is the movement for spending already counted in
the real category — so credit-funded spending gets counted twice and
`pct_of_total` is computed against an inflated denominator. The predicate has to
be the category's *group's* flag.

`group.internal` comes from GET /plans/{id}/categories, not from
GET /plans/{id}/months/{month}, so analytics resolve a category -> group.internal
map from the delta-synced category list. That call is already cached; no new
endpoint is involved.

A category missing from that map — a historical month can name one that has since
been deleted — is treated as unknown rather than as "not internal", and falls
back to the previous group-name check. Those two situations are counted and
logged separately: a missing category is routine churn at INFO, while a resolved
group with no `internal` flag is a schema regression at WARNING.

`internal` is modeled as `bool | None` so an absent flag stays distinguishable
from a false one; `bool` with a False default cannot tell them apart, and absent
must degrade to the name check while false must not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant