Repository navigation
feat: keep the service's hours for 90 days and serve them to the stats Worker - #51
Merged
Merged
Conversation
ftwcloudd is the gateway (RFC 9458) for the box's daily energy report, which reaches it through an independent relay, so the service sees the report and never the box's address. - POST /v1/ohttp opens a request with the gateway key, takes only the one padded inner request with one valid report (energywire), adds its six sums to the date's row and answers a sealed 204 or 400. Nothing is kept per report: no row, logged body or address. One shared rate budget, and at most 100,000 reports a date. - GET /.well-known/ohttp-gateway serves the key configuration. - GET /v1/energy/public lists the closed dates of the last 30 days; a date with fewer than 5 reports shows null. - The key comes from Parameter Store into tmpfs at boot, as the Noise key does (deploy/ohttp.md, deploy/ohttp.sh); keygen -ohttp prints the pin Core needs. Without a key the gateway routes answer 501. Pairs with srcfl/ftw feat/anonymous-energy-ohttp; go/protocol is pinned at its commit 28fddd802242. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4f2c86290f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+162
to
+163
| k.rise(k.set.state(), k.now()) | ||
| k.put(k.hour) |
There was a problem hiding this comment.
If SIGTERM arrives just before an hour boundary and the drain finishes after it, Keeper.Run stops with the signal, but Close adds the remaining duration and counters to the previous hour without checking the current hour. For example, beginning at 05:59 and closing at 06:00:03 after incrementing drains produces only a 05:00 row containing all 63 seconds and the drain; the 06:00 contribution is missing. This misattributes shutdown activity and can make a full hour exceed 3,600 seconds. Handle the hour boundary during close, splitting elapsed time and recording the drain in the hour in which it finishes.
Useful? React with 👍 / 👎.
This was referenced Oct 24, 2026
Fredrik's change of scope, 24 October 2026: the fleet report moves onto the same relay and gateway as the energy report, and the two stay apart. - internal/energy becomes internal/gateway. POST /v1/ohttp reads the one padded inner request (go/protocol/reportwire), picks the decoder by schema, and adds a fleet report to the existing fleet_days totals that /v1/fleet/public reads and an energy report to energy_days. Nothing is kept per report. - POST /v1/fleet stays for boxes on a Core release without a relay; it can go one stable Core release after the first that pins the relay. - The request cap fits the 1,591-byte sealed request; README and deploy/ohttp.md describe both reports and the two-per-day traffic. - go/protocol pinned at srcfl/ftw 4cfd65f7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frahlg
force-pushed
the
feat/internal-ops-stats
branch
from
October 24, 2026 06:43
4f2c862 to
999854e
Compare
A date with five reports and one car published that car's whole charge in vehicle_charge_kwh, because the sum counted the other four reports' zeros. energy_days now counts, for each sum, the reports that had any of it, and /v1/energy/public shows a sum only when none or at least five had some. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frahlg
force-pushed
the
feat/internal-ops-stats
branch
from
October 24, 2026 07:04
999854e to
d0fe105
Compare
…t view From review of #50: - Replays: each inner request now carries a random report id and its send day (srcfl/ftw a7c025e8). The gateway refuses a request sent before yesterday and keeps a hash of each id per day for three days, in a WITHOUT ROWID table with no time, so a replay or a retry adds nothing and is still answered 204. Seven posts of one ciphertext now count once. - /v1/fleet/public lists closed UTC days only, so the relay cannot poll it and match a new count to the report it just forwarded. - Each report kind has its own budget, spent only by a report that opened and decoded; junk no longer silences real reports. - A 10-second deadline on reading the body of /v1/ohttp and /v1/fleet; a failed read closes the connection. - ohttp keys: data after the last whole PEM block, such as a second key cut short in a rotation, is refused instead of silently ignored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frahlg
force-pushed
the
feat/internal-ops-stats
branch
from
October 24, 2026 07:16
d0fe105 to
e755d0f
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frahlg
force-pushed
the
feat/internal-ops-stats
branch
from
October 24, 2026 07:50
e755d0f to
f675fdc
Compare
The gateway now looks a report id up, read-only, before it spends the kind's budget. A repeat gets a sealed 204 and costs nothing, so one captured request sent many times cannot crowd out real reports. The OHTTP key file must hold whole PKCS #8 blocks and nothing else. A block cut short at either end, or text between blocks, stops the start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s Worker ftwcloudd now keeps an hourly summary of every series it counts in the registry (ops_hours), for 90 days: each counter's rise, each gauge's low, mean and high, each latency's buckets, how much of the hour a process ran and how many started. A deploy hands over its part of the hour after the drain; the next process adds the rest. GET /v1/ops serves a week of settled hours at a time, with the exact totals of the anonymous fleet and energy reports for closed days, only to the key the project stats Worker holds. The service holds the key's SHA-256 (-ops-key-sha256); without it the route answers 404 to everyone. New series: sockets by transport (QUIC or WebSocket), bytes written to boxes and phones, MCP requests and tool calls by outcome and their time, web push answers, and the drain's sockets. Every label is a fixed word (TestCountsNameNobody), and traffic makes no new series and leaves no home, device, key or address in an hour (TestAnHourOfCountsNamesNobody). Stacked on #50, whose energy day totals it reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A malformed FTW_CLOUD_OPS_KEY_SHA256, such as a stray quote or line end in compose.env, made ops.New fail, and with it the whole service. The hash is now read by UseKey after New: quotes and white space around it are dropped, and a hash that is still not 64 hex digits is logged and leaves /v1/ops answering 404, while boxes and phones are served as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
frahlg
force-pushed
the
feat/internal-ops-stats
branch
from
October 24, 2026 08:22
f675fdc to
fa879b0
Compare
# Conflicts: # CLAUDE.md # README.md # cmd/ftwcloudd/main.go # go.mod # go.sum # internal/server/server.go # internal/server/server_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Fredrik, 24 October 2026: Sourceful needs internal statistics to run FTW Cloud and understand the fleet, for internal use, in one place, without a new system. Today
/metricsand the minute's log line count well, but nothing keeps them: the log dies at every deploy.What changes
internal/metricsrolls every series up into clock hours (Keeper): each counter's rise, each gauge's low, mean and high over the minutes sampled, each histogram's buckets, how much of the hour a process ran, how many processes started. The registry keeps one row an hour (ops_hours) for 90 days, pruned at each write and at upkeep. A deploy hands over its part of the hour after the drain; the next process adds the rest to the same row.GET /v1/ops(internal/ops) serves up to a week of settled hours (?since=), as counts and p50/p95/p99 in ms, plus the exact totals of the anonymous fleet and energy reports for closed days. It answers onlyAuthorization: Bearer <key>. The service holds the key's SHA-256 (-ops-key-sha256,FTW_CLOUD_OPS_KEY_SHA256in compose), which is no secret; empty answers 404 to everyone and keeps the hours anyway. The key lives only as the stats Worker's secret.transport_sockets_{box,app}_{quic,websocket}), bytes written to boxes and phones (bytes_out_{box,app}, against AWS's free 100 GB of data out), MCP requests and tool calls by outcome (mcp_*) and their time, browsers' push answers (webpush_*), and the drain (drains,drained_sockets,drain_left_open).deploy/README.md"Watching it" (how to make the key and its hash), CLAUDE.md invariants, andprivacy/internal-operations.md: what is kept, how long, who sees it, why none of it is about a household, and what remains.Limits
secondsbelow 3600 andstarts1.-ops-key-sha256is read byUseKeyafterNew. Quotes and white space around it are dropped. A hash that is still not 64 hex digits is logged at start,/v1/opsanswers 404, and boxes and phones are served as before (review fix, second commit).Why none of it names a home
TestCountsNameNobodyholds the list of label names (kind,reason,path,dir,result,transport,outcome); each value is a fixed word.TestAnHourOfCountsNamesNobodyruns a box over QUIC and a phone through the tunnel, then checks that the stored hour holds the traffic, that its series are the ones that existed before the traffic, and that it holds no home id, device id, key or address./v1/energy/public). Two answers never differ by one report.TestTheFeedHoldsOnlyItsFieldspins the field set; the Worker refuses anything else.The limit, plainly: while the fleet is a handful of homes, an hour's phone count or a closed day with one report can still describe one of a few people staff know. Only Access sees it.
privacy/internal-operations.mdnames this and the options if Fredrik wants it smaller.Size
Lines of Go without tests, against #50:
internal/metrics+224 (hours.go),internal/ops+278 (new),internal/registry+68,internal/mcp+26,cmd/ftwcloudd+22,internal/webpush+15,internal/server+14,internal/appsession+10,internal/wire+10,internal/uplink+8. About +675 in all.Pairs and order
/v1/opsreads itsenergy_daysstore rather than keeping a second one. Merge feat: one Oblivious HTTP gateway for both anonymous daily reports #50 first, then rebase this ontomain(not with--delete-branchon feat: one Oblivious HTTP gateway for both anonymous daily reports #50).Nothing is deployed. Turning it on needs the key in the Worker and its hash in
/etc/ftw-cloud/compose.env.Checks
go vet ./... && go test -race ./...,go test -tags ftwclouddev ./...,bash scripts/check-imports.sh: all pass locally. (TestARefusalOverQUICCarriesItsCodehit its 3 s deadline once on a loaded laptop and passed three runs in a row after.)🤖 Generated with Claude Code