Validator Fire Drill

A submission portal for the gno.land validator fire drill: validators authenticate by proving ownership of their valopers operator address, then upload a diagnostic archive for automated validation and storage.

See prd.md for the full product spec, security rationale, and implementation status of every piece below.

Quick start

export PATH="/usr/local/go/bin:$PATH"   # if the Go toolchain isn't already on PATH

ADMIN_OPERATOR_ADDRESSES=<comma-separated bech32 admin addresses> go run ./cmd/portal \
  -remote https://rpc.topaz.testnets.gno.land \
  -addr localhost:8080 \
  -upload-dir ./portal-uploads

Then open http://localhost:8080/ for the validator submission flow, and http://localhost:8080/admin for the live submissions dashboard — admins sign in the same challenge-tx way validators do (see Admin endpoints), restricted to the addresses listed in ADMIN_OPERATOR_ADDRESSES.

Docker (fastest way to start everything)

No Go toolchain or real AWS credentials needed — docker compose starts the portal, a local S3-compatible backend (MinIO), and ClamAV for malware scanning:

cp .env.example .env   # fill in REMOTE and ADMIN_OPERATOR_ADDRESSES at minimum
docker compose up --build

The first up takes several extra minutes, and looks like a hang: the clamav service downloads its ~1 GB signature database before it reports healthy, and the portal waits for that. Subsequent starts reuse the cached clamav-data volume and are fast.

The URLs are the same as above, but on PORTAL_PORT rather than a fixed 8080 — with the .env.example default that is http://localhost:8080/ and http://localhost:8080/admin; set PORTAL_PORT=8888 and it is http://localhost:8888/ instead.

Everything — the RPC endpoint, the admin operator address whitelist, storage credentials, published port, and the upload size limits (MAX_UPLOAD_SIZE / MAX_LOG_SIZE, see Upload size and ClamAV before changing either) — is configured through .env. See .env.example for the full list of variables and defaults. Uploaded archives and the submission log persist across docker compose down / up in named volumes.

Production deployment

docker-compose.prod.yml runs the same stack from a published image instead of building from a checkout, so the deployment host needs neither the Go toolchain nor the source. Three files are enough: the compose file, clamd.conf, and your .env.

docker compose -f docker-compose.prod.yml up -d

The portal image is ghcr.io/samouraiworld/validator-diagnostics, published by .github/workflows/ci.yml on every merge to master — but only after go vet and go test pass, so a red build never reaches the registry. Alongside latest, every commit is tagged sha-<7 chars>. That tag is the rollback path: set PORTAL_IMAGE_TAG in .env to a known-good one and re-run the command above.

Two production-specific settings that development gets away with ignoring:

• Set SESSION_SECRET and ADMIN_SESSION_SECRET explicitly. Left unset, the portal generates a random secret per run, so any restart — including an automatic one from restart: unless-stopped — signs every validator and admin out mid-exercise.
• Pin the upstream clamav and minio image tags once you have tested against a given version. stable and latest both move underneath a running deployment.

For TLS, the compose file carries a commented Traefik v3 service with Let's Encrypt and an HTTP→HTTPS redirect; enabling it is a matter of uncommenting three blocks, setting PORTAL_DOMAIN and ACME_EMAIL, and removing the portal's ports: mapping so the plaintext port disappears. The instructions are in the file. If you put a different reverse proxy in front instead, note the two constraints Traefik is configured for there: no request buffering, and no read/write timeout on the route. A submission is a single long request — up to 4 GiB uploaded, then validated, then scanned by clamd for up to 15 minutes before the response is written. A proxy that buffers writes the whole archive to its own disk first; one with a finite timeout reports failure to the validator for a submission that actually succeeded.

Flags and environment variables

Flag	Required	Description
-remote	yes	gno.land RPC endpoint used to verify operator public keys (e.g. https://rpc.topaz.testnets.gno.land)
-addr	no	Address to listen on (default localhost:8080)
-session-ttl	no	How long an issued session token stays valid (default 5m)
-admin-session-ttl	no	How long an issued admin session token stays valid (default 1h)
-upload-dir	one of -upload-dir / -s3-bucket	Local directory to save archives into
-s3-bucket	one of -upload-dir / -s3-bucket	S3-compatible bucket to save archives into (AWS S3, Scaleway, Cloudflare R2, ...)
-s3-region	with -s3-bucket	S3-compatible region
-s3-endpoint	with -s3-bucket, optional	Custom S3-compatible endpoint (leave empty for real AWS S3)
-log-path	no	Path to the submission log file the admin dashboard reads (default ./submissions.jsonl)
-exercise-path	no	Path to the exercise config file written by the admin dashboard (default ./exercise.json)
-scores-path	no	Path to the scoring records file (default ./scores.json)
-clamav-addr	no (recommended)	clamd address to scan uploads against — host:port, or unix:/path/to/socket. Unset disables scanning: fine for local dev, not for production
-clamav-timeout	no	Time budget for one clamd scan, dial included (default 15m). Bounds a single scan window now (at most 1 GiB, roughly 7s at the measured rate), not the whole upload
-max-upload-size	no	Maximum accepted upload, in bytes (default 4294967296 — 4 GiB. Not a scan limit: clamd never sees more than one 1 GiB window at a time). Raising it is a disk/S3/time question — see Upload size and ClamAV
-max-log-size	no	Maximum accepted size of each log entry inside the archive (validator.log.gz, sentry.log.gz), in bytes (default 4294967296 — 4 GiB). Each entry is streamed, not buffered, so this bounds decompression work rather than memory
-av-scan-budget	no	Maximum decompressed bytes of log content the antivirus examines per submission, shared across validator.log.gz and sentry.log.gz under one budget (default 34359738368 — 32 GiB). Exceeding it records partial scan coverage rather than rejecting the submission — see Upload size and ClamAV
-max-concurrent-submissions	no	How many submissions are processed at once (default 4). Further uploads wait for a slot, then get a 503 with Retry-After. Keep it at or below clamd.conf's MaxThreads — see Concurrency and throughput. 0 disables the limit
-submission-queue-wait	no	How long an upload waits for a processing slot before being rejected (default 15m)

Environment variable	Required	Description
ADMIN_OPERATOR_ADDRESSES	yes	Comma-separated bech32 operator addresses allowed to authenticate against the admin dashboard. Admins sign in the same challenge-tx way validators do (/auth/challenge, /auth/admin/verify) — an address not in this list gets a 403 even with a valid signature. Replaces the old ADMIN_PASSWORD Basic Auth; existing deployments must set this before upgrading or the portal will refuse to start
SESSION_SECRET	no	Hex-encoded HMAC secret for session tokens. If unset, a random one is generated for the run — fine for a single exercise, not for a long-lived deployment (sessions won't survive a restart)
ADMIN_SESSION_SECRET	no	Hex-encoded HMAC secret for admin session tokens, kept separate from SESSION_SECRET so restarting the portal or rotating one secret doesn't affect the other session type. If unset, a random one is generated for the run
S3_ACCESS_KEY / S3_SECRET_KEY	with -s3-bucket	Credentials for the S3-compatible backend
MAX_UPLOAD_SIZE	no	Read by docker-compose.yml and passed through as -max-upload-size (default 4294967296 — 4 GiB: clamd never sees more than one 1 GiB scan window at a time, so raising this is a disk/S3/time question instead). Unlike the other variables here, the binary does not read it directly — see Upload size and ClamAV
MAX_LOG_SIZE	no	Read by docker-compose.yml and passed through as -max-log-size (default 4294967296 — 4 GiB). Not read directly by the binary either. Bounds each compressed log entry (validator.log.gz, sentry.log.gz); the antivirus no longer caps the decompressed size — that's AV_SCAN_BUDGET (coverage) and scoring.maxLogWindowBytes (partial credit for automatic scoring only) — see Upload size and ClamAV
AV_SCAN_BUDGET	no	Read by docker-compose.yml and passed through as -av-scan-budget (default 34359738368, i.e. 32 GiB of decompressed log bytes, shared across both validator.log.gz and sentry.log.gz under one budget per submission). Exceeding it records partial scan coverage rather than rejecting the submission — see Upload size and ClamAV

Upload size and ClamAV

clamd never sees the raw archive. Before anything is stored, the upload is taken apart and its extracted content is streamed to clamd: metadata.json whole, then each decompressed log entry — validator.log.gz, and sentry.log.gz when submitted — in 1 GiB windows with a 1 MiB overlap between consecutive windows, so a signature straddling a window boundary is still caught (clamav.WindowedScanner). Both logs are scanned under one shared per-submission budget, not one budget each. The scan still fails closed: an infected verdict or the scanner itself failing rejects the submission and stores nothing.

clamd.conf (bind-mounted by docker-compose.yml) raises clamd's stream and file limits to 2147483647 bytes — 2 GiB minus one. That is headroom for one window plus its overlap, not a bound tied to -max-upload-size / MAX_UPLOAD_SIZE — the two no longer have to agree. clamd.conf itself does not change with -max-upload-size: it was already sized for one scan window, and a larger upload still only ever produces 1 GiB windows.

-max-upload-size / MAX_UPLOAD_SIZE defaults to 4294967296 bytes — 4 GiB, raised from a previous 2147483647 (2 GiB - 1) so that a real archive this feature exists for — a validator.log.gz entry too big for the old ceiling — is actually accepted; see defaultMaxUploadSize in cmd/portal/main.go for the archive size that motivated it. Raising it further no longer costs clamd anything — it costs disk: uploads over multipartMemoryThreshold (32 MiB) spill from memory to a temp file (net/http's own multipart handling, not anything this project wrote), so free disk has to cover roughly one archive's worth of space per concurrent submission in flight. The next hard ceiling above that is storage.S3Store.Save's single PutObject, which S3 caps at 5 GiB.

The 2 GiB wall

That odd-looking number is a hard ceiling, not a tuning choice. libclamav cannot scan any single file of 2147483648 bytes or more. Configure a larger MaxFileSize and clamd accepts the value, logs

LibClamAV Warning: Max file-size was set to N bytes. Unfortunately, scanning
files greater than 2147483647 bytes (2 GiB - 1) is not supported.

and then rejects oversized input at scan time with Heuristics.Limits.Exceeded.MaxFileSize, which the portal reports as a 503. Raising the limits does not buy headroom; it only moves the failure later. (Verified against ClamAV 1.5.3.)

The ceiling applies to every file clamd extracts, not just what is handed to it directly — which is exactly why a scan window is 1 GiB rather than something closer to the wall: there has to be room left for the 1 MiB overlap on top without the total ever approaching 2147483647. Each of a validator's submitted logs — validator.log.gz and, when present, sentry.log.gz — however large it decompresses to, is scanned in bounded pieces instead of as one stream, so neither ever reaches this ceiling; a genuinely huge log just means more windows, not a rejection.

What now bounds a submission's antivirus coverage is -av-scan-budget / AV_SCAN_BUDGET (default 32 GiB of decompressed log, shared across both logs): once that much has been examined across the two of them, scanning stops — not because clamd rejects anything, but as the tar/zip-bomb defence prd.md asks for. See What a partial scan means below for what happens to a submission when it does.

It also sets AlertExceedsMax yes, which is what stops the AV layer from failing open. By default clamd silently skips content that exceeds MaxScanSize/MaxFileSize and still answers stream: OK — the portal would store an unscanned window as clean. With the setting on, clamd reports a Heuristics.Limits.Exceeded pseudo-signature instead, which the portal treats as a failed scan (503, logged for the operator), not as malware. If you see those 503s, raise the limits in clamd.conf; don't turn the setting off.

What a partial scan means

A submission's antivirus coverage can end early two ways and still be accepted: the shared budget (-av-scan-budget) runs out, or one log's stream itself breaks partway through — a truncated validator.log.gz or sentry.log.gz. Either way the submission is accepted, stored, and badged — never silently treated as fully scanned, and never rejected for this reason alone.

clamd disconnecting mid-window is different, not a third partial-coverage case: that is the scanner failing, not the log source, and clamav.WindowedScanner.ScanStream can only tell the two apart by whether reading the log itself produced the error. A scanner failure always comes back as a plain error, which the handler turns into a 503 and rejects the submission — the same outcome as clamd being unreachable from the start. Only an actual malware verdict, or the scanner itself failing (including disconnecting mid-window), rejects a submission. What was and wasn't examined travels with the record (Entry.Scan, a Coverage{Complete, Bytes}), and the admin dashboard shows it: scan ✓ for complete coverage, scan partiel with a byte count for partial.

A submitted log entry that cannot be decompressed at all is different — whether it's validator.log.gz or the optional sentry.log.gz: nothing in it was ever readable, so nothing in it could be scanned, and the upload is rejected outright (400) rather than stored unscanned.

With -clamav-addr unset, none of this runs: no scanner means no Entry.Scan is ever recorded, and the dashboard shows no scan badge at all on that row — not a reassuring one, none. Scanning is opt-in per deployment, but a deployment that opts out gets silence, not a false "clean".

Concurrency and throughput

A submission is expensive and, more importantly, expensive for a long time. The upload spills to a temp file that is held for the whole request; the archive is then walked three separate times (ValidateArchive, the antivirus pass, the scoring pass), each decompressing the outer gzip again because a gzip stream cannot be seeked; and clamd spools every INSTREAM window to its own disk before scanning it. A 2.4 GB archive whose log decompresses to tens of GB therefore moves on the order of 60 GB through the disk, and measured about an hour end to end on an 8-core host with its Docker volumes on RAID5 — with the network transfer not the slow part.

Two settings bound that, and they have to be read together:

• -max-concurrent-submissions (default 4) caps how many run at once. Beyond it, uploads wait up to -submission-queue-wait and are then rejected with a 503 and a Retry-After, rather than queueing invisibly — a validator stalled behind a full queue sees even the browser's own upload bar freeze, with nothing to explain it.
• clamd.conf's MaxThreads (6 here) caps concurrent scans.

Keep -max-concurrent-submissions at or below MaxThreads. Above it, scan windows queue inside clamd, where the wait is invisible to the portal and still counts against -clamav-timeout — which covers the dial. A window that times out is a failed scan, and a failed scan rejects the submission with a 503. Mis-size the pair and congestion stops producing slow submissions and starts producing lost ones, after an hour of work each.

Parallelism does not buy much here anyway: the work is disk- and CPU-bound on a single box, so running more of it at once mostly stretches each one. The limit protects throughput rather than capping it.

Scope the exercise instead. By far the largest lever is not a setting: it is how much log the exercise asks for. Real gnoland validator log runs about 20 MB compressed per 24 hours (305 MB decompressed), so an investigation window of a few hours yields archives in the single-digit MB — three orders of magnitude below the case above, fast enough that none of these limits are ever reached. If you do scope the exercise that way, consider lowering -max-upload-size to match: a validator who submits their entire log by mistake is then rejected in seconds by http.MaxBytesReader, instead of consuming a processing slot for an hour before anyone finds out.

Each submission logs a submission timings for <file>: validating=… scanning=… storing=… scoring=… line. That is the measurement to size all of the above against — including a submission that was rejected, whose line reports how long the failing phase ran before it gave up. To attribute a slow scanning phase between clamd and the disk, uncomment LogClean yes in clamd.conf for the duration of a test: with LogTime yes it gives the wall-clock cost of every window from the daemon itself.

Admin endpoints

All of these (except GET /admin itself, which serves the sign-in screen and contains no data) sit behind operator-address-whitelist auth: admins sign in via POST /auth/challenge + POST /auth/admin/verify, the same challenge-tx flow validators use, restricted to the addresses in ADMIN_OPERATOR_ADDRESSES. The POST routes accept Content-Type: application/json only.

Route	Purpose
GET /admin	The dashboard itself
GET /admin/submissions	Recorded submissions joined with their scores, as JSON
GET/POST /admin/exercise	Read or replace the exercise config (announce/deadline times, investigation window, expected genesis hash, supported versions, observations)
POST /admin/submissions/{id}/score	Enter the one manually judged criterion: incident response quality
GET /admin/summary	Generate the Markdown participation/score summary to publish on Discord

Running an exercise

1. Configure the exercise before announcing it. Open /admin, fill in the exercise form (announce time, deadline, investigation window, expected genesis_sha256, supported gnoland versions) and save. Submissions that arrive while no config exists are stored and logged but recorded as "not yet scored" — automatic scoring has nothing to score against, and it is not retroactive.
2. Announce the drill and collect submissions.
3. Enter the manual incident response quality score per submission from the dashboard. A total is shown as pending until it's in — the score is required, and leaving the box empty is rejected rather than recorded as a 0 (which is itself a valid score).
4. Generate the summary and publish it. A submission whose log scan stopped early is reported as "could not be fully verified" rather than as failing to cover the investigation window: the two are different claims, and only the second is about the validator.

There is no rescore. Automatic scores are computed once, at submit time, from the config in force at that moment — the log bytes they were derived from are not retained. So:

• A submission that arrived before the exercise was configured is permanently "not yet scored", and the portal will refuse manual scores for it (409) rather than record a total made only of the manual half.
• Editing announced_at, deadline_at, the investigation window, the expected genesis hash or the supported versions after submissions have landed leaves every existing score computed against the old values, with no warning and no way to recompute.

Both are recoverable by hand — the archives are still in object storage — but nothing in the portal does it for you. Get step 1 right before step 2.

How it works

1. Authentication (auth/) — a validator proves ownership of their valopers operator address by signing a server-issued, never-broadcast "challenge" transaction with gnokey sign (no wallet integration, no private key ever touches the server). A verified signature mints a short-lived, stateless session token.
2. Validation (submission/) — the uploaded archive is checked against the required naming convention, structure, and security rules (no path traversal, no symlinks, bounded decompression, schema-checked metadata) before anything is trusted.
3. Storage (storage/) — the original archive bytes are saved unchanged, either to local disk (LocalStore, for testing) or S3-compatible object storage (S3Store).
4. Orchestration (portal/) — SubmitHandler wires the three together into POST /submit, cross-checking that the archive's claimed identity actually matches the authenticated session. Successful submissions are recorded to an append-only log that the admin dashboard (portal.AdminAuth, portal.AdminSubmissionsHandler) reads.
5. Scanning and scoring (clamav/, exercise/, scoring/) — the archive's extracted content is streamed to clamd (fail-closed: an infected verdict or a failed scan rejects the submission and stores nothing; a scan that stops early because of a stream break or its budget is accepted, stored, and recorded as partial — see Upload size and ClamAV), then scored against the configured exercise: genesis hash, gnoland version, investigation-window coverage of the submitted logs (a covering sentry.log.gz earns extra log-quality credit on top of validator.log.gz's), and upload timeliness. The one criterion no code can observe — incident response quality — is entered by an admin afterwards.
6. Frontend (cmd/portal/static/) — a small, framework-free HTML/JS UI for both the validator flow and the admin dashboard, embedded into the cmd/portal binary at build time (embed.FS) — one binary, no separate deploy step.

Development

export PATH="/usr/local/go/bin:$PATH"
go build ./...
go vet ./...
go test ./...

Design and implementation history for the frontend/admin work live in docs/superpowers/specs/ and docs/superpowers/plans/.