Watchdog

Watchdog records AI requests, tool calls and agent steps in a local SQLite database. Its dashboard and JSON APIs show latency, token use, estimated costs and failures.

Point an OpenAI-compatible client at /proxy/v1. Watchdog forwards requests to your provider and records telemetry. Native producers can also submit canonical batches or supported OTLP traces.

Costs are estimated direct-provider equivalents, not invoices. See pricing estimates for catalog sources, model mappings and repricing commands.

Watchdog runs locally by default. Before exposing it to other machines, configure token authentication, TLS, network access, retention, backups and secret storage. See the deployment guide.

Current release

Watchdog 1.2.1 fixes credential forwarding, telemetry validation, OTLP retries and delivery queue limits. It reduces canonical intake work, client cloning and backup memory use, and updates pricing catalogs. Existing routes and telemetry schemas remain compatible; malformed records and conflicting OTLP retries now return errors. See the changelog for details.

How it works

Your client sends requests to this server at /proxy/v1/....
The server forwards them to a configurable upstream base URL.
The server logs request metrics and traces in SQLite.
The dashboard UI reads those logs from JSON API endpoints.

No changes are required in your app code beyond setting the base URL (and API key behavior) to use the proxy.

Quick start

Use Kujo 1.6.0 for the verified release workflow and runtime-measurement adapter. Watchdog does not authorize execution or replay.

1) Start the Watchdog server

cd /path/to/kujo-watchdog
kujo run --interpreter dashboard_server.kujo

2) Open the dashboard

http://localhost:7700

When WDG_API_AUTH_MODE=token, the dashboard opens an API access prompt. Enter the configured WDG_API_AUTH_TOKEN; the browser keeps it in sessionStorage for the current tab session and sends it as X-Watchdog-Token on dashboard API requests. A missing or invalid token is shown explicitly instead of leaving the dashboard in an empty loading state.

3) Verify proxy configuration endpoint

curl -s http://localhost:7700/api/proxy-config

Expected response shape:

{
  "ok": true,
  "data": {
    "upstream_base_url": "https://api.openai.com/v1",
    "auth_mode": "passthrough",
    "has_override_api_key": false,
    "api_auth_mode": "off",
    "api_auth_enabled": false,
    "proxy_auth_mode": "off",
    "proxy_auth_enabled": false
  }
}

4) Optional smoke test (passthrough auth mode)

curl -i -s \
    -X POST http://localhost:7700/proxy/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"gpt-4.1-mini","messages":[{"role":"user","content":"proxy smoke"}]}'

If no API key header is passed in passthrough mode, you should see an upstream 401 and a corresponding error row in /api/requests.

Canonical examples and generated paths

Use src/ as the canonical implementation source. The root files dashboard_server.kujo, dashboard.html, watchdog.kujo, and watchdog_shared.kujo are compatibility mirrors kept in sync from src/.

demo.kujo, this README, and docs/KENNEL_INTEGRATION_GUIDE.md are the primary copyable examples. Prefer compact, runnable snippets in those files and keep repeated output formatting behind small local helpers when it improves scannability.

The demo seeder writes to data/watchdog-demo.db and refuses the production watchdog.db. Demo rows are labeled source_app=watchdog-demo and data_class=fixture; live proxy and external application telemetry remain in the production database with distinct classifications.

Treat tests/ as contract and regression coverage. Do not shorten fixtures or expected-output checks just to reduce tokens when explicit examples make a test clearer.

Exclude generated and bulk runtime paths from broad readability sweeps unless a task explicitly targets them: tmp/, data/, vendor/, SQLite artifacts, and local proxy config files.

Configuration

Environment variables:

Variable	Default	Purpose
`WDG_HOST`	`0.0.0.0`	Bind address; use `127.0.0.1` for a local-only instance
`WDG_PORT`	`7700`	HTTP port for Watchdog + proxy
`WDG_DB_PATH`	`data/watchdog.db`	SQLite path for logs
`WDG_PROXY_CONFIG_PATH`	`watchdog_proxy_config.json`	JSON config file path
`WDG_SOURCES_CONFIG_PATH`	`watchdog_sources.json`	Private, versioned Connected Sources metadata registry path
`WDG_UPSTREAM_BASE_URL`	`https://api.openai.com/v1`	Upstream OpenAI-compatible base URL
`WDG_PROXY_AUTH_MODE`	`passthrough`	`passthrough` or `override`
`WDG_UPSTREAM_API_KEY`	empty	API key used when auth mode is `override`
`WDG_UPSTREAM_API_KEY_ENV`	empty	Name of env var to read API key from when `override`
`WDG_API_AUTH_MODE`	`off`	`off` or `token` for `/api/*` endpoints
`WDG_API_AUTH_TOKEN`	empty	Shared token required when `WDG_API_AUTH_MODE=token`
`WDG_PROXY_AUTHZ_MODE`	`off`	`off` or `token` for `/proxy/*` endpoints
`WDG_PROXY_AUTHZ_TOKEN`	empty	Shared token required when `WDG_PROXY_AUTHZ_MODE=token`
`WDG_DEPLOYMENT_PROFILE`	`local`	`local` or `production` startup policy profile
`WDG_ALLOW_INSECURE_STARTUP`	`false`	`true` allows break-glass startup when production policy checks fail
`WDG_PROXY_AUTHZ_ALLOWLIST`	`/healthz,/readyz`	Comma-separated exact-path bypass list for proxy auth checks
`WDG_DITHER_CHARTS_JS_PATH`	`vendor/dither-charts.js`	Built Dither Kit dashboard chart bundle
`WDG_DITHER_CHARTS_CSS_PATH`	`vendor/dither-charts.css`	Built Dither Kit dashboard chart styles
`WDG_DEPARTURE_MONO_PATH`	`vendor/fonts/DepartureMono-Regular.woff2`	Local Departure Mono font used for dashboard titles and numeric displays
`WDG_PROXY_TIMEOUT_SECS`	`120`	Upstream proxy timeout in seconds
`WDG_MAX_PROXY_BODY_BYTES`	`1048576`	Reject proxy request bodies larger than this many bytes
`WDG_MAX_PARSE_BODY_BYTES`	`524288`	Reject JSON request parsing over this many bytes
`WDG_REDACTION_MODE`	`basic`	`basic` or `off` telemetry redaction before persistence/export
`WDG_CONTENT_CAPTURE_MODE`	`off`	`off` stores no proxy prompt/response summaries; `summaries` explicitly stores bounded summaries
`WDG_REDACT_TERMS`	`api_key,authorization,bearer,password,secret,token,sk-`	Comma-separated redaction match terms
`WDG_RATE_LIMIT_MODE`	`off`	`off` or `basic` SQLite-backed throttling for `/api/` and `/proxy/`
`WDG_RATE_LIMIT_MAX_REQUESTS`	`60`	Max requests allowed per bucket per window
`WDG_RATE_LIMIT_WINDOW_SECS`	`60`	Rate-limit window duration in seconds
`WDG_RATE_LIMIT_MAX_BUCKETS`	`5000`	Maximum retained rate-limit buckets after eviction
`WDG_RATE_LIMIT_BUCKET_TTL_SECS`	`600`	TTL for idle rate-limit buckets before cleanup
`WDG_IDENTIFIER_MAX_LEN`	`128`	Max stored length for session/user/tenant/project/workflow/task/correlation identifiers
`WDG_EXPORT_MAX_ROWS`	`10000`	Default per-kind row cap for `/api/export` requests (override via query)
`WDG_EXPORTERS_CONFIG_PATH`	`watchdog_exporters.json`	Versioned exporter-profile configuration; credentials are referenced by environment-variable name only
`WDG_EXPORT_QUEUE_MAX_RECORDS`	`50000`	Pending/retry record bound per exporter profile
`WDG_EXPORT_QUEUE_MAX_BYTES`	`67108864`	Pending/retry canonical-byte bound per exporter profile
`WDG_CONTEXT_LIMIT_CATALOG_PATH`	`config/context_limit_catalog.json`	Separate, versioned model context-limit catalog; never inferred from pricing
`WDG_CONTEXT_WARNING_RATIO`	`0.80`	Global canonical context warning threshold
`WDG_CONTEXT_CRITICAL_RATIO`	`0.95`	Global canonical context critical threshold (clamped to warning or higher)
`WDG_BACKUP_ENABLED`	`true`	Seed automatic backup scheduling on first startup
`WDG_BACKUP_INTERVAL_MINUTES`	`1440`	Initial backup frequency; later changes persist from the dashboard
`WDG_BACKUP_DIR`	`data/backups`	Initial separate backup folder; absolute Dropbox/Google Drive paths are supported
`WDG_BACKUP_RETENTION_COUNT`	`30`	Initial number of generated backups to retain
`WDG_BACKUP_ENCRYPTION_ENABLED`	`false`	Seed AES-256 encrypted backups on first startup
`WDG_BACKUP_ENCRYPTION_KEY_FILE`	empty	Mode-600 passphrase file used by OpenSSL; never returned by the API
`WDG_BACKUP_SCRIPT_PATH`	`scripts/watchdog_backup.js`	Backup helper path
`WDG_NODE_BIN`	`node`	Node.js executable used by the backup helper
`WDG_OPENSSL_BIN`	`openssl`	OpenSSL executable used for encrypted backups

Optional watchdog_proxy_config.json (path controlled by WDG_PROXY_CONFIG_PATH):

{
    "upstream_base_url": "https://api.openai.com/v1",
    "auth_mode": "passthrough",
    "upstream_api_key": "",
    "upstream_api_key_env": ""
}

Precedence: env vars override file values, and file values override defaults.

Named upstream profiles

One Watchdog server can proxy several provider accounts into the same telemetry database and dashboard. Keep the existing top-level values as the default profile, then add named entries for additional accounts:

{
  "upstream_base_url": "https://ollama.com/v1",
  "auth_mode": "override",
  "upstream_api_key_env": "OLLAMA_PERSONAL_API_KEY",
  "upstream_profiles": {
    "openrouter-work": {
      "upstream_base_url": "https://openrouter.ai/api/v1",
      "auth_mode": "override",
      "upstream_api_key_env": "OPENROUTER_WORK_API_KEY"
    },
    "ollama-work": {
      "upstream_base_url": "https://ollama.com/v1",
      "auth_mode": "override",
      "upstream_api_key_env": "OLLAMA_WORK_API_KEY"
    }
  }
}

Clients select a configured non-default profile with the trusted proxy header X-Watchdog-Upstream-Profile: openrouter-work. Unknown profile names are rejected before any upstream request. The dashboard records the selected profile name as the request provider, making account/provider filtering unambiguous.

Proxy routes

Method	Path	Notes
`GET,POST,PUT,PATCH,DELETE`	`/proxy/v1/:resource`	Example: `/proxy/v1/models`
`GET,POST,PUT,PATCH,DELETE`	`/proxy/v1/:resource/:action`	Example: `/proxy/v1/chat/completions`
`GET,POST,PUT,PATCH,DELETE`	`/proxy/v1/:resource/:action/:subaction`	Example: `/proxy/v1/fine_tuning/jobs`
`GET,POST,PUT,PATCH,DELETE`	`/proxy/v1/:resource/:action/:subaction/:tail`	Example: `/proxy/v1/fine_tuning/jobs/<id>/cancel`

Proxy forwarding is broader than telemetry decoding. Chat Completions JSON/SSE has the first endpoint-specific decoder for model, usage, finish-reason, and bounded summaries. Other paths are opaque: Watchdog records transport metadata and correlation but forwards request/response bodies byte-for-byte without trying to interpret provider extensions, multipart uploads, images, audio, files, fine-tuning payloads, or arbitrary JSON as chat telemetry.

The proxy forwards JSON responses and incrementally passes through streaming chat SSE responses when run with a Kujo build containing generic HTTP response streaming. It returns the upstream status/body to the caller and records the first meaningful content delta plus completion or disconnect lifecycle facts. Safe scalar query parameters are forwarded to upstream list and retrieve endpoints, while suspicious path/query text such as path traversal, embedded URL schemes, fragments, and query delimiters in path segments is rejected before any upstream request is made.

Auth modes

`passthrough` (default)

Forwards incoming provider Authorization to upstream. A bearer value equal to the configured Watchdog proxy token is stripped; send X-Watchdog-Proxy-Token separately from provider authorization.
Best when each client profile manages its own key.

`override`

Ignores incoming auth and injects a server-side upstream key.
Set key via WDG_UPSTREAM_API_KEY or WDG_UPSTREAM_API_KEY_ENV.

Watchdog API auth mode

`off` (default)

Leaves Watchdog API endpoints (/api/*) open for local development.

`token`

Requires WDG_API_AUTH_TOKEN for every /api/* request.
Accepts either X-Watchdog-Token: <token> or Authorization: Bearer <token>.

Example:

WDG_API_AUTH_MODE=token \
WDG_API_AUTH_TOKEN='replace-with-long-random-token' \
kujo run --interpreter dashboard_server.kujo

Then call protected routes with:

curl -s http://127.0.0.1:7700/api/stats -H 'X-Watchdog-Token: replace-with-long-random-token'

Proxy route auth mode

`off` (default)

Leaves proxy endpoints (/proxy/*) open.

`token`

Requires WDG_PROXY_AUTHZ_TOKEN for non-allowlisted proxy requests.
Accepts either X-Watchdog-Proxy-Token: <token> or Authorization: Bearer <token>.
Uses exact-path allowlist entries from WDG_PROXY_AUTHZ_ALLOWLIST (default keeps /healthz and /readyz open).

Example:

WDG_PROXY_AUTHZ_MODE=token \
WDG_PROXY_AUTHZ_TOKEN='replace-with-long-random-proxy-token' \
WDG_PROXY_AUTHZ_ALLOWLIST='/healthz,/readyz,/proxy/v1/models' \
kujo run --interpreter dashboard_server.kujo

Non-local deployment hardening

Set WDG_HOST=127.0.0.1 for loopback-only access; the default is 0.0.0.0 (all interfaces).
Treat WDG_API_AUTH_MODE=off as local-only usage.
For non-local access, enable WDG_API_AUTH_MODE=token, enable WDG_PROXY_AUTHZ_MODE=token, and place Watchdog behind firewall rules or a reverse proxy.

Production startup policy

Set WDG_DEPLOYMENT_PROFILE=production for non-local deployments.
In production profile mode, Watchdog fails startup unless all of the following are configured:
- WDG_API_AUTH_MODE=token
- WDG_API_AUTH_TOKEN is non-empty
- WDG_PROXY_AUTHZ_MODE=token
- WDG_PROXY_AUTHZ_TOKEN is non-empty
WDG_ALLOW_INSECURE_STARTUP=true acts as break-glass override for controlled emergency scenarios and prints policy warnings at startup.

Response security headers

Watchdog responses include:

X-Content-Type-Options: nosniff
X-Frame-Options: DENY
Referrer-Policy: no-referrer
Cache-Control: no-store

These headers are applied to dashboard, API, and proxy responses.

Dashboard asset policy

Remote font stylesheets are not used by default; dashboard typography relies on local/system fonts.
Dashboard charts use Dither Kit components installed in source mode with npx @dither-kit/cli.
The generated dither-kit.json lockfile records the installed component versions and hashes.
Built JavaScript and CSS are served locally from /assets/vendor/dither-charts.js and /assets/vendor/dither-charts.css; charts do not require a runtime CDN.
Departure Mono is bundled locally under vendor/fonts/ with its SIL Open Font License and embedded into the dashboard response; titles, metrics, and chart labels use the SiteKit-compatible mono stack.

To update or rebuild the chart assets:

npx @dither-kit/cli update
npm run build:charts

Telemetry redaction policy

WDG_CONTENT_CAPTURE_MODE=off (default) prevents proxy prompt and response summaries from being persisted.
WDG_CONTENT_CAPTURE_MODE=summaries explicitly enables bounded proxy summaries.
WDG_REDACTION_MODE=basic (default) redacts sensitive terms from any enabled summaries, tool-call payload fields, step metadata, and error messages before data is stored or exported.
WDG_REDACTION_MODE=off keeps original text for local debugging workflows.
Extend the matching vocabulary with WDG_REDACT_TERMS to include environment-specific secret markers.

OTLP export

Copy config/exporters.example.json to the path selected by WDG_EXPORTERS_CONFIG_PATH, enable a profile, and run kujo run --interpreter export_worker.kujo once or with --watch. Profiles are isolated by durable queue/checkpoint state; optional destination failure never changes proxy success. Remote endpoints require HTTPS. Loopback HTTP is accepted for local collectors, while private-network HTTPS requires the profile's explicit allow_private_network opt-in.

encoding: "protobuf" is the recommended OTLP/HTTP transport and sends application/x-protobuf; it requires Kujo 1.2.2 or newer for binary request/response bodies. encoding: "json" remains the compatibility default when the field is omitted. Set compression: "gzip" for bounded binary-safe request compression. Mapping profiles are independently pinned as otel.base.v1, otel.genai.v1, or openinference.v1. Grafana/Tempo, Datadog, Honeycomb, Phoenix, Langfuse, and other compatible destinations should normally receive these profiles through their OTLP endpoint or an OpenTelemetry Collector, not a destination-specific Watchdog trace client.

Exporter credentials never belong in the profile. Map a header to an environment-variable name with headers_from_env, for example {"Authorization":"WATCHDOG_OTLP_AUTH"}, then set that environment variable only in the worker process. The worker caps batches at 256 records and 512 KiB, retries bounded transient failures, pauses on authentication rejection, expires old queue entries, and retains bounded dead-letter metadata without raw export payloads.

The guarded trace receiver is POST /telemetry/v2/otlp/v1/traces. It accepts OTLP/HTTP JSON and Protobuf, optionally gzip-compressed, under the same authentication, request-size, rate, and privacy controls as other ingestion. It retains only spans carrying recognized GenAI/OpenInference/Watchdog semantics, returns OTLP partial success for rejected spans, strips content at the authoritative policy boundary, and rejects generic traces, logs, metrics, malformed wire data, and over-limit decompression. Watchdog does not accept arbitrary OpenTelemetry data.

Rate limiting

WDG_RATE_LIMIT_MODE=basic enables lightweight SQLite-backed throttling for both Watchdog API routes and proxy routes. Persisted buckets avoid concurrent dashboard requests corrupting shared process state.
Buckets are keyed by X-Observe-Session-Id when present, otherwise by forwarded IP/host fallbacks.
Use WDG_RATE_LIMIT_MAX_REQUESTS and WDG_RATE_LIMIT_WINDOW_SECS to tune burst tolerance for your environment.
Use WDG_RATE_LIMIT_MAX_BUCKETS and WDG_RATE_LIMIT_BUCKET_TTL_SECS to bound limiter memory and evict stale buckets in long-running deployments.

Automated database backups

Watchdog enables verified SQLite backups by default every 24 hours. Open the Backups dashboard tab to change the frequency, destination folder, retention count, encryption setting, or to create a backup immediately.

Backups use SQLite's online VACUUM INTO path, so Watchdog does not need to stop.
Each backup is checked with PRAGMA quick_check and receives a .sha256 sidecar.
A Dropbox or Google Drive folder can be selected by entering its local synced path.
Scheduling is activity-aware: startup, dashboard refreshes, proxy traffic, and direct telemetry trigger a due check. If Watchdog is idle, the next activity creates the overdue backup before continuing.
Generated files use watchdog-backup-<UTC timestamp>-<suffix>.db or .db.enc; retention only removes files matching that pattern.
The main backup list now shows only files still present in the configured folder. Deleted or missing files stay available under the archived history view, and active backups can be deleted directly from the dashboard.

For encrypted backups, create a dedicated passphrase file outside the repository:

mkdir -p "$HOME/.config/watchdog"
openssl rand -base64 48 > "$HOME/.config/watchdog/backup-encryption-key"
chmod 600 "$HOME/.config/watchdog/backup-encryption-key"
export WDG_BACKUP_ENCRYPTION_KEY_FILE="$HOME/.config/watchdog/backup-encryption-key"

Restart Watchdog, then enable encryption in the Backups tab. Encryption uses AES-256-CBC with PBKDF2-SHA256 and 200,000 iterations. Restore an encrypted file:

openssl enc -d -aes-256-cbc -pbkdf2 -iter 200000 -md sha256 \
  -in watchdog-backup-TIMESTAMP.db.enc \
  -out restored-watchdog.db \
  -pass file:"$HOME/.config/watchdog/backup-encryption-key"
sqlite3 restored-watchdog.db 'PRAGMA quick_check;'

Integrating any OpenAI-compatible client

Set the client base URL to:

http://localhost:7700/proxy/v1

Keep normal OpenAI-style paths, for example chat/completions.

Optional headers for better dashboard grouping:

X-Observe-Session-Id
X-Observe-User-Id
X-Observe-Tenant-Id
X-Observe-Project-Id
X-Observe-Workflow-Id
X-Observe-Task-Id
X-Observe-Correlation-Id

Tenant/project values can also be supplied in JSON request bodies using tenant_id and project_id when headers are not available.

If omitted, the proxy auto-generates values.

Aggregating multiple apps and API keys

One Watchdog server aggregates every app that sends traffic through its /proxy/v1 URL into the same telemetry database. API keys do not partition dashboard data. Give each app a distinct X-Observe-Project-Id (for example, signalbox and ai-chat) and the unfiltered dashboard stats include both; the request table can still be filtered by project when needed.

Keep the three credential roles separate:

WDG_API_AUTH_TOKEN protects Watchdog's dashboard JSON APIs.
WDG_PROXY_AUTHZ_TOKEN controls which clients may use the proxy.
Upstream provider keys authenticate Watchdog to OpenAI-compatible providers.

For several apps using the same upstream account, override mode is simplest: all apps share the Watchdog proxy URL/token while Watchdog injects its single configured upstream key. For apps with different upstream keys, use passthrough mode so each client sends its own upstream Authorization value, and send the Watchdog access credential separately in X-Watchdog-Proxy-Token. Both modes still write to the same database.

Named upstream profiles let one Watchdog process route several provider accounts while retaining one telemetry database and dashboard.

AI Chat integration example (no AI Chat code changes)

For a Kennel-focused end-to-end setup and validation flow, see docs/KENNEL_INTEGRATION_GUIDE.md.

In kujo-ai-chat:

In Settings, choose provider Custom OpenAI-Compatible.
Set profile Base URL to your Watchdog proxy base (for example, https://your-watchdog-host/proxy/v1).
Set the profile API key as usual.
Ensure ALLOWED_CUSTOM_PROVIDER_HOSTS in AI Chat includes your proxy host.

Important: current AI Chat security validation rejects localhost/private hosts for custom providers and requires HTTPS. For local testing, use an HTTPS host that resolves externally (for example, via a secure tunnel) and allowlist that host in ALLOWED_CUSTOM_PROVIDER_HOSTS.

What is logged

Table	Contents
`requests`	Session/user/tenant/project/workflow/task/correlation IDs, provider/model, status, latency, tokens, cost, prompt/response summaries, errors
`tool_calls`	Proxy forward events linked to request rows
`agent_steps`	High-level proxy lifecycle steps (`proxy_received`, `proxy_forwarded`, `proxy_completed`, `proxy_failed`)
`audit_events`	Security-sensitive operations (`api_auth_failure`, `proxy_auth_failure`, `proxy_config_view`, `prune_operation`) with actor key, result, and metadata
`traces`	End-to-end workflow timing, token breakdown, and direct-API-equivalent cost components
`trace_spans`	Provider-neutral workflow, model, tool, persistence, and internal timing spans
`trace_events`	Ordered milestones such as connect, first token, thinking, tool execution, and persistence
`rate_limit_buckets`	Short-lived request counters used when basic throttling is enabled

Watchdog API reference

All endpoints return { "ok": true, "data": ... } on success.

API responses include X-Watchdog-API-Version: v1 so consumers can pin behavior to a stable contract.

Method	Path	Description
`GET`	`/`	Dashboard HTML
`GET`	`/healthz`	Liveness probe
`GET`	`/readyz`	Readiness probe (DB connectivity)
`GET`	`/api/version`	API compatibility/version metadata
`GET`	`/api/proxy-config`	Effective proxy config summary
`GET`	`/api/sources`	Merged, evidence-backed inbound source inventory
`GET`	`/api/sources/:id`	Safe Connected Source detail and bounded evidence
`GET`	`/api/sources/setup-templates`	Non-secret, checked-in source setup templates
`POST`	`/api/sources`	Register source metadata or create a safe named proxy profile
`PATCH`	`/api/sources/:id`	Update allowlisted registration metadata with revision protection
`DELETE`	`/api/sources/:id`	Remove registration metadata while retaining telemetry
`POST`	`/api/sources/:id/verify`	Verify accepted local telemetry without network contact
`GET`	`/api/stats`	Overview stats
`GET`	`/api/requests`	Latest request logs; filter by `source_app`, `data_class`, or `pricing_kind`
`POST`	`/api/telemetry/requests`	Authenticated, idempotent intake for telemetry from trusted local apps
`POST`	`/api/telemetry/traces`	Append independent spans/events to a trace without creating a model-request row
`GET`	`/api/traces`	Latest granular traces
`GET`	`/api/trace-spans`	Latest spans; filter by `trace_id`
`GET`	`/api/trace-events`	Latest events; filter by `trace_id`
`GET`	`/api/tool-calls`	Latest tool call logs
`GET`	`/api/agent-steps`	Agent step traces
`GET`	`/api/audit-events`	Structured security/audit event stream
`GET`	`/api/errors`	Error aggregates
`GET`	`/api/sessions`	Per-session aggregates
`GET`	`/api/insights`	Agent run outcomes, tool effectiveness, latency by span kind, workflow transitions, and context-pressure summaries
`GET`	`/api/telemetry/v2/records`	Canonical records with bounded logical-request, application, outcome, trace, kind, producer, and time filters
`GET`	`/api/telemetry/v2/observability`	Canonical attempt chains, lifecycle outcomes, evidence, and lineage diagnostics
`GET`	`/api/charts/requests-over-time`	Hourly request/error counts
`GET`	`/api/charts/stat-trends`	Time-bucketed request, cost, latency, error, token, session, tool, and trace-span metrics
`GET`	`/api/charts/cost-over-time`	Hourly cost totals
`GET`	`/api/charts/latency-hist`	Latency buckets
`GET`	`/api/charts/status-breakdown`	Success vs error counts
`GET`	`/api/charts/provider-breakdown`	Per-provider/model breakdown
`GET`	`/api/admin/diagnostics`	Protected runtime/migration/DB diagnostics summary
`GET`	`/api/admin/backups`	Backup settings, schedule status, and recent run history
`PUT`	`/api/admin/backups/settings`	Persist backup frequency, folder, retention, enablement, and encryption mode
`POST`	`/api/admin/backups/run`	Create and verify a backup immediately
`POST`	`/api/admin/backups/delete`	Delete a backup file (and checksum sidecar) while keeping its archived record
`POST`	`/api/admin/prune`	Prune old telemetry rows (supports dry-run)
`POST`	`/api/admin/prune-fixtures`	Remove only rows explicitly classified as fixture data (supports dry-run)
`GET`	`/api/export`	Full export (`json` default or `jsonl`/`ndjson`)

The granular contract is optional and producer-neutral. Watchdog is a passive collector: applications, model providers, and tool executors remain independently usable, and a tool can append its own telemetry without importing or depending on another tool. New producers should send schema_version: "kujo.telemetry.v1"; the checked-in contract is schemas/telemetry-trace-v1.schema.json. Trace metrics are cumulative absolute values, and trace/span/event/tool identities are replay-safe for durable at-least-once delivery. See Granular Tracing.

The Sources tab and watchdog.sources-panel.v1 API report inbound producers without conflating them with exporter destinations. Status is based only on configuration and accepted local telemetry. Native/OTLP registration changes manage display/setup metadata; they do not start, stop, or disconnect another application. Named proxy profile changes are saved to the existing proxy configuration and apply to new requests without restarting Watchdog. In-flight requests retain the profile snapshot they started with. See Connected Sources.

/api/insights reports observed legacy telemetry: runs are grouped by task ID, workflow ID, or session ID; a success signal means no request error was recorded; retry signals are inferred from retry-named trace events; and its token-volume section does not measure context capacity. The Canonical Evidence dashboard tab uses explicit terminal events, typed relations, nullable timing, versioned context limits, and distinct cost kinds instead.

List endpoints now support optional query parameters for pagination and filtering:

page (default 1)
page_size (default endpoint-specific, capped)
common filters where applicable: session_id, user_id, tenant_id, project_id, workflow_id, task_id, correlation_id, status, provider, model, tool_name, agent_id, step_type
time window: since_ms, until_ms (epoch milliseconds)
export format: format=json|jsonl|ndjson (/api/export; invalid values return 400)
export row bound: max_rows (/api/export; default from WDG_EXPORT_MAX_ROWS, max 50000)
export chunk cursor: cursor and chunk_size (/api/export; works for json/jsonl/ndjson; max_rows caps the effective chunk size per exported kind)
tenant/project scoping extends to /api/sessions and chart endpoints such as /api/charts/requests-over-time, /api/charts/status-breakdown, and /api/charts/provider-breakdown

Retention and export control examples:

Dry-run prune before timestamp: curl -s -X POST http://127.0.0.1:7700/api/admin/prune -H 'Content-Type: application/json' -d '{"before_ms": 1700000000000, "dry_run": true}'
Filtered export by session/time window: curl -s "http://127.0.0.1:7700/api/export?session_id=sess_123&since_ms=1700000000000&until_ms=1800000000000"
Filtered export by tenant/project scope: curl -s "http://127.0.0.1:7700/api/export?tenant_id=team_alpha&project_id=checkout-api"
Row-bounded export for controlled payload size: curl -s "http://127.0.0.1:7700/api/export?format=json&max_rows=500"
Cursor-based chunk export progression: curl -s "http://127.0.0.1:7700/api/export?format=json&chunk_size=500&cursor=0"
JSONL export for pipeline ingestion: curl -s "http://127.0.0.1:7700/api/export?format=jsonl&session_id=sess_123"

Benchmark script

Run quick and soak benchmark profiles with human-readable output plus JSON summary output:

node scripts/benchmark_profiles.js --profiles=quick,soak --json-out=tmp/benchmark-report.json

Run deterministic fixture mode for local validation/schema checks:

node scripts/benchmark_profiles.js --fixture --profiles=quick,soak --json-out=tmp/benchmark-fixture.json

Constrained CI or Workcell environments may set WDG_LOAD_MIN_RPS or WDG_LOAD_MIN_RPS_QUICK/WDG_LOAD_MIN_RPS_SOAK to document the local CPU budget. They may also set WDG_LOAD_MAX_P95_MS or WDG_LOAD_MAX_P95_MS_QUICK/WDG_LOAD_MAX_P95_MS_SOAK when container scheduling makes strict local-laptop latency budgets unrealistic. The suite still runs the live proxy, dashboard latency, and SQLite growth assertions.

Project structure

watchdog/
├── kujo.toml              # Project metadata
├── src/                   # Canonical implementation sources
│   ├── dashboard_server.kujo
│   ├── dashboard.html
│   ├── watchdog.kujo
│   └── watchdog_shared.kujo
├── dashboard_server.kujo  # Root compatibility entrypoint (mirrors src/dashboard_server.kujo)
├── dashboard.html         # Root compatibility entrypoint (mirrors src/dashboard.html)
├── watchdog.kujo          # Root compatibility entrypoint (mirrors src/watchdog.kujo)
├── watchdog_shared.kujo   # Root compatibility shared module (mirrors src/watchdog_shared.kujo)
├── demo.kujo              # Demo data seeding script
├── scripts/               # Operational scripts (benchmarks, helpers)
├── tests/                 # Contract, static, proxy, and runtime checks
│   └── fixtures/          # Non-product runtime fixtures
├── data/                  # Runtime SQLite files (created at runtime)
└── docs/                  # Backlogs and implementation checklists

Keep root compatibility entrypoints synced from src/:

node scripts/sync_compat_entrypoints.js --check

Implementation backlog

Native producer integration contract: docs/native-producers.md
Connected Sources management panel implementation prompt: docs/plans/connected-sources-management/IMPLEMENTATION_PROMPT.md
Primary execution checklist: docs/ENTERPRISE_RELEASE_LOOP_CHECKLIST.md
Deployment and hardening runbook: docs/DEPLOYMENT_HARDENING_RUNBOOK.md
Kennel integration guide: docs/KENNEL_INTEGRATION_GUIDE.md
Enterprise deployment architecture: docs/ENTERPRISE_DEPLOYMENT_ARCHITECTURE.md
Enterprise release loop checklist: docs/ENTERPRISE_RELEASE_LOOP_CHECKLIST.md
Release checklist and versioning policy: docs/RELEASE_CHECKLIST.md
Security policy: SECURITY.md
Contribution guide: CONTRIBUTING.md
Changelog: CHANGELOG.md
CI workflow: .github/workflows/watchdog-ci.yml
Use the checklist Item Completion Template and Work Log format when completing backlog items.

Requirements

Kujo CLI/runtime installed
Node.js and the sqlite3 CLI for automated backups
OpenSSL when encrypted backups are enabled
No API key required to run the dashboard itself
API key required only when proxying to an authenticated upstream

Canonical batch intake bounds

POST /telemetry/v2/batches requires 1–100 record objects in the raw request. Oversized arrays and non-object entries return HTTP 400 before privacy filtering; they are never silently reduced to a successful partial batch. Valid batches are privacy-filtered once, then retain repository validation, identity-conflict checks and transactional persistence.

Canonical repository intake reuses each validated record's stable JSON and SHA-256 within its bounded batch for conflict checks and insertion. It does not cache records across requests or bypass privacy, validation, or transactional conflict rejection. Legacy rows without stored hashes still backfill from their canonical JSON. The identity-conflict fixture covers mixed duplicate/new batches and this legacy retry path.

Canonical HTTP intake relies on the repository's structural validation before its transaction rather than repeating the same full validation in the route. Raw container bounds and authoritative privacy projection still precede it; validation failures return HTTP 400 and identity conflicts return HTTP 409.

Kujo 1.6 runtime measurements

Attach bounded runtime summaries to existing execution observations; retain their exact-byte artifact references. See adapter contract and offline proof.

Concurrent canonical intake

Disk-backed canonical batches use a dedicated SQLite connection per request so overlapping HTTP callbacks cannot nest transactions on the dashboard connection. Validation, privacy filtering, atomic insertion and identity-conflict checks remain unchanged. Private :memory: stores retain their shared connection semantics and are not qualified for concurrent intake. Run the bounded regression with node tests/canonical_concurrency_check.js (Node with node:sqlite, Kujo available).

Repository verification

Run KUJO_BIN=/path/to/kujo npm test for all self-contained JavaScript and MJS checks. Logs and a structured receipt are saved in tmp/verification/; override that directory with WDG_VERIFICATION_DIR. Set AGENTS_SDK_PATH to include the external Agents SDK integration (otherwise explicitly reported as skipped). Run npm ci --ignore-scripts, npm run build:charts, and npm run typecheck for dashboard assets. CI also rejects generated asset drift.

The local watchdog_proxy_config.json is ignored; start from config/proxy.example.json when file configuration is needed. Environment-only startup continues to work.

Name		Name	Last commit message	Last commit date
.github		.github
adapters		adapters
clients/javascript		clients/javascript
components/dither-kit		components/dither-kit
config		config
data		data
docs		docs
schemas		schemas
scripts		scripts
src		src
tests		tests
vendor		vendor
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
VERSION		VERSION
components.json		components.json
dashboard.html		dashboard.html
dashboard_server.kujo		dashboard_server.kujo
demo.kujo		demo.kujo
dither-kit.json		dither-kit.json
export_worker.kujo		export_worker.kujo
kennel.toml		kennel.toml
kujo.toml		kujo.toml
package-lock.json		package-lock.json
package.json		package.json
runtime_measurements_adapter.kujo		runtime_measurements_adapter.kujo
source_management.kujo		source_management.kujo
tailwind.config.js		tailwind.config.js
telemetry_delivery.kujo		telemetry_delivery.kujo
telemetry_native_adapter.kujo		telemetry_native_adapter.kujo
telemetry_otlp.kujo		telemetry_otlp.kujo
telemetry_otlp_ingest.kujo		telemetry_otlp_ingest.kujo
telemetry_otlp_protobuf.kujo		telemetry_otlp_protobuf.kujo
telemetry_repository.kujo		telemetry_repository.kujo
telemetry_v2.kujo		telemetry_v2.kujo
tsconfig.json		tsconfig.json
watchdog.kujo		watchdog.kujo
watchdog.spec.yml		watchdog.spec.yml
watchdog_shared.kujo		watchdog_shared.kujo

Latest commit

History

Folders and files

Repository files navigation

Watchdog

Current release

How it works

Quick start

1) Start the Watchdog server

2) Open the dashboard

3) Verify proxy configuration endpoint

4) Optional smoke test (passthrough auth mode)

Canonical examples and generated paths

Configuration

Named upstream profiles

Proxy routes

Auth modes

passthrough (default)

override

Watchdog API auth mode

off (default)

token

Proxy route auth mode

off (default)

token

Non-local deployment hardening

Production startup policy

Response security headers

Dashboard asset policy

Telemetry redaction policy

OTLP export

Rate limiting

Automated database backups

Integrating any OpenAI-compatible client

Aggregating multiple apps and API keys

AI Chat integration example (no AI Chat code changes)

What is logged

Watchdog API reference

Benchmark script

Project structure

Implementation backlog

Requirements

Canonical batch intake bounds

Kujo 1.6 runtime measurements

Concurrent canonical intake

Repository verification

About

Topics

Resources

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`passthrough` (default)

`override`

`off` (default)

`token`

`off` (default)

`token`