diff --git a/.env.example b/.env.example index df8f639..ce9f60f 100644 --- a/.env.example +++ b/.env.example @@ -73,6 +73,9 @@ RATE_LIMIT_WINDOW_MS=900000 # Maximum requests per client per window before a 429 is returned. [default: 100] RATE_LIMIT_MAX_REQUESTS=100 +## Prometheus metrics endpoint (disabled by default; admin JWT required) +ENABLE_METRICS=false + # Outbound webhook that receives dispute lifecycle notifications. Empty string # (the default) disables notifications. [default: "" — set to enable] # NOTE: declared in the schema but not yet read by any service; wire it up in diff --git a/README.md b/README.md index 21b13f5..5f92aef 100644 --- a/README.md +++ b/README.md @@ -376,3 +376,8 @@ linked PR 5,4,6 ## 📄 License This project is licensed under the ISC License. + + +## Prometheus metrics endpoint + +Operational metrics are available at `GET /metrics` and `GET /api/v1/metrics` when `ENABLE_METRICS=true`. The endpoint is disabled by default and requires a valid admin Bearer token when enabled. See [Prometheus metrics and operations guidance](docs/prometheus-metrics.md) for metric names, scrape setup, and alert recommendations. diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..a29e943 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,23 @@ +# Deployment notes + +## Prometheus metrics endpoint + +The operational metrics endpoint is disabled by default. To enable it in a deployment environment, set the validated variable in the service environment: + + ENABLE_METRICS=true + +Restart the application after changing this setting. No database migration is required. + +When enabled, the protected endpoint is available at GET /metrics and GET /api/v1/metrics. Both require a valid Bearer JWT for a user with the admin role. Use HTTPS and restrict the endpoint to monitoring infrastructure through a private network, firewall, or trusted authentication proxy. Do not place long-lived admin credentials in source control or commit scrape tokens to a Prometheus configuration file. + +A Prometheus-compatible scraper must send an Authorization: Bearer header. Use a secret file or a trusted proxy that can securely rotate credentials. If your deployment cannot supply rotating Bearer tokens safely, keep ENABLE_METRICS=false until an appropriate authentication boundary is configured. + +For exported metric names, scrape configuration guidance, and suggested alerts, see [Prometheus metrics](prometheus-metrics.md). + +## Operational safety + +- Confirm the endpoint returns 404 while ENABLE_METRICS is false. +- Confirm a request without a valid admin token receives an authentication error when metrics are enabled. +- Confirm an authenticated non-admin receives 403. +- Confirm a valid admin token receives Prometheus text exposition with a text/plain content type. +- Monitor scrape latency: the indexer lag collector reads the persisted checkpoint and requests the current network ledger, so scrapes may be slower when the Stellar RPC service is degraded. diff --git a/docs/prometheus-metrics.md b/docs/prometheus-metrics.md new file mode 100644 index 0000000..7a96300 --- /dev/null +++ b/docs/prometheus-metrics.md @@ -0,0 +1,62 @@ +# Prometheus metrics + +SwiftChain Backend exposes a Prometheus text exposition endpoint for operational monitoring. + +## Enable and access + +Set this validated environment variable in .env: + + ENABLE_METRICS=true + +The endpoint is disabled by default. When disabled, both paths return 404: + +- GET /metrics +- GET /api/v1/metrics + +When enabled, both require a valid Bearer JWT for a user with the admin role. Keep the endpoint behind TLS and an operator-only network boundary; do not expose admin credentials in Prometheus scrape configuration. Use a dedicated monitoring identity and rotate its token according to your security policy. + +Example request: + + curl -H "Authorization: Bearer $METRICS_ADMIN_TOKEN" http://localhost:3000/metrics + +The response uses text/plain; version=0.0.4; charset=utf-8 and can be scraped by Prometheus-compatible collectors. + +## Exported metrics + +| Metric | Type | Meaning | +| --- | --- | --- | +| swiftchain_http_requests_total | Counter | Completed HTTP requests, labelled by method, Express route template, and status code | +| swiftchain_http_request_duration_seconds | Histogram | Request duration buckets, sum, and count | +| swiftchain_process_uptime_seconds | Gauge | Process uptime | +| swiftchain_process_resident_memory_bytes | Gauge | Resident memory | +| swiftchain_process_heap_used_bytes / swiftchain_process_heap_total_bytes | Gauge | Node.js heap usage | +| swiftchain_mongodb_connection_ready | Gauge | Whether Mongoose is connected | +| swiftchain_mongodb_pool_stats_available | Gauge | Whether the installed MongoDB driver exposes pool statistics | +| swiftchain_mongodb_pool_connections | Gauge | Total, available, and in-use MongoDB pool connections | +| swiftchain_redis_healthy | Gauge | Result of a Redis PING check | +| swiftchain_circuit_breaker_open | Gauge | Circuit-breaker state by registered breaker | +| swiftchain_indexer_lag_available | Gauge | Whether a lag snapshot could be read | +| swiftchain_indexer_lag_ledgers | Gauge | Gap between persisted processed ledger and the live network ledger | +| swiftchain_socket_* | Gauge / counter | Current connection and latency snapshots are gauges; cumulative connection, disconnection, and message totals are counters | + +HTTP route labels use Express route templates rather than raw request URLs, preventing user IDs, arbitrary paths, and query values from creating high-cardinality labels. The indexer metric reads the current checkpoint and live ledger without persisting alerts or sending webhooks during a scrape. + +MongoDB pool metrics depend on runtime pool fields exposed by the installed MongoDB driver. Check swiftchain_mongodb_pool_stats_available before relying on pool connection values. + +## Suggested scrape configuration + +Configure your Prometheus-compatible scraper to use the application URL, HTTPS in production, and an authorization mechanism that securely supplies a short-lived admin token. Do not commit tokens or other credentials to prometheus.yml. If your scraper cannot safely rotate a Bearer token, place a trusted authentication proxy in front of the endpoint and restrict direct network access. + +## Initial alert guidance + +Treat these as starting points and tune them to the service's normal traffic: + +- HTTP 5xx ratio remains above 5% for 5 minutes. +- Request p95 latency exceeds the service's agreed SLO for 5 minutes. +- swiftchain_mongodb_connection_ready is 0. +- swiftchain_redis_healthy is 0. +- Any swiftchain_circuit_breaker_open sample is 1. +- swiftchain_indexer_lag_ledgers remains above INDEXER_LAG_ALERT_THRESHOLD. +- MongoDB pool in-use connections remain close to total connections while request latency rises. + +All in-memory request counters and socket totals are process-local and reset when that instance restarts. Scrape each instance separately and aggregate in the monitoring backend; do not treat one instance's values as a fleet-wide total. This endpoint exports metrics; it does not configure Prometheus, Grafana, Datadog, alert delivery, or retention automatically. diff --git a/src/app.ts b/src/app.ts index e1c8a53..ea969b3 100644 --- a/src/app.ts +++ b/src/app.ts @@ -8,6 +8,8 @@ import rateLimit from 'express-rate-limit'; import swaggerUi from 'swagger-ui-express'; import routes from './routes'; +import metricsRoutes from './routes/metricsRoutes'; +import prometheusMetricsMiddleware from './middleware/prometheusMetricsMiddleware'; import logger from './config/logger'; import { sendError } from './utils/responseWrapper'; import { connectDatabase } from './config/database'; @@ -37,6 +39,7 @@ app.use(compression()); // Track in-flight requests and reject new ones during graceful shutdown. app.use(requestTracker); app.use(requestLogger); +app.use(prometheusMetricsMiddleware); // Swagger UI needs inline