feat: the browser's logs reach Loki over OTel, and the collector decides what to drop

AGENTS.md

@@ -721,7 +721,7 @@ Mutations with `meta: { inlineErrors: true }` (the login and register forms) ski
 
 ## Frontend: Logging
 
-The frontend logs through LogTape, and nothing else: ESLint's `no-console` rejects `console.*` in `src/`. `init` in `src/hooks.client.ts` configures it once, sending the `integr8scode` category to the browser console from `debug` up in dev and from `warning` up in a production build. Each module takes its own logger and writes a sentence whose `{placeholders}` name every value it passes, so a line reads on its own:
+The frontend logs through LogTape, and nothing else: ESLint's `no-console` rejects `console.*` in `src/`. `init` in `src/hooks.client.ts` configures it once: every build sends every record to the collector through LogTape's OpenTelemetry sink, the way the backend sends its logs, and nothing goes to the browser console. Loki labels the lines `service_name="browser"`, and the collector's `filter/browser` decides what's dropped: below `warning` when `ENVIRONMENT` is `production`, nothing anywhere else. `init` also logs uncaught errors and unhandled rejections from window listeners. Each module takes its own logger and writes a sentence whose `{placeholders}` name every value it passes, so a line reads on its own:
 
 ```typescript
 import { getLogger } from '@logtape/logtape';
@@ -732,7 +732,7 @@ log.warning('Notification stream lost its connection: {error}', { error });
 
 - The category is `['integr8scode', area]`; the areas today are `app`, `auth`, `api`, `editor`, `notifications` and `replay`
 - Values go in the properties, never into the message string, the same rule as the backend's structlog
-- `error` is for something broken (a 5xx, no response, a stream handler that threw), `warning` for a degraded state the app recovers from (a dropped SSE connection), `debug` for what the user already sees in a toast or a form (a 4xx)
+- `error` is for something broken (a 5xx, no response, a stream handler that threw) and fires the error-lines alert, `warning` for a degraded state the app recovers from (a dropped SSE connection), `debug` for what the user already sees in a toast or a form (a 4xx)
 - Unit tests leave LogTape unconfigured, so it drops every record and no test silences the console. A test that checks a log line calls `captureLogs()` from `#test/test-utils.ts` and asserts on the records
 
 ## Frontend: Session

backend/otel-collector-config.yaml

@@ -6,6 +6,13 @@ receivers:
       http:
         endpoint: 0.0.0.0:4318
 
+  # The browser's logs, through nginx's /otlp/v1/logs; anyone can post here, so they get a pipeline of their own
+  otlp/browser:
+    protocols:
+      http:
+        endpoint: 0.0.0.0:4319
+        max_request_body_size: 65536
+
   # Every container's stdout and stderr, from Docker's fluentd log driver; the tag is the container name
   fluent_forward:
     endpoint: 0.0.0.0:24224
@@ -847,6 +854,22 @@ processors:
         statements:
           - set(resource.attributes["service.name"], resource.attributes["fluent.tag"]) where resource.attributes["fluent.tag"] != nil
 
+  # Whatever resource a browser sends, its records are labelled browser
+  transform/browser:
+    error_mode: ignore
+    log_statements:
+      - context: resource
+        statements:
+          - keep_keys(resource.attributes, [])
+          - set(resource.attributes["service.name"], "browser")
+
+  # The browser sends every level; production keeps its warnings and errors, any other environment all of it
+  filter/browser:
+    error_mode: ignore
+    logs:
+      log_record:
+        - severity_number < SEVERITY_NUMBER_WARN and "${env:ENVIRONMENT}" == "production"
+
   # Docker's lines arrive without a level, and Loki's own guess marks Mongo and Redis info lines as errors
   transform/level:
     error_mode: ignore
@@ -948,4 +971,9 @@ service:
     logs:
       receivers: [otlp, fluent_forward]
       processors: [memory_limiter, filter/app_json_stdout, groupbyattrs/container, transform/container, transform/level, batch, resource]
-      exporters: [otlp_http/loki, debug]
+      exporters: [otlp_http/loki, debug]
+
+    logs/browser:
+      receivers: [otlp/browser]
+      processors: [memory_limiter, filter/browser, transform/browser, batch, resource]
+      exporters: [otlp_http/loki]

docker-compose.yaml

@@ -401,6 +401,7 @@ services:
       # The backend creates this user on start; until then the mongodb receiver's scrapes fail and retry
       - MONGO_MONITOR_USER=${MONGO_MONITOR_USER:-otel-monitor}
       - MONGO_MONITOR_PASSWORD=${MONGO_MONITOR_PASSWORD:?see .env.example}
+      - ENVIRONMENT=${ENVIRONMENT:-development}
     ports:
       - "127.0.0.1:4317:4317"   # OTLP gRPC
       - "127.0.0.1:4318:4318"   # OTLP HTTP

docs/operations/logging.md

@@ -76,4 +76,8 @@ That keeps fields queryable, and it stops log injection. If user input goes into
 
 ## Frontend
 
-The browser logs through [LogTape](https://logtape.org), configured once in `init` in `frontend/src/hooks.client.ts`. Lines go to the browser console only, under the `integr8scode` category plus an area (`api`, `auth`, `editor`, `notifications`, `replay`, `app`), from `debug` up in dev and from `warning` up in a production build. Values travel as properties named by `{placeholders}` in the message, the same rule as the backend's keyword arguments, and ESLint's `no-console` keeps every other way of logging out of `src/`. Nothing ships these lines to the backend or Loki yet.
+The browser logs through [LogTape](https://logtape.org), configured once in `init` in `frontend/src/hooks.client.ts`, under the `integr8scode` category plus an area (`api`, `auth`, `editor`, `notifications`, `replay`, `app`). Values travel as properties named by `{placeholders}` in the message, the same rule as the backend's keyword arguments, and ESLint's `no-console` keeps every other way of logging out of `src/`.
+
+Nothing goes to the browser console. Every build sends every record to the collector the way the backend sends its logs: LogTape's OpenTelemetry sink ([@logtape/otel](https://github.com/dahlia/logtape/tree/main/packages/otel)) hands them to an OTel `LoggerProvider` with a `BatchLogRecordProcessor` and an `OTLPLogExporter`, which posts OTLP JSON to `/otlp/v1/logs`. nginx passes that to the collector ([Nginx](nginx-configuration.md#browser-logs)), and Loki labels the lines `service_name="browser"`. What gets dropped is the collector's call, not the browser's: `filter/browser` drops lines below `warning` when `ENVIRONMENT` is `production` and keeps everything in any other environment ([Loki](loki.md#collector)). Without the observability profile nginx answers 502 and the records are lost. Besides the errors the app logs itself, `handleError` logs what breaks while a page loads or renders, and two window listeners log an uncaught error and an unhandled promise rejection. An `error` event without an error object, like Chromium's "ResizeObserver loop completed with undelivered notifications", is skipped.
+
+Three things in that setup are there for the browser. The sink's default setup loads its exporter with an `import()` a browser can't resolve, so `init` builds the `LoggerProvider` itself, as [LogTape #166](https://github.com/dahlia/logtape/issues/166) suggests. Its default object renderer falls back to `JSON.stringify` in a browser, and an Error comes out as `{}`, so the sink uses `objectRenderer: 'json'`, which writes the error's name, message and stack into the line. And @logtape/otel 2.4 asks for the OTel 0.208 packages, whose `@opentelemetry/core` carries [GHSA-8988-4f7v-96qf](https://github.com/advisories/GHSA-8988-4f7v-96qf), so the `overrides` in package.json move its SDK and exporters to the 0.223 line. `@opentelemetry/api-logs` stays at 0.208 for it, since it imports `NOOP_LOGGER` and later versions dropped that export.

docs/operations/loki.md

@@ -21,7 +21,9 @@ The OTel resource on every record has `service.name` from `SERVICE_NAME` (`integ
 
 ## Collector
 
-The logs pipeline receives OTLP and container output (`fluent_forward`), runs `memory_limiter`, `filter/app_json_stdout`, `groupbyattrs/container`, `transform/container`, `batch` and `resource`, and exports with `otlp_http/loki` to `http://loki:3100/otlp`. The full config is [`backend/otel-collector-config.yaml`](https://github.com/HardMax71/Integr8sCode/blob/main/backend/otel-collector-config.yaml).
+The logs pipeline receives OTLP and container output (`fluent_forward`), runs `memory_limiter`, `filter/app_json_stdout`, `groupbyattrs/container`, `transform/container`, `transform/level`, `batch` and `resource`, and exports with `otlp_http/loki` to `http://loki:3100/otlp`. The full config is [`backend/otel-collector-config.yaml`](https://github.com/HardMax71/Integr8sCode/blob/main/backend/otel-collector-config.yaml).
+
+The browser's records come in on a pipeline of their own, `logs/browser`. Its receiver, `otlp/browser`, takes OTLP over HTTP on port 4319 with bodies up to 64 KB, and only nginx's `/otlp/v1/logs` reaches it ([Frontend logging](logging.md#frontend)). The browser sends every level, and `filter/browser` is where lines get dropped: below `warning` when the collector's `ENVIRONMENT` is `production`, none anywhere else. Anyone can post to that path, so `transform/browser` drops whatever resource a record arrives with and sets `service.name` to `browser`: a posted record can't pass as the backend or any other service. The Errors dashboard and the error-lines alert group by `service_name`, so `browser` shows up there like any container.
 
 ## Container output
 

docs/operations/nginx-configuration.md

@@ -1,12 +1,13 @@
 # Nginx configuration
 
-The frontend container is nginx. It serves the Svelte build, proxies `/api/` to the backend, and has a second server block that proxies Grafana on its own hostname. The config is [`frontend/nginx.conf.template`](https://github.com/HardMax71/Integr8sCode/blob/main/frontend/nginx.conf.template).
+The frontend container is nginx. It serves the Svelte build, proxies `/api/` to the backend and `/otlp/v1/logs` to the collector, and has a second server block that proxies Grafana on its own hostname. The config is [`frontend/nginx.conf.template`](https://github.com/HardMax71/Integr8sCode/blob/main/frontend/nginx.conf.template).
 
 ```mermaid
 flowchart LR
     Browser -->|"app host :5001"| App["server _"]
     Browser -->|"grafana.integr8scode.cc :5001"| G["server grafana.integr8scode.cc"]
     App -->|"/api/*"| Backend["${BACKEND_URL}"]
+    App -->|"/otlp/v1/logs"| Collector["otel-collector:4319"]
     App -->|"/grafana"| Redirect["302 to ${GRAFANA_PUBLIC_URL}"]
     App -->|"everything else"| Static["static files, SPA fallback"]
     G --> Grafana["${GRAFANA_URL}"]
@@ -62,6 +63,14 @@ The backend's address goes through a variable and Docker's resolver, like Grafan
 
 The SSE location under `/api/v1/events/` speaks HTTP/1.1 with an empty `Connection` header, turns off buffering and caching, and raises both timeouts to 24 hours so long-lived streams aren't cut after the default 60 seconds. `proxy_buffering off` alone is enough to pass each event through as it arrives.
 
+### Browser logs
+
+```nginx
+--8<-- "frontend/nginx.conf.template:browser_logs"
+```
+
+The app sends its log records here as OTLP JSON, at most one batch a second, and nginx passes them to the collector's `otlp/browser` receiver ([Loki](loki.md#collector) has the rest of the path). Anyone can post to this path, so it takes POST only, bodies up to 64 KB, and 30 requests a minute per address with a burst of 30, from the `limit_req_zone` at the top of the file; past that nginx answers 429. The upstream goes through a variable like the backend's, so nginx starts without the observability profile and this path returns 502 instead.
+
 ### Static files and SPA routing
 
 ```nginx

docs/reference/configuration.md

@@ -107,7 +107,7 @@ A language can raise its own pod's limits above the system settings. Go gets at
 | `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://otel-collector:4317` | config.toml | OTLP endpoint for metrics and logs; unset turns both off |
 | `SERVICE_NAME` | `integr8scode-backend` | config.toml | `service.name` on traces, metrics and logs, Kafka client id, event metadata; `config.worker.toml` sets `integr8scode-worker` |
 | `SERVICE_VERSION` | `1.0.0` | config.toml | `service.version` on traces, metrics and logs |
-| `ENVIRONMENT` | `development` | env var, then TOML | `deployment.environment` attribute; `test` also turns off notification throttling. Read from the environment like the secrets, so the production deploy writes `ENVIRONMENT=production` into `.env` |
+| `ENVIRONMENT` | `development` | env var, then TOML | `deployment.environment` attribute; `test` also turns off notification throttling. Read from the environment like the secrets, so the production deploy writes `ENVIRONMENT=production` into `.env`. The collector gets it too and drops the browser's lines below `warning` in `production` |
 | `HOSTNAME` | the machine's hostname | default | `service.instance.id` on logs and spans; inside a container that's the container ID |
 | `LOG_LEVEL` | `INFO` | config.toml | structlog level |
 

frontend/nginx.conf.template

@@ -1,3 +1,5 @@
+limit_req_zone $binary_remote_addr zone=browser_logs:1m rate=30r/m;
+
 # --8<-- [start:server_block]
 server {
     listen 5001 ssl;
@@ -83,6 +85,17 @@ server {
         # --8<-- [end:sse_proxy]
     }
 
+    # --8<-- [start:browser_logs]
+    location = /otlp/v1/logs {
+        limit_except POST { deny all; }
+        limit_req zone=browser_logs burst=30 nodelay;
+        limit_req_status 429;
+        client_max_body_size 64k;
+        set $browser_logs_upstream http://otel-collector:4319/v1/logs;
+        proxy_pass $browser_logs_upstream;
+    }
+    # --8<-- [end:browser_logs]
+
     # Browsers and crawlers ask for /favicon.ico, the app ships favicon.png, and each miss logged an nginx [error]
     location = /favicon.ico {
         try_files /favicon.png =404;