oscope is an embeddable, local-first telemetry explorer for Jolt programs.
One bounded, serializable EDN contract drives a server-rendered web interface,
a Glitter/GTK native interface, and a Glimmer native interface. Live sources
query the ClickStack-shaped OpenTelemetry tables provided by
jolt-otel-clickhouse in an embedded, in-process chDB database.
This repository is an initial extraction from the observability integration demo and native UI spike. It is standalone: its manifests contain no mutable sibling source paths and its source contains no demo namespaces.
- spans, logs, and metrics distribution queries through a closed allowlist;
- reusable metric-series recipes with fixed time buckets, service/unit/scope/ environment dimensions, and named count/sum/min/max/avg/p50/p95/p99 fields;
- reset-aware cumulative monotonic Sum recipes with exact observed increase, per-second rate, stored provenance, interval duration, and reset counts;
- a ClickStack-style trace workbench with bounded service, operation, status, duration, and time filters, complete parent/child span trees, span events, and trace-correlated logs;
- a raw event explorer for bounded log-body/severity and metric-name/kind searches, with trace links from correlated log records;
- 15 minute, 1 hour, 6 hour, and 24 hour bounded windows;
- a semantic accessible table and a validated Plotje-compatible chart spec;
- portable spec-to-SVG rendering for line, point, bar, area, rule, and tick marks, with bounded palettes, colors, opacity, sizing, and grid options;
- mountable Plotje and safe-Hiccup editors with bounded, versioned edit documents, an in-page grammar reference and loadable examples, a server-rendered fallback, and progressive previews;
- a standalone loopback OTLP/HTTP JSON receiver and viewer using one process, one connection, and one schema owner;
- static-first, high-contrast, responsive Ring UIs: the trace workbench at
/oscope/telemetryand aggregate/chart explorer at/oscope; - an opt-in live web mode with bounded refresh, stale-response rejection, and an exact Freeze-for-export snapshot while preserving the static default;
- raw Arrow and Parquet downloads for spans, logs, gauges, sums, and histograms through a closed export contract;
- Glitter and Glimmer adapters consuming exactly the same screen model;
- caller-owned collector connections and oscope-owned connections with explicit, idempotent retirement; and
- deterministic headless tests plus an opt-in real-chDB lifecycle gate.
The distribution, metric-series, and counter-series query plans contain no SQL. User input
selects only values from closed sets plus an exact bounded metric name. The
shared exporter library maps those recipes to parameterized queries; callers
cannot supply identifiers, aggregate functions, or expressions. Raw export is
a separate versioned data-only command: oscope maps its closed signal and
metric-kind choices to one of five physical tables, then generates a
parameterized SELECT. Export requests cannot supply SQL, table or column
names, filesystem paths, or filenames.
Install or point Jolt at libchdb.so, then start the persistent local
collector. This command form works in fish as well as POSIX shells:
cd oscope
env JOLT_CHDB_LIB=/path/to/libchdb.so jolt -M:serverOscope listens only on 127.0.0.1:4318, stores data in
chdb:./oscope-data, receives OTLP/HTTP JSON at /v1/traces, /v1/logs, and
/v1/metrics, and serves the trace workbench at
http://127.0.0.1:4318/oscope/telemetry. / redirects to that workbench;
the aggregate logs/metrics/chart explorer remains at
http://127.0.0.1:4318/oscope, and raw log/metric rows are available at
http://127.0.0.1:4318/oscope/events. /healthz reports process health, and
/oscope/export serves bounded Arrow or Parquet downloads.
The viewer's Edit this chart link opens /oscope/edit/plotje with the
current bounded query selection; /oscope/edit/hiccup provides the companion
data-only Hiccup surface. Both editors remain functional when JavaScript is
disabled. The Plotje editor documents the supported grammar and includes
loadable bar and layered area/point/rule examples; its JavaScript only adds
debounced preview and one-click example loading. Its editable chart contains a
bounded :telemetry-query with grouping, equality filters, and named
count/sum/average/min/max/percentile series instead of copying returned
telemetry rows into the spec, so it can be reused as values change.
Reusable expressions can group those results into Unix-epoch-aligned :1m,
:5m, :15m, or :1h buckets (or :none) and expose the bucket start as a
named chart column.
See Reusable Plotje charts for the current contract and the
bounded aggregate/query grammar.
The reusable grammar also supports bounded server-side :add, :subtract,
:multiply, and :divide calculations over named aggregate series, with
explicit null and division-by-zero behavior. Per-series equality filters allow
ratios such as error count divided by total request count while global filters
continue to constrain every series.
Plotje compile and execution can also emit an opt-in, privacy-shaped semantic
lifecycle to a caller-owned sink for offline Hegel validation. The bounded
events contain closed grammar metadata and outcomes, never filter values, SQL,
result rows, credentials, user aliases, or exception details; see
Reusable Plotje charts.
Override the port or database without shell-specific export syntax:
env JOLT_CHDB_LIB=/path/to/libchdb.so \
OSCOPE_PORT=14318 \
OSCOPE_CHDB_SPEC=chdb:/absolute/path/to/oscope-data \
jolt -M:serverThe standalone server also accepts a versioned EDN file and command-line overrides. See Configuration. Validate the resolved, redacted configuration without opening storage or a listener:
jolt -M:server --config config/oscope.example.edn --check-configWithout --config or OSCOPE_CONFIG, Oscope optionally discovers the existing
platform user config (for example $XDG_CONFIG_HOME/oscope/config.edn or
$HOME/.config/oscope/config.edn on Linux). It never creates that file and
never loads a working-directory config implicitly.
The library also provides a revisioned managed-config store for future settings
and first-run interfaces. It may write only the canonical user path, never a
file selected through --config or OSCOPE_CONFIG. Its private atomic adapter
is currently qualified on Linux x86-64; macOS and Windows writes fail closed.
The standalone launcher itself remains read-only and does not invoke this API.
OSCOPE_HOST is accepted only as 127.0.0.1; jolt-http's current transport
bind is intentionally loopback-only. The receiver accepts uncompressed
application/json, caps the consumed request body at 1 MiB, and admits one
OTLP export at a time because every signal shares the embedded connection.
The standalone HTTP dispatcher uses two handler workers and admits at most
eight additional waiting tasks by default. Override those bounds with positive
integers in OSCOPE_HTTP_WORKERS and OSCOPE_HTTP_QUEUE_CAPACITY. Once both
the workers and waiting capacity are occupied, the next connection is rejected
and closed promptly before its Ring handler runs; it does not receive a 503
response. OTLP, Plotje, and export-specific admission limits still apply after
this connection-level bound.
Oscope enforces this limit with a small admission wrapper around its owned
ThreadPoolExecutor. At Oscope's Jolt 0.8.6 runtime floor, the modeled
ArrayBlockingQueue constructor still treats its capacity as advisory and
otherwise queues tasks without a bound. The wrapper rejects before submission,
while the ordinary shutdown path first stops jolt-http ingress and drains
admitted work, then terminates the owned pool.
The standalone server's default chdb:./oscope-data setting provides
local path persistence. Oscope closes its embedded connection during orderly
shutdown, and the real-chDB integration test
standalone-persistent-restart-retains-telemetry
proves that telemetry remains queryable after the server reopens the same local
directory. This is a useful local restart guarantee, but it depends on that
directory remaining available on the same filesystem.
Local path persistence does not provide object-storage recovery, cross-machine
restore, a portable checkpoint plus write-ahead log (WAL), single-writer
leases or fencing, checkpoint/WAL integrity verification, or an acknowledged
remote durability boundary. The current exporter shutdown and connection
close are not chDB Durable close() operations.
| Mode | Oscope configuration | Status and recovery boundary |
|---|---|---|
| Ephemeral/in-memory | OSCOPE_CHDB_SPEC=chdb::memory: |
Available now. Process-local state is not expected to survive close or restart. |
| Local path persistence | OSCOPE_CHDB_SPEC=chdb:/absolute/path/to/oscope-data; the standalone default is chdb:./oscope-data |
Available now. Reopening the same local directory retains the database, as covered by the restart integration test. It is not object-backed Durable recovery. |
| Local Durable development mode | jolt -M:durable-server-dev --config config/oscope-durable.example.edn, or the existing OSCOPE_DURABLE_ROOT alias |
Available with a qualified chDB Durable V1 native library. Uses the POSIX object backend, fenced single-writer lease, acknowledged WAL flushes, checkpoints, and recovery through the published head. |
| Object-backed Durable mode | A :durable-s3 file section, or OSCOPE_DURABLE_BACKEND=s3 plus the S3 settings below |
Available with the same qualified native library through the Jolt-native libcurl/SigV4 backend. Recovery uses the last CAS-published Durable checkpoint and WAL chain. |
| In-process Durable SDK | oscope.embedded/start! with a Durable writer dbspec |
Available with the same qualified native library. Instrumented application code exports spans, logs, and metrics directly into the shared writer and queries them through oscope without OTLP or HTTP framing. |
These modes reserve “Durable” for the official chDB Durable overview and Durable V1 protocol:
execute()runs one mutation locally and appends it to the in-memory WAL buffer; its success is not yet a durability acknowledgement.flush()uploads a WAL segment and commits its reference tohead.json; an application promising successful-request recovery on another machine must wait for this operation to complete before returning success.checkpoint()publishes a full database backup as the new recovery base and clears its covered WAL references only after the manifest update succeeds.open()restores the published base and replays its WAL segments in order.close()stops new operations, drains queued work, flushes, releases the writer lease, and cleans up local resources.
The ordinary -M:server path does not establish that remote durability
promise. The -M:durable-server-dev path does: it does not return a successful
OTLP response until the request's statement WAL is flushed and published.
Arrow and Parquet remain bounded data exports, not database checkpoints, WAL
segments, or a database recovery mechanism.
This process does not initialize an OTel SDK and does not wrap its HTTP routes with tracing or logging middleware. Viewer, health, export, and receiver traffic therefore cannot feed telemetry back into the collector. Shutdown is retry-safe and ordered: stop ingress, retire the oscope source, close the span, log, and metric exporter faces, then close the shared connection.
With the receiver running, use a second terminal to run the companion Jolt application:
cd oscope
jolt -M:emit-sampleEach invocation emits a new five-span checkout trace with correct parentage,
three trace-correlated logs, and one counter, gauge, and histogram sample. It
uses the public otel.sdk, tracing, logging, and metrics APIs rather than
posting fixture JSON. Re-run it to exercise live updates, then open
http://127.0.0.1:4318/oscope.
The standard OTel endpoint variable works in fish and POSIX shells, or the base URL may be supplied as the only argument:
env OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:14318 jolt -M:emit-sample
jolt -M:emit-sample http://127.0.0.1:14318The emitter checks /healthz before creating telemetry and exits visibly when
the local receiver is unavailable. Its own HTTP export calls are not woven with
instrumentation, so running it cannot create a collector feedback loop. New
traces and correlated logs appear in the workbench on its bounded two-second
poll; hidden tabs stop polling. Trace links and detail pages still work when
JavaScript is disabled.
The raw event explorer uses the same static-first pattern. Log queries can select service, case-insensitive body text, severity, time window, and a limit up to 100. Metric queries select service, case-insensitive metric name, gauge, sum, or histogram kind, window, and limit. These fields are parameters or closed choices; they cannot supply SQL, table names, or expressions.
The Playwright storyboard drives a fixed checkout OTLP fixture through the real receiver, trace/event viewers, metric distribution, and Plotje editor. Its assertions generate four screenshots and an optional GIF, so the documentation path exercises the same behavior as the browser regression test.
During cross-repository development, the standalone collector can use the real
Durable V1 POSIX backend instead of opening oscope-data directly:
env JOLT_CHDB_LIB=/path/to/chdb-26.7.2-or-newer/libchdb.so \
jolt -M:durable-server-dev --config config/oscope-durable.example.ednValidate the resolved file without opening a Durable backend, reading any credential reference, or binding a listener:
jolt -M:durable-server-dev \
--config config/oscope-durable.example.edn --check-configThe existing OSCOPE_DURABLE_* variables remain compatible. When present,
their Durable storage selection replaces the complete file storage section;
the ordinary command-line server overrides remain strongest.
The Durable root contains immutable checkpoint/WAL objects and the CAS-protected
head. Native chDB recovery uses a private scratch directory and never opens that
root as a database directory. The default owner is oscope; every process start
gets a UUIDv4 instance identity. Startup checkpoints schema migrations before
the listener is bound. Each successful OTLP export flushes its statement WAL
before the server returns 2xx. Every 1,000 acknowledged batches, the standalone
collector checkpoints the full database instead, replacing the preceding WAL
chain; set OSCOPE_DURABLE_CHECKPOINT_EVERY_BATCHES to another positive count.
This bounds the number of WAL objects between checkpoints, not the byte size of
the database or checkpoint. Durable admission spans export through persistence,
so a concurrent request receives 429 and cannot cross an earlier request's
persistence boundary; a failed durability boundary returns 503. Both failures
close the HTTP connection because the rejected streaming body may be unread. A
clean shutdown flushes WAL and releases the lease. After an unclean stop, a new
instance waits for the 30-second lease to expire. OSCOPE_DURABLE_FORCE=true
is an explicit operator takeover and should only be used after proving the old
process is gone.
For S3-compatible storage, select the Jolt-native libcurl SigV4 backend. The bucket must already exist; oscope creates only keys below the configured prefix and stable object identity:
env JOLT_CHDB_LIB=/path/to/chdb-26.7.2-or-newer/libchdb.so \
OSCOPE_DURABLE_BACKEND=s3 \
OSCOPE_DURABLE_OBJECT_ID=telemetry-prod \
OSCOPE_DURABLE_S3_ENDPOINT=https://s3.example.com \
OSCOPE_DURABLE_S3_BUCKET=observability \
OSCOPE_DURABLE_S3_PREFIX=team-blue \
OSCOPE_DURABLE_S3_REGION=us-west-2 \
OSCOPE_DURABLE_S3_ACCESS_KEY="$AWS_ACCESS_KEY_ID" \
OSCOPE_DURABLE_S3_SECRET_KEY="$AWS_SECRET_ACCESS_KEY" \
OSCOPE_DURABLE_S3_SESSION_TOKEN="$AWS_SESSION_TOKEN" \
jolt -M:durable-server-devOSCOPE_DURABLE_OBJECT_ID defaults to oscope and is the stable logical
database identity; it is deliberately separate from the per-process UUIDv4
lease instance. Optional transport controls are
OSCOPE_DURABLE_S3_MAX_ATTEMPTS (default 3, maximum 8),
OSCOPE_DURABLE_S3_CONNECT_TIMEOUT_MS (default 10000), and
OSCOPE_DURABLE_S3_TIMEOUT_MS (default 300000). Its retry budget is configured
separately with OSCOPE_DURABLE_S3_RETRY_DEADLINE_MS (default 300000),
OSCOPE_DURABLE_S3_RETRY_INITIAL_BACKOFF_MS (default 25), and
OSCOPE_DURABLE_S3_RETRY_MAX_BACKOFF_MS (default 1000). File configuration
contains environment-variable references rather than credential values. The
values are resolved only after check-only handling, passed only to the backend,
and are not copied into bounded operator diagnostics.
Startup and terminal failures print a bounded operator category before exiting
nonzero. Recognized categories include lease-held, lease-fenced,
storage-timeout, commit-ambiguous, corrupt-head, engine-incompatible,
and bounded object-store authentication, permission, throttling, transport,
provider, response, and configuration failures; each includes a fixed recovery
action. Exception messages,
credentials, backend paths, owner names, and instance IDs are not copied into
this diagnostic.
The qualified process-crash gate builds the standalone Durable server, crosses
two HTTP 200/flush boundaries separated by SIGKILL and normal lease expiry,
then verifies both traces through a separate read-only executable:
env JOLT_CHDB_LIB=/path/to/qualified/libchdb.so \
test/durable_crash_reopen.shThe hosted S3 variant starts the pinned MinIO image, uses the ordinary exact dependency pins, kills two distinct server processes after their HTTP 200/flush boundaries, and verifies exactly one copy of each trace through a fresh S3 reader:
env JOLT_CHDB_LIB=/path/to/qualified/libchdb.so \
OSCOPE_DURABLE_CRASH_BUILD_ALIAS=test-durable-s3 \
test/durable_s3_crash_reopen.shThe manual durable-aws-recovery workflow performs that same crash/reopen
acceptance against a pre-provisioned AWS bucket using GitHub OIDC temporary
credentials. Configure the aws-durable-ci GitHub environment with the
non-secret variables AWS_DURABLE_ROLE_ARN, AWS_DURABLE_BUCKET, and
AWS_DURABLE_REGION. Each workflow attempt uses the isolated prefix
ci/oscope/<run-id>-<run-attempt>, passes the AWS session token to the SigV4
transport, and neither creates nor deletes the bucket. The role needs only
s3:GetObject and s3:PutObject on
arn:aws:s3:::BUCKET/ci/oscope/*; expire the shared ci/ subtree with a bucket
lifecycle rule.
Optional settings are OSCOPE_DURABLE_OWNER, OSCOPE_DURABLE_INSTANCE,
OSCOPE_DURABLE_DATABASE, OSCOPE_DURABLE_SCRATCH_PARENT,
OSCOPE_DURABLE_LEASE_TTL_MS, OSCOPE_DURABLE_HEARTBEAT_INTERVAL_MS, and
OSCOPE_DURABLE_CLOCK_SKEW_MS. Writer control retries use
OSCOPE_DURABLE_MAX_ATTEMPTS (default 4),
OSCOPE_DURABLE_RETRY_DEADLINE_MS (default 5000),
OSCOPE_DURABLE_RETRY_INITIAL_BACKOFF_MS (default 10), and
OSCOPE_DURABLE_RETRY_MAX_BACKOFF_MS (default 250). These are distinct from
the S3 transport settings above. Invalid settings fail before the backend root
is created. The server remains loopback-only.
This development alias intentionally uses sibling-local library roots. The ordinary dependency is pinned to the reviewed control-plane commit. Both paths require a chDB release exporting the Durable V1 ABI; the current stable 26.7.0 bundled pin is insufficient.
Render a deterministic, self-contained HTML snapshot:
cd oscope
jolt -M:web-snapshot > oscope.htmlFor a live database snapshot:
export JOLT_CHDB_LIB=/path/to/libchdb.so
export OSCOPE_CHDB_SPEC=chdb:/absolute/path/to/telemetry
jolt -M:live-web > oscope.htmlThe functional web UI is an embeddable Ring adapter rather than a second HTTP stack. Mount it in an existing collector or application:
(require '[oscope.live :as oscope]
'[oscope.ui.web :as oscope-web])
(def source (oscope/open! {:db-spec "chdb:/absolute/path/to/telemetry"}))
(def oscope-handler
(oscope-web/handler
source
{:path "/admin/telemetry"
:visualization-editor-path "/admin/telemetry/edit/plotje"}))
;; Compose normally; the adapter returns nil for routes it does not own.
(defn app [request]
(or (oscope-handler request)
{:status 404 :headers {} :body "not found"}))
;; At application shutdown:
(oscope/close! source)The visualization editor is independently mountable. Its Plotje page queries the selected canonical screen once to seed a versioned edit document; preview and fallback POSTs only parse and render the submitted bounded document:
(require '[oscope.ui.visualization-editor :as editor])
(def editor-handler
(editor/handler
source
{:path "/admin/telemetry/edit"
:viewer-path "/admin/telemetry"
;; An instrumented host supplies its generic context wrapper here.
:run-suppressed (fn [thunk] (thunk))}))Compose editor-handler like any other Ring adapter; it returns nil for
paths it does not own. :run-suppressed receives a zero-argument thunk for
every owned editor route. The standalone collector needs no wrapper because it
does not initialize an SDK or install telemetry middleware. An instrumented
host should provide the same generic suppression context it uses for its
viewer and telemetry-storage routes. Encoded editor bodies are capped at
40,000 bytes; Plotje source is capped at 32,768 characters and safe-Hiccup at
16,384 characters before any document can reach a renderer.
The configured mount path drives both Ring routing and form navigation. Every control is an ordinary GET form control, so querying, charts, tables, and navigation work with JavaScript disabled. The page ships a restrictive CSP and no script element; a host application can add progressive enhancement outside the adapter contract.
A live page includes an ordinary no-JavaScript export form. Its download route
is derived from the mount path: /oscope/export by default and, for the example
above, /admin/telemetry/export. The form requires an absolute half-open
[start-unix-nano, end-unix-nano) window no longer than 24 hours, explicitly
selects gauge, sum, or histogram for metrics, and can only lower these hard
limits:
- 100,000 physical rows;
- 64 MiB of encoded output; and
- Arrow file or Parquet output.
The row limit truncates the ordered physical-row selection; it is a bound, not a pagination cursor. These are result bounds, not execution-cost, rows-read, or wall-time bounds: ClickHouse may scan and sort more physical rows before it produces the bounded result. Hosts that expose large or untrusted datasets should also configure engine-side resource limits.
One live source admits one export at a time by default. Its permit remains held
until jolt-http finishes writing the response body or the write fails, so slow
clients cannot accumulate multiple maximum-sized byte arrays after native
queries complete. Additional requests receive 503 with Retry-After; an
embedding application may choose a small capacity up to 16 with
:export-capacity, accounting for the corresponding memory exposure.
Oscope supplies the MIME type and a filename made only
from closed source names, epoch integers, and the selected extension. It never
writes a server-side export path. The returned byte array is copied by
jolt-chdb before the native query result is destroyed, so the Ring response
does not retain a libclickhouse buffer.
The default adapter rejects browser requests marked Sec-Fetch-Site: cross-site. A host may supply :authorize-export? for stronger policy. The
standalone composition is loopback-only; any future remote exposure must add
an authenticating reverse proxy or an equivalent host authorization hook.
The deterministic sample page renders the same controls disabled and its export route returns 404. It never manufactures a data file when no live exporter exists.
GTK4 and a working display/WSLg are required.
# Deterministic Glitter UI
jolt -M:native
# Live Glitter UI
env JOLT_CHDB_LIB=/path/to/libchdb.so \
OSCOPE_CHDB_SPEC=chdb:/absolute/path/to/telemetry \
jolt -M:live-native
# Standalone OTLP/HTTP receiver plus the native viewer, sharing one connection
env JOLT_CHDB_LIB=/path/to/libchdb.so \
OSCOPE_CHDB_SPEC=chdb:/absolute/path/to/telemetry \
jolt -M:native-server
# Alternative Glimmer/GTK adapter
jolt -M:glimmer-native-M:live-glimmer-native is the live Glimmer equivalent. The shared model is
compatible with Glimmer plus glimmer-uikit on macOS, but this Linux checkout
cannot execute the AppKit backend. Glitter is the default GTK architecture;
Glimmer remains useful for components that benefit from local reactive state.
The native server receives OTLP/HTTP on 127.0.0.1:4318 and also retains the
web viewer at http://127.0.0.1:4318/oscope. Its Glitter window renders the
canonical screen[:chart] through oscope's bounded Plotje-to-SVG renderer and
GtkPicture; the accessible distribution remains below the chart. Each window
owns and removes its temporary SVG files. For an opt-in WSLg smoke that closes
itself, set OSCOPE_NATIVE_AUTO_QUIT_MS to a positive millisecond count.
Current Glitter and Glimmer application runners own their mounted root and do
not return a complete unmount handle. Each oscope adapter isolates its model
and callbacks per instance. Selection callbacks enqueue into a capacity-one
latest-wins path and return without running the query: one owned OS worker runs
the blocking/lazy load, and generation fencing discards a completion when a
newer selection is current. The screen exposes :query-state as :loading,
:ready, or bounded :error; an error retains the prior display as :stale.
The adapter's close! rejects future selections and waits for that physical
worker to exit before returning, so close the shared source only afterward.
The toolkit retains the mounted root and callbacks until window teardown.
A future runner API returning that root can add explicit unmount and release
without changing the oscope contract. Applications that require independently
owned embedded native windows should treat that runner enhancement as a gate.
oscope.embedded packages the direct in-process path behind one lifecycle:
Applications that do not need Oscope's listener or UI dependencies can use the
same repository with :deps/root "profiles/embedded". The profile keeps
oscope.embedded as the only lifecycle implementation and adds only the SDK,
Durable/exporter, typed-schema, and bounded-query dependencies. See
Minimal embedded dependency profile for the
coordinate, exact dependency audit, fixture, and current Samizdat provider
qualification boundary.
(require '[jdbc.chdb.durable :as durable]
'[oscope.embedded :as embedded]
'[oscope.embedded.viewer :as viewer]
'[otel.sdk :as sdk]
'[otel.trace :as trace])
(def runtime
(embedded/start!
{:db-spec (durable/writer-dbspec
{:backend durable-object-backend
:owner "checkout"
:database "default"})
:sdk-options {:service-name "checkout"
:processor :batch
:metrics? true
:logs? true}}))
(trace/with-span [_ (sdk/tracer "checkout.http") "POST /checkout"]
(handle-checkout))
;; Optional: expose the normal UI without installing any OTLP receiver routes.
(def viewer-lifecycle (viewer/start! runtime {:port 0}))
(:url viewer-lifecycle) ; open the actual loopback URL selected for port 0
;; Stop application ingress and the borrowed viewer first, then drain and
;; retire its source, SDK, exporter, connection, and Durable writer owner.
(viewer/stop! viewer-lifecycle)
(embedded/stop! runtime)The viewer-only lifecycle accepts only 127.0.0.1, validates each request's
exact numeric Host authority, and returns 404 for every OTLP ingestion path.
It borrows the embedded runtime's source and connection, and owns only its HTTP
listener and bounded request workers. Its stop operation is idempotent and
retries an incomplete listener or worker boundary. Always stop the viewer
before embedded/stop! so an admitted query cannot race source retirement.
An embedded runtime can also send the same completed spans to a remote OTLP/HTTP JSON collector while retaining spans, logs, and metrics locally. Each span destination has its own bounded queue and worker, so a slow or unavailable remote collector cannot block local ingestion:
Each span worker exports a complete batch without waiting out its schedule delay, checking readiness within its existing short wait slices (not a hard real-time deadline). Batch size must not exceed queue capacity, including when using a custom small queue. Invalid combinations are rejected before opening the database or constructing the remote exporter.
(def runtime
(embedded/start!
{:db-spec durable-writer-dbspec
:sdk-options {:service-name "checkout"
:metrics? true
:logs? true}
:span-pipelines
{:local {:max-queue-size 4096
:max-export-batch-size 512
:schedule-delay-ms 1000}
:remote {:endpoint "https://collector.example.com:4318"
;; The variable contains comma-separated OTLP key=value headers.
;; Header values are never accepted inline here.
:headers-env "CHECKOUT_OTLP_HEADERS"
:timeout-ms 10000
:max-retries 3
:insecure? false
:max-queue-size 2048
:max-export-batch-size 512
:schedule-delay-ms 1000}}}))
(embedded/span-pipeline-stats runtime)
;; => {:local {:queue-size 0
;; :attempted-span-count 12 :exported-span-count 12
;; :failed-span-count 0 :dropped-count 0}
;; :remote {:queue-size 0
;; :attempted-span-count 12 :exported-span-count 12
;; :failed-span-count 0 :dropped-count 0}}
(embedded/status runtime)
;; => {:oscope.embedded.status/version 1
;; :phase :open
;; :span-pipelines
;; {:mode :independent
;; :local {:availability :available, :queue-size 0, ...}
;; :remote {:availability :available, :queue-size 0, ...}}
;; :durable {:view-current? :unavailable
;; :last-successful-persistence :unavailable}}
(embedded/force-flush! runtime)
;; => {:sdk {:ok? true}
;; :span-pipelines
;; {:local {:ok? true} :remote {:ok? true}}}Use either remote :endpoint (Oscope appends /v1/traces) or the full
:traces-url, never both. URLs with userinfo, query strings, or fragments are
rejected so credentials stay out of transport diagnostics. :headers-env
names an environment variable; its
resolved header values are retained only inside the private remote exporter,
and neither the variable name nor values appear in the returned lifecycle or
operation results. Standard OTEL_EXPORTER_OTLP_* variables do not override or
supplement this closed embedded destination. Inline :headers, custom SDK
:exporter, and custom SDK :span-processors are rejected before Oscope opens
JDBC or starts a batch worker.
For the exact Langfuse endpoint/header profile, the standalone two-exporter topology, and the opt-in semantic readback gate, see Langfuse span export.
Delivery counts reported by span-pipeline-stats cover each processor's
lifetime. Consequently, a failed background export remains visible to later
force-flush! and stop! calls even when the exporter's own flush callback
succeeds. force-flush! and stop! report a safe overall SDK marker beside
closed per-destination outcomes. This keeps a log or metric lifecycle failure
distinct from a local span-pipeline failure. False returns and throws are visible
only as {:ok? false :failure :returned-false} or :threw. Terminal SDK and pipeline
shutdown runs exactly once, so a telemetry failure is retained rather than
pretended retryable; source retirement, the final Durable checkpoint or flush,
and connection close still run. Only a failed query, persistence, or connection
boundary leaves the lifecycle :closing for a later stop! retry. Logs and
metrics are intentionally local-only in this slice. This is generic OTLP/HTTP
export; interoperability with a particular hosted product is not implied.
embedded/status is the safe surface for a read-only application health
endpoint. It returns a closed versioned map containing only lifecycle phase and
bounded scalar counters for the local and remote span queues. Direct-local
lifecycles and malformed or unavailable counter sources use explicit
:unavailable values. Durable view freshness and the last successful
persistence boundary are also :unavailable: the current public jolt-chdb API
does not expose evidence for those claims, and Oscope does not infer them from
an open connection. The snapshot never contains the dbspec, backend,
connection, exporter, schema descriptors, endpoint, credentials, exceptions,
or telemetry attribute values.
Installer-confirmed log-record attributes are also available through bounded, schema-bound typed-log filters in the embedded and standalone viewer. The catalog and result surface retain only logical field identity, type, location, manifest version, projected scalar values, and conserved coverage counts. String, Boolean, and Int64 use the operators advertised by the exporter. Historical rows are distinguished as valid, empty, absent, invalid, fallback-readable, or unavailable; generic log exploration remains the fallback when no confirmed typed-log capability is present. Saved typed-log URLs carry the complete binding and fail closed if the active manifest changes. Log bodies and non-selected attribute values never enter the typed result UI.
To promote reviewed span attributes into typed ClickHouse columns, compile the approved manifest before startup and pass it with the registry backend. Oscope binds schema observation and additive DDL to its own connection; application telemetry never receives either capability:
(require '[otel.exporter.chdb.attribute-manifest :as attribute-manifest])
(def approved
(attribute-manifest/compile-manifest
{:dataset-id "telemetry-prod"
:application-id "checkout"
:lineage "checkout-v1"
:version 1
:fragments
[{:schema attribute-manifest/reviewed-fragment-schema
:authority :advice
:source "advice/checkout.edn"
:entries [{:signal :spans
:table "otel_traces"
:location :resource-attributes
:key "deployment.environment"
:type :string}
{:signal :spans
:table "otel_traces"
:location :scope-attributes
:key "instrumentation.flavor"
:type :string}
{:signal :spans
:table "otel_traces"
:location :span-attributes
:key "checkout.complete"
:type :boolean}]}]}))
(def runtime
(embedded/start!
{:db-spec (durable/writer-dbspec
{:backend durable-object-backend
:owner "checkout"
:database "default"})
:typed-schema {:approved-manifest approved
;; Reuse the same object-scoped backend in Durable mode.
:registry-backend durable-object-backend}}))oscope.server/start! accepts the same :typed-schema map for a programmatic
standalone receiver. Startup fails closed unless the registry record is active
and its physical columns were freshly observed. Only the opaque confirmed
descriptor capability reaches the exporter and returned live source. The
ordinary local server has no Durable checkpoint promise; when its existing
:durability callbacks are supplied, base and typed schema changes are
checkpointed before HTTP ingress starts. With no :typed-schema map, exporter
options remain unchanged and the exporter retains its default base-schema
ownership. Version 2 file configuration validates a digest-pinned approved
manifest or read-only registry selector with --check-config. Ordinary
local-path startup reopens a persistent database-scoped registry and passes the
authorized install or acquire mode through this same boundary. A confirmed
capability also adds table-only Boolean, signed Int64, and string filters for
resource, instrumentation-scope, and span attributes to the web UI. The same
logical key at different locations remains three visibly distinct fields and
is always sent to exporter-owned queries with its confirmed location. Int64
values support exact eq, gte, and lt predicates and
remain decimal text in the browser until Oscope performs bounded parsing, so
values beyond JavaScript's safe-number range remain exact. Approved Int64
fields also have a table-first summary with an inclusive lower bound, optional
exclusive upper bound, optional service grouping, and selectable count,
min, max, and avg columns.
Those controls use logical attribute names, locations, and exact
manifest-version bindings;
successful field-selection forms redirect to a canonical URL containing the
complete binding before executing the query. Partial bindings, nonpositive or
overflowing signed-Int64 manifest versions, and stale saved URLs fail visibly,
while physical columns and registry ownership
identity remain exporter-private. Filter coverage and matches, or aggregate
rows and aggregate coverage, are two bounded live queries, so the UI calls out
that concurrent ingestion can advance one between them. Both views list valid,
empty, absent, invalid, historical fallback, and historical unavailable rows
plus the conserved total. Boolean results preserve false as a typed value;
only valid typed values enter numeric aggregates, and generic historical text
is never parsed as a number. Resource and span history can use their matching
generic ClickStack maps when present. ClickStack stores no generic scope
attribute map, so pre-promotion scope values are reported as historical
untyped-unavailable rather than guessed or silently omitted. Legacy saved URLs
without a location redirect to the five-part form only when they still identify
one span field; resource, scope, ambiguous, and stale legacy bindings fail
closed before SQL execution.
The opt-in typed query and storage benchmark drives this real standalone OTLP/chDB path against an otherwise identical string-fallback database. It reconciles Boolean and Int64 filters, Int64 aggregates, coverage, restart bindings, and a narrowly scoped active-part footprint before reporting bounded latency summaries. Its smoke profile checks the harness; the larger profile is manual performance evidence, not a CI threshold or a cold-cache claim. A one-process run is explicitly unbalanced; an optional even repetition count launches fresh processes with alternating typed/fallback order and rejects incomplete or stale repetition sets.
The versioned Durable launcher places the typed registry in a fixed object scope distinct from the telemetry object. Typed promotion is never inferred from telemetry.
The returned :source is the ordinary live oscope query source and can be
given to the web or native UI handlers. The exporter owns schema migration,
checkpoints it before startup returns, and confirms a Durable flush after every
non-empty SDK batch. Shutdown drains the SDK, retires queries, checkpoints the
committed WAL by default, and finally closes the writer. A failed boundary is
reported as {:status :closing ...} and may be retried with stop!.
This path avoids OTLP encoding, HTTP framing, and receiver decoding. The current
chDB insert path still materializes JSONEachRow SQL internally; protobuf or
gRPC would not remove that in-process storage encoding. Export acknowledgement
is at-least-once: an ambiguous failed attempt followed by an SDK retry can
produce a duplicate.
Embedded render loops and other UI fibers must not call the source's JDBC-backed
load commands directly. Use oscope.embedded.query to move that blocking work
onto one owned OS thread and publish a small immutable cache:
(require '[oscope.embedded.query :as embedded-query])
(def sampler
(embedded-query/start!
{:load! #(get-in ((:load-command (:source runtime))
:recent-spans
{:signal :spans :field :span-name
:window :15m :limit 20})
[:table :rows])
:interval-ms 1000
:timeout-ms 2000
:max-rows 20}))
;; Rendering only dereferences the cached value; it never queries chDB.
(def display-model (embedded-query/snapshot sampler))
;; Retry a bounded stop until the native query has really completed and joined.
(loop []
(when-not (= :closed (:status (embedded-query/stop! sampler)))
(Thread/sleep 50)
(recur)))
;; Only now may the shared source be retired.
(embedded/stop! runtime)The load function must use a backend-bounded selection; :max-rows separately
bounds retained display data. A timeout publishes :error before the first
good sample or :stale while retaining the last good rows. It does not interrupt
native JDBC work. The cadence fiber notices stop without waiting for the query
timeout. Result, stop, interval, and timeout events wake it through Jolt channel
selection rather than periodic result polling. The owned query thread is joined
rather than interrupted. If a native call is still running, stop! returns
:stopping/:joining-query after the configured stop bound; retry it and keep
the source open until it returns :closed. Total shutdown latency can therefore
include the native operation's actual completion time.
Query failures use a closed public descriptor:
;; :load! threw
{:type :query-failure :phase :load :category :load-failed}
;; executor submission threw
{:type :query-failure :phase :submission
:category :executor-submission-failed}
;; the validated configured timeout n elapsed
{:type :query-failure :phase :load :category :timeout :timeout-ms n}A later successful sample removes the failure; a failure after success
publishes :stale with the last good rows.
These descriptors never retain exception class or message text, causes,
ex-data, stacks, SQL, parameter or result values, database/backend/object
paths, endpoints, or credentials. A character limit bounds retention size but
does not redact a sensitive prefix, so arbitrary exception strings are not
included at all.
If the owned query executor throws while stop! shuts it down or waits for
termination, the call returns the same closed lifecycle failure shape used by
the other embedded owners:
{:status :stopping
:phase :joining-query
:errors [{:type :oscope.error/lifecycle-operation-threw
:operation :stop-query-workers}]}The result contains no Throwable class or message, cause, ex-data, stack,
query value, path, endpoint, or credential. It remains safe to retry; :closed
is still reported only after the executor really terminates, and a blocked
native query is never interrupted.
An out-of-process viewer opens a read-only Durable connection after a published flush or checkpoint. It restores the immutable head selected at open time into private scratch storage; it does not need the writer's local state and does not tail uncommitted changes.
An in-process OTLP collector should share its existing connection:
(def source (oscope/open! {:connection collector-connection}))oscope/close! retires queries but never closes a caller-owned connection. If
:db-spec is supplied instead, oscope opens and closes the connection itself.
Both paths run the existing exporter schema check by default; oscope does not
fork or own a competing schema. The standalone composition lets the exporter
perform the sole schema migration and opens the shared oscope source with its
redundant check disabled.
An embedded chDB path should normally have one process-level owner. Coordinate ingest and query access through that owner rather than opening the same physical database independently in multiple processes.
The dependency direction is deliberately one-way:
selection -> versioned command -> effect -> bounded query -> screen -> views
|
raw selection -> export command -> closed SQL -> query-bytes -> Ring download
|
screen chart -> versioned visualization document -> safe preview renderer
oscope.queryvalidates selection and builds exact SQL-free plans.oscope.telemetryowns the bounded parameterized trace list, detail, correlated-log, filter-option, span-tree, raw-log, and metric-point contracts.oscope.raw-exportvalidates absolute windows, source and format choices, caps, generated parameterized SQL, the complete owned-byte result envelope, MIME type, and suggested filename.oscope.commandis the versioned, portable intent envelope.oscope.effectbuilds and validates a complete screen before one atomic replacement; renderers never mutate individual result fields.oscope.view-modelis serializable EDN with semantic controls, exact plan provenance, a Plotje-compatible chart, and accessible table rows.oscope.plotje.specandoscope.plotje.svgare the bounded portable chart dependency. The supported subset currently includes line, point, bar, area, rule, and tick marks plus explicit safe style options; it is not the full Plotje API.oscope.visualization.documentowns the closed Plotje/Hiccup edit envelope;oscope.hiccup.specrejects active tags, URLs, and event attributes; andoscope.ui.visualization-editoris the mountable Ring surface.oscope.liveis the only owned/shared chDB lifecycle boundary and owns the source-wide export admission state.oscope.ui.workbenchandoscope.ui.webare sibling mountable Ring surfaces over the same connection: detailed traces/logs and aggregate logs/metrics/charts respectively. A host may supply:advise-traceto the workbench so library-specific Kindly metadata is applied only at render time.oscope.ui.eventsis the third sibling surface: bounded raw log and metric rows with progressive two-second refresh and direct correlation links back to trace detail.
The first implementation is synchronous. Before moving database queries onto GUI workers, retain monotonically increasing request IDs and reject stale completions before the whole-screen mutation.
The full deterministic suite does not open a native window or a real chDB, but it does load the Glitter/Glimmer adapter namespaces. It exercises query/model/command invariants, Plotje SVG, the Ring adapter, headless Glitter reconciliation, Glimmer/Glitter instance isolation, bounded visualization documents and editor routes, OTLP body policy, route composition, shared-connection ownership, and retry-safe shutdown:
jolt -M:testHosted CI intentionally runs the portable collector, query, export, and web layers without loading the GTK-facing Glitter/Glimmer namespaces:
jolt -M:test-headlessThe optional JVM-only Typed Clojure pilot checks a deliberately small subset of the pure command/query/view-model contracts and nine named mutation controls, including cumulative-histogram temporality provenance:
clojure -M:typed-checkThe checker and its Clojure/JVM dependencies are dev-only and are absent from all ordinary Jolt aliases. See the exact checked boundary, limitations, and mutation evidence.
That narrower gate supplements rather than replaces -M:test; native adapter,
real chDB, independent-reader, and standalone receiver coverage remain explicit
local/release gates below.
Run the real embedded database gate separately. It starts a real loopback server, ingests spans, logs, and metrics through OTLP/HTTP JSON, queries the canonical live source, renders the viewer, downloads Parquet, proves viewer traffic does not change telemetry counts, and closes the lifecycle twice:
env JOLT_CHDB_LIB=/path/to/libchdb.so \
jolt -M:test-chdbWhen clickhouse-local is installed, independently parse both encoded formats
and prove half-open boundary and row-truncation semantics with:
env JOLT_CHDB_LIB=/path/to/libchdb.so \
jolt -M:test-readersThe Durable implementation has a sibling-local integration lane. It uses the real oscope loopback server/exporter/viewer composition and real native chDB, checkpoints schema before ingress, sends one OTLP trace, log, gauge, sum, and histogram over HTTP/1.1, flushes before each 200 response, and verifies exact per-kind counts, UI querying, and Arrow/Parquet export after provider reconstruction and read-only recovery. It also proves the nondeterministic schema-migration record is covered by the checkpoint and absent from every statement WAL:
/home/chuck/ai-src/tools/jolt-with-chez-10.4.1 \
jolt -M:test-durable-devThis alias intentionally overrides jolt-chdb and jolt-otel-clickhouse with
../jolt-chdb and ../jolt-otel-clickhouse so it can test coordinated local
changes. The ordinary dependency declaration uses the reviewed exact commit
pin listed below.
When JOLT_CHDB_LIB names the qualified 26.7.2-rc.2 library, this lane runs the
real integration, including backup/restore and classification. With the stable
26.7.0 pin, the runner reports that integration as skipped and runs the
configuration, startup ordering, request acknowledgement, failure, and
shutdown controls; it never fabricates a successful checkpoint over the absent
ABI. The test uses the POSIX Durable provider and reconstructs it from its
filesystem root before read-only reopen, so recovery does not rely on the
in-memory backend oracle.
The opt-in S3 app gate starts an immutable MinIO container, launches the real oscope collector over the S3 namespace configuration above, crosses three HTTP 200/WAL flush boundaries for a trace, correlated log, and gauge/sum/histogram batch, performs a clean release, and reconstructs the backend. A fresh native connection proves exact per-table counts and valid Arrow/Parquet exports for every recovered signal shape:
env MINIO_IMAGE=your-registry/minio@sha256:YOUR_MANIFEST_DIGEST \
JOLT_CHDB_LIB=/path/to/qualified/libchdb.so \
test/durable_s3_minio.shThe durable-s3-e2e workflow runs the same script with the publishable
:test-durable-s3 alias. Its old Quay image is no longer publicly available.
CI builds the historical MinIO source commit
07c3a429bfed433e49018cb0f78a52145d4bedeb using Go 1.24.6 and read-only
module resolution, and supplies the resulting local Docker image ID to all
three S3 fixtures. That ID is content-addressed, not a registry manifest
digest; both immutable forms are accepted by the woven fixture. This minimal
image is only for loopback HTTP tests with synthetic credentials and data:
it disables the browser and update checks, and is not a supported production
or TLS deployment image. Local runs must supply an available immutable image.
The workflow qualifies the native library through jolt-chdb's
pinned asset and upstream ABI oracle, then exercises oscope only through exact
Git dependency pins. The script defaults to :test-durable-s3-dev locally so
coordinated sibling changes remain testable before publication.
The same service path can be compiled with the Durable aspect pack and checked against its Hegel transition model. This also proves the explicit libcurl transport entry point remains reachable in a standalone Jolt executable:
env JOLT_ASPECT_JOLT=/absolute/path/to/aspect-capable/jolt \
JOLT_CHDB_LIB=/path/to/qualified/libchdb.so \
test/durable_s3_aspect_smoke.shThe same integration test also has an opt-in compiled aspect lane. It binds a semantic journal around the real oscope lifecycle, weaves only the six Durable control entry seams, and validates the completed privacy-shaped command history offline with the Hegel transition model. The same build composes an oscope consumer that emits one internal span and one duration-histogram observation per Durable control operation. Its attributes are limited to the closed operation, outcome, and failure-category sets; it never records arguments, object keys, owner identities, result bodies, or exception messages. Generic OpenTelemetry suppression is checked before observation, so an exporter backed by the same Durable chDB connection cannot recursively instrument itself:
env JOLT_ASPECT_JOLT=/absolute/path/to/aspect-capable/jolt \
JOLT_CHDB_LIB=/absolute/path/to/libchdb.so \
test/durable_aspect_smoke.shThis subproject uses sibling-local roots for jolt-chdb,
jolt-otel-clickhouse, and jolt-aspect-packs; it is an integration/release
gate rather than a publishable dependency. Its compiler report at
target/aspects.edn must be checked with the pack's report validator, while the
ordinary -M:test-durable-dev lane remains the non-woven comparison. The build
explicitly embeds jolt-chdb's canonical runtime resources so the standalone
binary reads the same ABI descriptor used during compilation. The woven lane
uses a release-mode build and the same real loopback transport as the unwoven
integration.
Oscope requires Jolt v0.8.6 or newer for source execution and ordinary
standalone builds. The separately pinned aspect-capable compiler and Durable
libraries remain at their currently qualified revisions until the next
protocol/model qualification stack advances them together. Older app builders
may incorrectly inherit jolt.ffi from the compiler image and produce a binary
with an unbound jolt.ffi/errno. The standalone smoke builds and starts the
real receiver long enough to reject that artifact class:
env JOLT_CHDB_LIB=/path/to/libchdb.so \
JOLT_BIN=/path/to/jolt-v0.8.6-or-newer \
JOLT_TOOLCHAIN=/path/to/jolt-with-chez-10.4.1 \
test/standalone_build_smoke.shcasselc/otel0c50b0f8254713ce9df8a3f201f345b1854000b8- resolved OTLP transport
casselc/http-clienteab6b78d5957f88690faf6768360572a3f185341 chucklehead-dev/jolt-otel-clickhouse14a2998a27f64a9bff329811461be9157a00c849chucklehead-dev/jolt-otel-viewer5723a7c28c3bb3ae7cb27f9856b90463e77df523chucklehead-dev/jolt-chdbdbc2db22130c7e783739c79bc24691dcbba21906chucklehead-dev/jolt-aspect-packsd5f42a309ac52ccd8dfa54d256fa2fa502d1fd7ccasselc/joltf00bc93bdd8274b14087b74272aadeffb60e0447(woven Durable compiler,jolt v0.8.6-5-gf00bc93b)casselc/jolt-http35d1d7f9ebdc796ee9bd4c80745298b2c8b7fdf8casselc/glitterf4e3eb83015566e4cadaedd7f5e8ad80dc57404fcasselc/glimmer6dab5597dc0d912793fe175d0d3cbb9e75f11426jolt-lang/glimmer-gtkce79d45698d36ccf496397bb85974e3cce6abfd8casselc/data.json932444043c0c06f9e295ba4963419b2481e9dd07
The hosted headless gate inspects the resolved classpath and rejects the prior
ClickHouse exporter coordinate instead of trusting this list or deps.edn
alone. The browser lane separately runs the typed standalone integration
through both direct export and a real loopback OTLP socket; it needs no
credentials or external service. Durable CI retains the embedded local/remote
pipeline isolation gate.
The pinned ClickHouse exporter owns the required Jolt DB bootstrap at its
public explorer entrypoint. A clean oscope consumer therefore needs no hidden
load-order require and cannot accidentally compile jdbc.core before the
ResultSet compatibility model exists.
No dependency on the source demo or native spike remains. No upstream pull request is required to build or test this repository.
- add explicit export pagination or partition manifests for workflows that need more than one bounded physical-row download; and
- add a preserve-to-table command after the frozen-screen contract has a caller-owned destination/schema policy.
Copyright contributors. Distributed under the Eclipse Public License 2.0.