Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion .github/workflows/stock-span-codec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,12 @@ jobs:
'{:paths ["src" "test"] :deps {io.github.chucklehead-dev/jolt-codec {:git/url "https://github.com/chucklehead-dev/jolt-otel-clickhouse.git" :git/sha "98e93e2a6726c435d9b55ada9a088c2dbf5b1c99" :deps/root "incubator/codec"}}}' \
-m oscope.json-reader-legacy-probe
"$JOLT_BIN" -Srepro -M:test-export-options
"$JOLT_BIN" -Srepro -Sdeps '{:paths ["src" "test"]}' -e \
"(require 'clojure.test)
(binding [*assert* false]
(require 'oscope.field-slot-native-probe-test)
(let [r (clojure.test/run-tests 'oscope.field-slot-native-probe-test)]
(System/exit (if (zero? (+ (:fail r) (:error r))) 0 1))))"
"$JOLT_BIN" -Srepro -A:stock-span-codec -Sdeps '{:paths ["src" "test"]}' \
-m oscope.stock-codec-runner-test
"$JOLT_BIN" -Srepro -Sdeps '{:paths ["src" "test"]}' \
Expand All @@ -74,11 +80,16 @@ jobs:
shell: bash
run: |
set -euo pipefail
for task_consumer in embedded standalone embedded-typed; do
for task_consumer in embedded standalone; do
timeout --signal=TERM --kill-after=5s 120s "$JOLT_BIN" \
-Srepro -A:test-durable:stock-span-codec \
-m oscope.export-options-native-test-runner stock-codec "$task_consumer"
done
# Reuse the typed live-query fixture, adding a numeric witness
# for actual field-slot results AFTER startup, not construction alone.
timeout --signal=TERM --kill-after=5s 120s "$JOLT_BIN" \
-Srepro -A:test-durable:stock-span-codec \
-m oscope.field-slot-native-probe
timeout --signal=TERM --kill-after=5s 120s "$JOLT_BIN" \
-Srepro -A:test-durable:stock-json-reader \
-m oscope.export-options-native-test-runner stock-reader standalone
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,14 @@

## Unreleased

- Update the root exporter to avoid temporary maps and vectors when projecting
small typed span manifests on the selected native path. Preserve complete
captured attributes, value/status rules, fallback and Durable confirmation.
Keep encoding defaults, SDK/driver/JSON pins and the separate minimal embedded
profile unchanged. Qualify real post-startup typed projection separately from
existing file-backed capture/fresh-reader checks; this is not a combined
typed-recovery proof, application throughput or slow-tail guarantee.

- Update the root OTel SDK pin to reuse its scalar normalization fast path
and avoid lazy fixed-field classification during OTLP ingestion, reducing
temporary allocation while preserving value limits and error handling.
Expand Down
2 changes: 1 addition & 1 deletion deps.edn
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
:git/sha "649c993780a0f269938c9c6afa3bae7e9709bb48"}
io.github.chucklehead-dev/jolt-otel-clickhouse
{:git/url "https://github.com/chucklehead-dev/jolt-otel-clickhouse.git"
:git/sha "1e7cc8d0a821e589ec087774d118effcaab1d234"}
:git/sha "a6cbf9390d1521026e5da5d2ea3cfc0b8916ae2b"}
io.github.chucklehead-dev/jolt-otel-viewer
{:git/url "https://github.com/chucklehead-dev/jolt-otel-viewer.git"
:git/sha "5723a7c28c3bb3ae7cb27f9856b90463e77df523"}
Expand Down
16 changes: 16 additions & 0 deletions docs/stock-codec.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,14 @@ The selected component supports the SDK's ordinary record link contexts without
map copies. Older components without that capability reject during startup;
custom record `seq` implementations remain unsupported.

The root exporter can now collect small approved typed span projections directly
into field slots, avoiding temporary location maps and vectors. Eligible plans
have up to eight distinct fields across the three trace attribute locations;
unsupported shapes, aliases or changed policies keep the existing fallback.
Complete generic attributes and typed value/status rules are preserved. There
is no new configuration switch or default encoder change, and component savings
do not establish application throughput or slower-batch latency improvements.

Activate the root `:stock-span-codec` alias when starting Oscope. A library
consumer must instead declare the independent component in its own deps.edn;
dependency aliases do not propagate:
Expand Down Expand Up @@ -78,3 +86,11 @@ forbids the legacy native JSON writer, and checks independent recovery after
SDK or socket capture. It does not establish S3, Langfuse, application standalone
packaging, sustained throughput or population-tail performance. No throughput
target is claimed by enabling these options.

The embedded typed fixture additionally requires a positive field-slot result
after application startup, before its existing terminal receipt. Constructor
calls, startup-only activity and false/nil declines cannot satisfy that witness.
This fixture checks a live typed query against an in-memory object backend;
the separate file-backed embedded/standalone cases check independent recovery.
Those are different gates, not one combined typed-recovery proof. Its counters
are test instrumentation, not a performance measurement or delivery receipt.
114 changes: 114 additions & 0 deletions research/exporter-field-slot-pin-20261021/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# Root exporter field-slot integration

Base Oscope `2615d30e28ff9c2a58197187c72622e5a551f6d1`; update ONLY the root
exporter from `1e7cc8d0a821e589ec087774d118effcaab1d234` to merged exporter
PR157 `a6cbf9390d1521026e5da5d2ea3cfc0b8916ae2b`. Preserve the direct SDK
`3366a332`, driver `3314b79`, ordinary JSON `0f51b991`, codec aliases `7ab2bed`,
crypto/provider pins, runtime floor, storage/encoding/configuration defaults.
No application implementation, persisted format or lifecycle/model change.

The separately pinned `profiles/embedded` remains unchanged. Root
`oscope.embedded/start!` and that minimal dependency profile are NOT the same
dependency authority. Minimal-profile refresh stays in #167; stacked PR144 is
not implicitly merged or approved by this root update.

## Local checks executed

Official stock Jolt0.8.23, SHA256
`7bf7acae4b4bcb5c27b3651fa4a94d2d4e6a7fc43b5b821def2f8be3d6f00e94`,
mandatory Chez10.4.1 wrapper. Child `JOLT_BIN` and PATH explicitly select that
same executable, not installed plain Jolt0.8.10. No compiler rebuild or timing
benchmark. Native stock-codec cases explicitly use cached libchdb26.9.0.
Headless cases use each graph's declared native package, including the old
minimal profile's26.7.3 package.

| Gate | Result |
| --- | --- |
| Current-main headless baseline |361tests /3539assertions pass|
| Candidate headless |363 /3560 pass|
| New witness controls |1 /18 pass|
| Witness controls with assertions disabled |1 /18 pass|
| Actual typed SDK capture/live query |1 /8 pass;1 post-startup field-slot result|
| File-backed embedded SDK capture + fresh reader1 |1 /18 pass;reader terminal/settled/valid|
| Standalone HTTP capture + fresh reader0 |1 /45 pass;reader terminal/settled/valid|
| Native HTTP JSON reads + fresh reader0 |1 /45 pass;3 accepted /0 declined document reads|

The extra headless assertions are18 witness controls and3 exact prior-exporter
resolution controls. Existing resolution tests confirm root SDK3366a332 still
wins over exporter transitive SDK, with the expected HTTP/provider selections.
The root graph is not silently upgraded to exporter-own SDK6cd57c86.

Real red/green:before changing deps.edn, the new native probe with the actual
prior exporter rejected with the fixed unavailable-field-slots error, before
application storage acquisition. After the pin update, the same probe observes
one real selected result during capture and the existing typed live query passes.
Startup/constructor-only calls, missing projectors and false/nil declines are
rejected by pure controls. Positive output is numeric only; no spans, credentials
or persisted data are logged by the observer. It is not a timing instrument.

## Important coverage boundary

Native fixture3 installs an approved Boolean manifest, performs actual SDK
capture and checks typed value/status via the live source. It uses an in-memory
object backend. It DOES NOT independently recover that typed capture.
Fixtures4/2 separately use file-backed stores and independently joined fresh
reader processes; they verify other multi-signal content. Do not combine these
successes into a claim of typed physical-field recovery. That existing
application-level composition gap is tracked in
[#177](https://github.com/chucklehead-dev/oscope/issues/177).
The exporter library's older-stack full physical-row recovery evidence remains
separate from this consumer integration.

No new throughput/p50/p90/p95/p99/S3/Langfuse/Samizdat/packaging or zero-allocation
claim. Defaults are not switched to the experimental codec. No unchanged
exhaustive or model-checker run; there is no new protocol action or ACK meaning.

## Failed launch/configuration attempts retained

- First baseline command exited127 before Jolt:an unquoted inherited Windows
PATH with spaces split the env assignment. Quoted PATH fixes the launcher.
- First actual baseline forced26.9 globally, causing the OLD minimal profile's
timestamp gate to reject (361tests,3537passes,2failures). This was the wrong
native/runtime pairing, not an exporter regression. Removing JOLT_CHDB_LIB
lets that graph select its declared26.7.3 package; full baseline then passes.
- First assertions-disabled invocation tried to analyze clojure.test/run-tests
before loading that namespace inside the binding form. It stopped before
tests. A separate require of clojure.test before the binding fixes selection.

All corresponding logs are retained, not presented as passing tests. The typed
probe's initial docstring called it a capture/recovery fixture; source inspection
corrected that label to live-query only (no executable observer change). This
note and issue177 explicitly preserve the distinction. Final-head hosted checks
must execute the current source before merge; local successes do not stand in
for those checks.

## Reproduce / retention

From root, prefix the pinned stock CLI with
`/home/chuck/ai-src/tools/jolt-with-chez-10.4.1`:

```text
jolt -Srepro -M:test-headless
jolt -Srepro -A:test-durable:stock-span-codec -m oscope.field-slot-native-probe
jolt -Srepro -A:test-durable:stock-span-codec -m oscope.export-options-native-test-runner stock-codec embedded
jolt -Srepro -A:test-durable:stock-span-codec -m oscope.export-options-native-test-runner stock-codec standalone
jolt -Srepro -A:test-durable:stock-json-reader -m oscope.export-options-native-test-runner stock-reader standalone
```

Set JOLT_BIN/PATH for children as above; use26.9 only for the current-root native
cases, not globally across the old minimal profile. Existing fixture controllers
join their readers and retire successful stores through their established gates.
No build/binary/new native asset was created. Small original logs at
`/home/chuck/ai-src/evidence/oscope-exporter-pin-20261021.ldxeRHQs`;
copies below are for public synthetic qualification evidence. The selected
pre-native source manifest predates the docstring-only correction and is not
complete transitive compile-time attestation.

## Independent review

Claude's bounded source-only review found no blockers. It checked the observer's
five rejected scenarios, transparent return/error handling, CI case retention,
unchanged SDK/profile pins, red-control provenance and assertion-count arithmetic.
The raw review is retained in `claude-review.txt`; Claude did not execute the
tests. Codex inspected the actual diff and owns the local execution evidence.
Final-head hosted CI remains required before merge.
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
9a251d7f224f2e13f6f991ee212264e12a4bed86a4feeb4bd9a386bdd00dc1dc deps.edn
d4a03658a27c176a28995876917af686421314e9b61e890554cf3dd522296c09 test/oscope/field_slot_native_probe.clj
0e1e83a20c15e222175cd409e260ef91a4f55c3b95c9904f683bcd30c65d37e3 test/oscope/field_slot_native_probe_test.clj
347c0a09d92de22b7a760050214d00e974f9b458e622803298bffcb7ec3d97e6 test/oscope/export_options_native_test_runner.clj
b08cac64545a527ca55e17c5338650c42b608679c520c11b9b2c15e0b855262e test/oscope/embedded_durable_integration_test.clj
20 changes: 20 additions & 0 deletions research/exporter-field-slot-pin-20261021/claude-review.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
No ExitPlanMode or Write tool is available in this session, so I'll deliver the review directly here.

## Review: root exporter field-slot pin (a6cbf939)

**Blockers: none found.**

### Verified claims

- **Witness correctness** — traced all 6 `run-control` scenarios in `field_slot_native_probe_test.clj` against `field_slot_native_probe.clj`: `:missing-projector` (no projector fn), `:constructor-only` (fn built, never invoked), `:startup-only` (invoked only while `active?` is still false, before `embedded/start!` returns), and `:declined-false`/`:declined-nil` (invoked post-startup but with a falsy result) all fail the `(and @active? (pos? @selected))` gate and throw `::unobserved-field-slots`. Only `:selected` (genuinely post-startup, truthy result) passes and prints the counter. No construction/startup/false/nil path can satisfy the witness.
- **Wrapper transparency** — the `factory-var` wrapper always returns `project`'s real return value to the span caller, and the `test/test-vars` wrapper always returns the real `result` after its check; neither swallows results or errors.
- **Existing receipt checks unaffected** — the gate's check/print happens *after* `run-tests`/fixture execution, so the wrapped fixture's own terminal receipt assertions still run and still determine pass/fail independently.
- **Red control confirmed in logs** — `prior-exporter-field-slot-probe.log` shows the prior exporter (`1e7cc8d0`) throwing `:oscope.field-slot-native-probe/unavailable-field-slots` from `ns-resolve` returning nil — a genuine missing-capability rejection, not a masked `require` failure.
- **Assertion-count arithmetic checks out** — 18 witness-control assertions (12 from the two `is` groups in the single deftest + 6 from `fixture/-main`'s mock `is` firing once per `run-control` call) + 3 new assertions in `dependency_resolution_test.clj`'s `prior-field-slot-exporter-is-a-causal-red-control` = 21, matching the reported 3539→3560 assertion delta and 361→363 test delta exactly.
- **CI coverage retained** — `stock-span-codec.yml` still runs embedded+standalone stock-codec, the new typed field-slot probe, and stock-reader standalone, plus the pre-existing json-reader/stock-codec/legacy-probe checks. No step removed.
- **No inadvertent SDK/profile rollout** — `deps.edn` changes only the `jolt-otel-clickhouse` root pin to `a6cbf939`; SDK/driver/JSON/crypto pins and the `:stock-span-codec`/`:stock-json-reader` codec alias SHA are unchanged. `profiles/embedded` is untouched; the workflow uses no `:override-deps` affecting it.
- **Scope honesty** — CHANGELOG, `docs/stock-codec.md`, and the research README all consistently disclose that fixture 3 is a memory-backed live query (not independent typed recovery), point to issue #177 for that gap, and disclaim throughput/perf/S3/Langfuse claims. Nothing is masked.

### Non-blocking observations

None — implementation, tests, CI, and docs are internally consistent and match each other's claims exactly.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer
error[analyze/unknown-namespace]: No such namespace: clojure.test
--> 1:96
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer

FAIL in (minimal-fixture-runs-the-real-native-stalled-remote-stack) (embedded_profile_test.clj:432)
Unhandled exception: Unqualified chDB telemetry timestamp wire
ex-data: {:type :otel.exporter.chdb/unqualified-timestamp-wire}
trace:
otel.exporter.chdb/exporter (~/.jolt/gitlibs/https___github.com_chucklehead-dev_jolt-otel-clickhouse.git/dfbc9ea64962b25c2f38912daa1c5a7b7339deaf/src/otel/exporter/chdb.clj:1204)

expected: (zero? (:exit run-result))
actual: (not (zero? 1))

FAIL in (minimal-fixture-runs-the-real-native-stalled-remote-stack) (embedded_profile_test.clj:434)
expected: (str/includes? (:out run-result) "minimal embedded native fixture: PASS")
actual: false
oscope HTTP overload load {:attempts 10000, :rejected 10000, :max-queue-depth 1, :rss-before-kib 631372, :rss-after-kib 631372}

Ran 361 tests. 3537 assertions passed, 2 failures, 0 errors.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
env: ‘Files/Eclipse’: No such file or directory
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer
oscope HTTP overload load {:attempts 10000, :rejected 10000, :max-queue-depth 1, :rss-before-kib 655348, :rss-after-kib 655348}

Ran 361 tests. 3539 assertions passed, 0 failures, 0 errors.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer
:export-encoding-fixture stock-codec embedded
:durable-native-executed fixture 4
:durable-native-child reader 1 :exit 0 :terminal true :settled true :valid true
:stream-native-calls 5
:stock-codec-calls 5
:durable-native-receipt fixture 4 1 18 0 0 1
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer
:export-encoding-fixture stock-codec embedded-typed
:durable-native-executed fixture 3
:bounded-field-slots-ingestion-calls 1
:stream-native-calls 1
:stock-codec-calls 1
:durable-native-receipt fixture 3 1 8 0 0 1
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer
oscope HTTP overload load {:attempts 10000, :rejected 10000, :max-queue-depth 1, :rss-before-kib 637692, :rss-after-kib 637692}

Ran 363 tests. 3560 assertions passed, 0 failures, 0 errors.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer
:export-encoding-fixture stock-codec standalone
:durable-native-executed fixture 2
:durable-native-child reader 0 :exit 0 :terminal true :settled true :valid true
:stream-native-calls 1
:stock-codec-calls 1
:durable-native-receipt fixture 2 1 45 0 0 1
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer
:export-encoding-fixture stock-reader standalone
:durable-native-executed fixture 2
:durable-native-child reader 0 :exit 0 :terminal true :settled true :valid true
:stream-native-calls 1
:stock-codec-calls 1
:stock-reader-results {:accepted 3, :declined 0}
:durable-native-receipt fixture 2 1 45 0 0 1
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer

Ran 1 tests. 18 assertions passed, 0 failures, 0 errors.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer

Ran 1 tests. 18 assertions passed, 0 failures, 0 errors.
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[jolt.deps] deps declare :deps/prep-lib steps jolt does not run (their compiled/generated assets will be missing): yogthos/selmer
Unhandled exception: Bounded field-slot exporter capability is unavailable
ex-data: {:type :oscope.field-slot-native-probe/unavailable-field-slots}
trace:
oscope.field-slot-native-probe/field-slot-factory-var (./test/oscope/field_slot_native_probe.clj:12)
oscope.field-slot-native-probe/-main (./test/oscope/field_slot_native_probe.clj:19)
Loading
Loading