Repository navigation
Decode Tesla Fleet Telemetry protobuf for the NATS add-on for Splunk - #1
Merged
Merged
Conversation
A static Go binary implementing the splunk_nats decoder protocol: one JSON
request per line on stdin, one JSON event (or an empty line plus a stderr
reason) per line on stdout, flushed per line, exiting when stdin closes.
- The record type comes from the subject's last token (data, alerts, errors,
connectivity, metrics), verified against live telemetry.> traffic.
- Vehicle data is flattened to one top-level field per signal by default
("VehicleSpeed": 28.6, "Location": {"latitude": ...}), with invalid signals
listed in "invalid"; -format protojson gives fleet-telemetry's own JSON.
Field names follow the protobuf JSON mapping (createdAt); -proto-names
switches to created_at.
- Messages are generated from Teslemetry/tesla-protocol at a pinned commit by
scripts/generate.sh with pinned protoc and protoc-gen-go; CI checks the
committed code matches.
- Golden tests on anonymized live payloads for every record type, protocol and
binary tests, and throughput benchmarks.
- CI (gofmt, vet, race tests, benchmarks, generated code, build) and a
tag-triggered release with linux/darwin amd64/arm64 binaries and SHA256SUMS.
- scripts/e2e-splunk.sh proves it in a disposable Splunk with the add-on
against live NATS.
Alerts and errors carry vin and createdAt after a long array (alert events reach 16 KB), past where Splunk stops extracting fields at search time: a live natssubscribe sample had an alert event without vin. The flat format now starts every record type with vin and createdAt; -format protojson keeps proto order. The Splunk E2E checks every indexed event has both.
ℹ️ Recent review info
📝 Walkthrough
Note 🎁 Summarized by CodeRabbit Free
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Go payload decoder for the NATS Add-on for Splunk (Bre77/splunk_nats) that turns Tesla Fleet Telemetry protobuf on
telemetry.<VIN>.{data,alerts,errors,connectivity,metrics}into one-line JSON events.What it does
subject,headers, base64payload), one line out per request, flushed each time. The line is the event, or an empty line with a one-line reason on stderr. It exits 0 when stdin closes. Bad JSON, bad base64, an empty payload, a payload that is not valid protobuf, an unknown subject suffix and even a panic each get an empty line, and the decoder keeps serving.telemetry.>capture (12,989 messages, 790 vehicles), every subject had 3 tokens and a suffix ofdata,alerts,connectivityorerrors. There was nometricsstream, butmetricsis mapped toVehicleMetricsanyway.transmit_decoded_recordsJSON: lowerCamel (createdAt,networkInterface), enums as names, every field emitted. By default, vehicle data is flattened to one top-level field per signal, so Splunk extractsVehicleSpeed,Location.latitudeandDoorState.DriverFrontdirectly. Invalid signals are listed ininvalid. A signal or value type that is newer than the protos is kept (named by its number, ornull), not dropped.-format protojsongives fleet-telemetry's exact key/value shape.-proto-namesgives snake_case (created_at,network_interface). That matches the Python example decoder in the splunk_nats README, whose example searches (status network_interface) use proto names.Generation and builds
internal/protosis generated byscripts/generate.shand committed, with these pins:e6f7afb1b633d7215e34d0cdac66dab983c6ae78tooldirective ingo.modCI regenerates the code and fails on any diff. A local rerun is byte-identical.
scripts/build.shbuilds staticCGO_ENABLED=0binaries for linux/amd64, linux/arm64, darwin/amd64 and darwin/arm64, plusSHA256SUMS. Two builds of the same commit are byte-identical..github/workflows/ci.ymlruns gofmt,go vet,go test -race, the benchmarks, the generated-code check and the build.release.ymlruns onv*tags: it tests, builds and creates a GitHub release with the binaries andSHA256SUMS. No tag was created. The Go toolchain is pinned to go1.27.2, and the actions are pinned by SHA.Tests
Golden tests run 10 real payloads from live traffic through all 4 output options:
tools/mkfixturesanonymized them: fake VINTESTVIN0000000001, fake connection IDs and UUIDs, and Fremont factory coordinates. Destination, vehicle and media names were replaced. None of the 790 live VINs appear anywhere in the repo, and that was checked.Unit tests cover every value kind, duplicate signals, signals and enum values newer than the protos, float32 formatting, JSON string escaping and invalid UTF-8.
Protocol tests cover bad base64, an unknown subject, empty and missing payloads, non-protobuf, non-JSON, an empty line, CRLF, EOF with no input, a last line without a newline, a 1 MB request, and per-line flushing over a pipe.
A binary test builds the static binary and drives it as the add-on does: a request, waiting for the answer, then closing stdin and expecting exit 0.
Throughput:
BenchmarkRun(protocol loop, fixture mix): about 15 µs per message, about 65,000 msg/s.golangci-lint v2 (latest) reports 0 issues, and actionlint is clean.
End-to-end proof
scripts/e2e-splunk.sh(in this PR) uses a disposable Splunk container:9c5d52a) and installed.local/decoders/telemetry-decoder, with an account on the local hq leafnats://host.docker.internal:4222. Credentials come from 1Password over curl stdin and are never in argv or on disk.natssubscribe ... decoder=telemetry-decodersearch, then a NATS Subscribe input ontelemetry.>for 5 minutes.nats:message(base64) or if splunkd.log has any "failed to decode".Containers are removed with
docker rm -fv. Output below has VINs and locations aggregated away.Results with the final commit (
67edffa), 5 minutes of livetelemetry.>each:nats:json, nodecode_errornats:json, nodecode_errornats:message)vinorcreatedAtextractedFields extracted directly from the decoded data events, 10.4.1 run:
The first run on the earlier commit also passed with 0 failures (29,242 and 29,583 events), but in its natssubscribe sample one large alert event had no
vinextracted. protojson putsvinafter the alerts array, and Splunk stops extracting JSON fields partway into big events. The second commit putsvinandcreatedAtfirst in every flat event, and the E2E now asserts it.Notes
errorsmessages wereunsupported_field(for exampleLifetimeEnergyGainedRegen) andunsuported_interval(Tesla's spelling). Vehicles send these about the fleet-telemetry config, so they are not a decoder issue.Geararrives both as theShiftStateenum (ShiftStateD) and, from some vehicles, as a plain string (P,R). One run saw both. The decoder keeps values as sent, as fleet-telemetry does, so a search across both forms has to match both.