Skip to content

Feat/docker codegen - #2

Merged
KevinRomanello merged 12 commits into
mainfrom
feat/docker-codegen
Nov 10, 2026
Merged

KevinRomanello merged 12 commits into
mainfrom
feat/docker-codegen

Conversation

@stepdossi1

@stepdossi1 stepdossi1 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Add a Docker image with the whole generator toolchain pinned (protoc 3.21.12, protobuf 7.35.1, protoc-gen-jsonschema at a fixed commit). Commands: generate and check.
  • Publish it on ghcr.io/eagletrt/serializers:<version> for linux/amd64 and linux/arm64 when a vX.Y.Z tag is pushed.
  • Run the tests inside the image.
  • Fix the Python wrapper template: setdefault() on a map of messages is rejected by protobuf >= 4, so use map[key].CopyFrom().

Why

The generated code now depends only on the image version and on the protos, so telemetry-serializers can stop generating it and every project generates its own.

Tested

check with protobuf 7.35.1 and protoc 3.21.12: 187 tests pass (also with 3.20.0).

Not tested

The image has not been built yet, and the workflows have not run. The C++ output is unchanged and not tested.

After merging

Tag v1.0.0 to publish the image, and make the package readable by telemetry-serializers.

Copilot AI balanced review requested due to automatic review settings November 10, 2026 14:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@radix54 radix54 added the enhancement New feature or request label Nov 10, 2026
Comment thread Dockerfile Resolved
Comment thread README.md Resolved
Comment thread README.md Resolved
Comment thread Dockerfile Resolved

@KevinRomanello KevinRomanello left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice direction overall: one versioned image as the single source of the generator is a big improvement. Two things I'd fix before merging (the .serializers-version marker logic and publishing without running check), plus a few suggestions inline. The setdefault → map[key].CopyFrom() fix is correct and still works with protobuf 3.20, so the legacy workflows are fine.

Merge order with eagletrt/telemetry-serializers#18: merge this → tag v1.0.0 → make the ghcr package public → merge #18.

Comment thread docker/serializers.sh Resolved
Comment thread docker/serializers.sh Resolved
Comment thread .github/workflows/docker.yaml Resolved
Comment thread .github/workflows/docker.yaml Resolved
Comment thread .github/workflows/docker.yaml Resolved
Comment thread Dockerfile Resolved
Comment thread Dockerfile Resolved
Comment thread Dockerfile Resolved
Comment thread src/generator/main.py Resolved
Comment thread scripts/generate_schemas Resolved

@KevinRomanello KevinRomanello left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works but it has some problem and it can be improved

KevinRomanello and others added 7 commits November 10, 2026 22:57
The path was taken from the text after 'proto/', so the generator failed
with an IndexError when the input folder has another name, such as /in in
the Docker image. The output is unchanged for the existing layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HoaykH7nb7C6u744T4xSG4
The marker only protected the first run: after writing into a wrong,
non-empty folder, the next run deleted everything in it. generate now
refuses a non-empty folder it did not create (unless --force), records
the files it writes in .serializers-manifest and deletes only those.
Add a clean command, and build in a temporary copy so that
/opt/serializers is never written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HoaykH7nb7C6u744T4xSG4
Drop the world-writable /opt/serializers, now that every run works in a
temporary copy. Cross-compile protoc-gen-jsonschema on the build platform
(static, CGO_ENABLED=0) instead of building it under QEMU for arm64.
On a tag the image was pushed without running the tests. Build and load
each platform, run check against the telemetry-serializers protos, then
publish with the OCI labels and the major.minor tag too.
Without set -e a protoc failure was ignored. Also find the .proto files
at any depth and drop the unused PROTO_FILES.

@KevinRomanello KevinRomanello left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review after the new commits (e4c1d1a).

CI: the image job is green: real Docker build with the real protoc-gen-jsonschema, then check passing on linux/amd64 and on linux/arm64 (emulated), both as a non-root user. Run Tests is green too.

Fixed, threads resolved:

  • output folder: generate refuses a non-empty folder it did not create (unless --force) and deletes only the files listed in .serializers-manifest; new clean command;
  • every run works in a temporary copy, /opt/serializers is no longer world-writable;
  • the image is tested on both platforms before being published; OCI labels and the major.minor tag are added;
  • generate_schemas stops on the first protoc error and finds .proto files at any depth;
  • no more pre-commit install from the generator;
  • README examples use <version> instead of 1.0.0 (@radix54).

Blocker found and fixed while testing: utils.load_files took the path from the text after proto/, so with the image defaults (/in) both generate and check crashed with an IndexError. It now uses the path relative to the input folder; the output is byte-identical on the current layout.

Still open (non-blocking, team decisions): Python 3.14 and protoc from the official release instead of Debian, protobuf/lock file in requirements.txt, base images pinned by digest. See the open threads.

Merge order unchanged: merge → tag v1.0.0 → make the package public → merge eagletrt/telemetry-serializers#18.

Comment thread .github/workflows/test.yaml Resolved
…uirements

protoc now comes from the official protobuf release (35.1, sha256-checked
by ADD) instead of Debian, so it always matches the protobuf 7.35.1
runtime, which is now pinned in requirements.txt. The base image moves to
python:3.14-slim-trixie.
@KevinRomanello
KevinRomanello merged commit f13445d into main Nov 10, 2026
3 checks passed
@KevinRomanello
KevinRomanello deleted the feat/docker-codegen branch November 10, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants