Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Repository files navigation

E-Agle TRT

Serializers

Serializers is a code generation library for C++ to be used with Google's Protocol Buffers.

Given the .proto files, it generates a header file and a source file for each proto, besides creating a CMakeLists.txt file to integrate the code with your main project and a header "serializers.h".

The library wraps the pb files generated by protoc, providing a simple and convenient interface to handle, serialize and deserialize the defined proto messages. It also provides functions to convert and parse the messages to JSON format.

The generated header consist in a struct for each proto Message and the following methods for serialization:

std::string serializeAsJsonString() const;
std::string serializeAsProtobufString() const;
bool deserializeFromJsonString(const std::string& str);
bool deserializeFromProtobufString(const std::string& str);

The methods implementation is built upon the pb files, which have to be added to the main project and included in the global include path.

Example

bar.proto (input)

syntax = "proto3";

package bar;

enum MyEnum
{
    MY_ENUM_0 = 0;
    My_ENUM_1 = 1;
}

message MySmallMessage
{
    int64 val = 1;
}

message MyMessage
{
    double val = 1;
    string str = 2;
    MySmallMessage mySmallMessage = 3;
    repeated MyEnum enumVec = 4;
    map<uint32, MySmallMessage> structMap = 5;
}

bar.h (output)

#ifndef BAR_H
#define BAR_H

#include "bar.pb.h"

[...]

namespace Serializers
{
enum class MyEnum
{
    MY_ENUM_0 = 0,
    My_ENUM_1 = 1
};

struct MySmallMessage
{
    int64_t val;

    [...]
};

struct MyMessage
{
    double val;
    std::string str;
    MySmallMessage mySmallMessage;
    std::vector<MyEnum> enumVec;
    std::unordered_map<uint32_t, MySmallMessage> structMap;

    MyMessage() = default;
    MyMessage(const bar::MyMessage& proto);
    operator bar::MyMessage() const;

    std::string serializeAsJsonString() const;
    std::string serializeAsProtobufString() const;
    bool deserializeFromJsonString(const std::string& str);
    bool deserializeFromProtobufString(const std::string& str);
};
}

#endif

Usage

  • Generate the code with: python3 src/generator/main.py proto_dir output_dir.
  • Copy the output directory in your project and generate the pb files with protoc, using the files in output_dir/proto.
  • Add add_subdirectory(path/to/output/directory) to your main CMakeLists.txt.
  • Include "serializers.h" in your files and use the library code.
#include "serializers.h"

#include <iostream>
#include <string>

int main()
{
    Serializers::MySmallMessage mySmallMessage;
    mySmallMessage.val = 3.14;

    Serializers::MyMessage myMessage;
    myMessage.str = "Federico Carbone";
    myMessage.enumVec.push_back(Serializers::MyEnum::MY_ENUM_0);
    myMessage.structMap.emplace(404, mySmallMessage);

    std::string myMessageSerialized = myMessage.serializeAsProtobufString();

    std::cout << myMessage.serializeAsJsonString() << std::endl;

    return 0;
}

Features

Implemented / Not implemented yet

  • Multiple file packages
  • All primitive fieldtypes support
  • Enum
  • "repeated" option
  • Maps
  • Package check
  • Import external proto definitions
  • Nested types
  • OneOf
  • Custom options

Docker image

The generator ships as a Docker image with the whole toolchain pinned (protoc, protoc-gen-jsonschema, Python and its dependencies), so the output only depends on the image version and on the .proto files.

docker run --rm -v $PWD/proto:/in:ro -v $PWD/out:/out \
  ghcr.io/eagletrt/serializers:<version> generate

Replace <version> with a published version (e.g. 1.0.0). latest and <major>.<minor> tags exist too, but projects should pin the exact version so that the generated code only changes when they decide to.

  • generate [--in DIR] [--out DIR] [--force] writes the C++/Python wrappers, the CMake project and the JSON Schemas in --out (defaults: /in and /out). The output folder must be empty or created by a previous run (it holds .serializers-version); any other non-empty folder is refused unless --force is given. The files written are listed in .serializers-manifest, and the next run deletes only those, so files you add to the folder are never removed.
  • check [--in DIR] generates the code, compiles the protos and runs the generated tests. This is what the CI runs.
  • clean [--out DIR] deletes the files listed in .serializers-manifest.

The image never writes into its own files (each run works in a temporary copy), so it can run as any user (--user "$(id -u):$(id -g)", so that the generated files are not owned by root) and with --read-only.

generate, check and clean are also commands inside the image, so a service can use it in a multi-stage build:

ARG SERIALIZERS_VERSION
FROM ghcr.io/eagletrt/serializers:${SERIALIZERS_VERSION} AS codegen
COPY lib/external/serializers/proto /in
RUN generate --out /out

FROM python:3.14-slim
COPY --from=codegen /out /app/generated

The image is published for linux/amd64 and linux/arm64, so Docker picks the native variant on Intel and Apple Silicon Macs, Windows and Linux.

Releasing

The image is published by .github/workflows/docker.yaml when a tag is pushed. The workflow first runs check with the image on both platforms against the protos of eagletrt/telemetry-serializers, and publishes only if it passes:

git tag v1.0.0 && git push origin v1.0.0   # publishes 1.0.0, 1.0 and latest

After the first release, make the serializers package public in the organization's package settings, so that it can be pulled without logging in.

To build and try it locally:

docker build -t serializers:dev .
docker run --rm -v $PWD/../telemetry-serializers/proto:/in:ro serializers:dev check

Development

The generator no longer installs the git hooks by itself: run pre-commit install once after cloning the repository.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages