Is your feature request related to a problem? Please describe.
If using the OpenTelemetry output to send metrics to AWS CloudWatch's OTLP endpoint, AWS's implementation has a hard limit of 1,000 datapoints per request. This means that if you generate a lot of datapoints (eg, using the process_exporter_metrics input on a host with a lot of processes), sending these metrics to AWS via OpenTelemetry can fail. This will end up generating an error and a response like this back from AWS:
[error] [output:opentelemetry:opentelemetry.1] monitoring.us-east-1.amazonaws.com:443, HTTP status=400
Maximum number of datapoints per request exceeded: 1000.
Describe the solution you'd like
The OpenTelemetry output seems to support some options to restrict the amount of data sent in one request, but only for logs (not metrics):
batch_size: Set the maximum number of log records to be flushed at a time.logs_max_resources: Set the maximum number of OTLP log resources per export request (0 disables the limit).
Would some similar options be possible for metrics, so that the requests sent to AWS could be batched and always be under the 1,000 datapoint limit?
Describe alternatives you've considered
The only real alternative I've found is trying to limit the number of datapoints collected at a time, but this may not always be feasible.
Alternatively, I haven't tested this, but I believe the OpenTelemetry Collector could support this via their Batch Processor (since it looks like that supports batching on metrics too), but I'd prefer to keep using Fluent Bit for my current purposes.
Additional context
AWS's OpenTelemetry support is fairly new, and I realize this is a limitation with their implementation, but it would be great if Fluent Bit's OpenTelemetry output could better support this. Thanks!
Is your feature request related to a problem? Please describe.
If using the OpenTelemetry output to send metrics to AWS CloudWatch's OTLP endpoint, AWS's implementation has a hard limit of 1,000 datapoints per request. This means that if you generate a lot of datapoints (eg, using the
process_exporter_metricsinput on a host with a lot of processes), sending these metrics to AWS via OpenTelemetry can fail. This will end up generating an error and a response like this back from AWS:Describe the solution you'd like
The OpenTelemetry output seems to support some options to restrict the amount of data sent in one request, but only for logs (not metrics):
Would some similar options be possible for metrics, so that the requests sent to AWS could be batched and always be under the 1,000 datapoint limit?
Describe alternatives you've considered
The only real alternative I've found is trying to limit the number of datapoints collected at a time, but this may not always be feasible.
Alternatively, I haven't tested this, but I believe the OpenTelemetry Collector could support this via their Batch Processor (since it looks like that supports batching on metrics too), but I'd prefer to keep using Fluent Bit for my current purposes.
Additional context
AWS's OpenTelemetry support is fairly new, and I realize this is a limitation with their implementation, but it would be great if Fluent Bit's OpenTelemetry output could better support this. Thanks!