[connector/elasticapm] Configure all required overflow attributes

[isaacaflores2]

Motivation

Metrics created by the connector need to match the expected overflow behavior required by the Observability UI. The expected overflow keys from the UI are service.name and transaction.name. See slack thread

Implementation

Currently only the service.name: _other is configured as an overflow attribute for all metrics. The connector should add the correct overflow attributes for metrics the are associated with transaction groups.

Challenge

The challenge is the attributes should only be added aggregated metrics associated with span signals i.e transaction.name: _other should not be added to aggregation metrics derived from logs and metrics. Currently the connector only creates one lsminterval processor which applies overflow attributes to all aggregation metrics. This can be done using OTTL, see comment

Expectation

When the transaction.name overflows it should be correctly label to drive the below UI (example derived from apm-aggregation). The Transaction section shows Remaining Transaction

[isaacaflores2]

@lahsivjar I have been attempting to compare the overflow behavior around transaction.name between apm-aggregation and elasticapmconnector. The goal is to observe the overflow in the elasticapmconnector to determine the best way to correctly label the overflow metric with transaction.name: _other.

I can trigger the transaction.name overflow behavior easily in apm-aggregation using otel bench with the below collector config.

otel-bench-config-host-name-cardinality

receivers:
  loadgen:
  nop:

exporters:
  otlp:
    endpoint: ${env:ELASTIC_APM_URL}
    headers:
      # The value of the authorization header does not matter, but it should
      # follow the format the elasticcloudauth expects it to be: base64 encoded,
      # and there should be one : splitting ID from the encoded value.
      Authorization: "ApiKey ${env:ELASTIC_APM_API_KEY}"
    sending_queue:
      enabled: false
    tls:
      insecure_skip_verify: true
    timeout: 60s
  otlphttp:
    endpoint: ${env:ELASTIC_APM_URL}
    headers:
      # The value of the authorization header does not matter, but it should
      # follow the format the elasticcloudauth expects it to be: base64 encoded,
      # and there should be one : splitting ID from the encoded value.
      Authorization: "ApiKey ${env:ELASTIC_APM_API_KEY}"
    sending_queue:
      enabled: false
    tls:
      insecure_skip_verify: true
    timeout: 60s
  nop:

processors:
  transform/rewrite:  # Rewrite telemetry to increase cardinality
    trace_statements:
      - context: span
        statements:
          # The worst way to generate a random ID, but is the simplest in OTTL
          # Only randomize trace ID such that span relationships are still maintained
          - set(trace_id.string, Substring(MD5(UUID()), 0, 32))
      - context: resource
        statements:
          - set(attributes["host.name"], Concat([attributes["host.name"], Substring(MD5(UUID()), 0, 5)], "_" ))
    log_statements:
      - context: resource
        statements:
          - set(attributes["host.name"], Concat([attributes["host.name"], Substring(MD5(UUID()), 0, 5)], "_" ))
    metric_statements:
      - context: resource
        statements:
          - set(attributes["host.name"], Concat([attributes["host.name"], Substring(MD5(UUID()), 0, 5)], "_" ))
service:
  pipelines:
    logs:
      receivers: [loadgen]
      processors: [transform/rewrite]
      exporters: [otlp] # this will be set dynamically to otlp or otlphttp depending on config
    metrics:
      receivers: [loadgen]
      processors: [transform/rewrite]
      exporters: [otlp] # this will be set dynamically to otlp or otlphttp depending on config
    traces:
      receivers: [loadgen]
      processors: [transform/rewrite]
      exporters: [otlp] # this will be set dynamically to otlp or otlphttp depending on config
  telemetry:
    logs:
      level: ERROR
    metrics:
      level: none

Focusing now on the elasticapmconnector. The above otel bench config also triggers the resource level overflow for elasticapmconnector. Based on my understand of the connector the transaction.name overflow should occur at the metric level. but I am unable to trigger an overflow. I tried the below otel bench config:

otel-bench-config-span-name-cardinality

receivers:
  loadgen:
  nop:

exporters:
  otlp:
    endpoint: "https://local.ingest.127.0.0.1.ip.es.io:8443"
    headers:
      # The value of the authorization header does not matter, but it should
      # follow the format the elasticcloudauth expects it to be: base64 encoded,
      # and there should be one : splitting ID from the encoded value.
      Authorization: "ApiKey ${env:ELASTIC_APM_API_KEY}"
    sending_queue:
      enabled: false
    tls:
      insecure_skip_verify: true
    timeout: 60s
  otlphttp:
    endpoint: "https://local.ingest.127.0.0.1.ip.es.io:8443"
    headers:
      # The value of the authorization header does not matter, but it should
      # follow the format the elasticcloudauth expects it to be: base64 encoded,
      # and there should be one : splitting ID from the encoded value.
      Authorization: "ApiKey ${env:ELASTIC_APM_API_KEY}"
    sending_queue:
      enabled: false
    tls:
      insecure_skip_verify: true
    timeout: 60s
  nop:

processors:
  transform/rewrite:  # Rewrite telemetry to increase cardinality
    trace_statements:
      - context: span
        statements:
          # The worst way to generate a random ID, but is the simplest in OTTL
          # Only randomize trace ID such that span relationships are still maintained
          - set(trace_id.string, Substring(MD5(UUID()), 0, 32))
          - set(name, Concat([name, Substring(MD5(UUID()), 0, 5)], "_" )) where name != nil

service:
  pipelines:
    logs:
      receivers: [loadgen]
      processors: [transform/rewrite]
      exporters: [otlp] # this will be set dynamically to otlp or otlphttp depending on config
    metrics:
      receivers: [loadgen]
      processors: [transform/rewrite]
      exporters: [otlp] # this will be set dynamically to otlp or otlphttp depending on config
    traces:
      receivers: [loadgen]
      processors: [transform/rewrite]
      exporters: [otlp] # this will be set dynamically to otlp or otlphttp depending on config
  telemetry:
    logs:
      level: ERROR
    metrics:
      level: none

and the below elasticapmconnector limits

  limits:
    resource:
      max_cardinality: 1
    scope:
      max_cardinality: 1
    metric:
      max_cardinality: 1
    datapoint:
      max_cardinality: 1

This config creates transaction.name with high cardinality (the UUID suffix), but I only continue to see overflow at the resource level this identified by querying for resource.attributes.overflow: * in Kibana.

Questions:

1. Should transaction.name overflow at the metric level?
2. Do you see anything wrong with my approach to trigger the transaction.name overflow?

[lahsivjar]

Didn't get enough time to go through the setup today but if we want to trigger a metric overflow then we will have to generate data such that resource and scope are same and same metric has multiple attributes giving higher than max cardinality.

Also, apm-aggreagtion will not 100% correlate with the Elastic APM connector's overflow. I remember we discussed that eventually we will have to get UI handle the overflow for OTel separately - I can fish out the exact discussion and conclusions tomorrow.

[isaacaflores2]

Thanks @lahsivjar. I will double check that the setup I used is keeping the same resource/scope. For my last attempt, I increased the span.name cardinality to attempt to increase the attribute.transaction.name cardinality. My assumption was that this should trigger an overflow at the metric level

My understanding after double checking with the UI team (see slack thread, is that we only need to handle service.name and transaction.name for now since these are the only attributes where the UI is relying on the _other value.

[isaacaflores2]

I have a better idea of when transaction.name will overflow. It should overflow at the datapoint level. Sharing my notes below:

Overflow notes

Observations made by running connector_test.go test case: go test -v -run=TestConnector_TracesToMetrics/traces/transaction_metrics -update with different inputs and aggregation limits.

• https://github.com/elastic/opentelemetry-collector-components/compare/main...isaacaflores2:opentelemetry-collector-components:test-tx-name-overflow?expand=1

Level

Overflow at the resource and scope level are not affected by the cardinality of the span attribute: transaction.name

metric

Metric overflow is unlikely since the elasticapmconnector has a fixed number of metrics at fixed intervals i.e for each scope the connector will not create any metrics not defined below

Metric	Source Signal
service_summary	Logs, Metric, Spans
span.destination.service.response_time.sum.us	Spans
span.destination.service.response_time.count	Spans
transaction.duration.histogram	Spans
transaction.duration.summary	Spans
event.success_count

Overflow was only observed when the metric limit was < 4

elasticapm:
  aggregation:
    limits:
        metric:
            max_cardinality: 3

resourceMetrics:
  - resource:
    scopeMetrics:
      - metrics:
          - description: Overflow metric count due to metric limit
            name: _overflow_metric
            sum:
              aggregationTemporality: 1
              dataPoints:
                - asInt: "1"
                  attributes:
                    - key: data_stream.dataset
                      value:
                        stringValue: <nil>.1m
                    - key: metricset.interval
                      value:
                        stringValue: 1m
                    - key: overflow
                      value:
                        stringValue: metric
                    - key: processor.event
                      value:
                        stringValue: metric

datapoint

Adding unique transaction.names for the same resource and scope results in a new datapoint per transaction metric (metrics with metric.set: transaction)

resourceMetrics:
  - resource:
    scopeMetrics:
      - metrics:
          - description: Overflow datapoint count due to datapoint limit
            name: _overflow_datapoints
            sum:
              aggregationTemporality: 1
              dataPoints:
                - asInt: "1"
                  attributes:
                    - key: data_stream.dataset
                      value:
                        stringValue: <nil>.1m
                    - key: metricset.interval
                      value:
                        stringValue: 1m
                    - key: overflow
                      value:
                        stringValue: datapoint
                    - key: processor.event
                      value:
                        stringValue: metric