[beats receivers] Output translation does not use correct TLS configuration

[belimawr]

When enabling agent.monitoring._runtime_experimental: "otel" the TLS configuration of the output is not translated to the Beats receivers configuration.

Steps to reproducs

1. Create a stack using Elastic-Package (easy way to use custom TLS certificates)

elastic-package stack up -v -d --version=9.2.0-SNAPSHOT

2. Create a log file constantly appending data to it (e.g: using flog)

flog -d1 -s1 -f rfc5424 -w -o /tmp/flog.log -t log &

3.Create a policy with the Custom Logs (Filestream) integration

When adding the integration set paths to /tmp/flog.log

4. Deploy the Elastic-Agent following the instructions on Kibana and adding --insecure

sudo ./elastic-agent install --url=https://fleet-server:8220 --enrollment-token=a2F3SEtwa0JhQm4yRzJpWDNKcTA6Mnl6N0xVM3dINnp1THdycmdCaUhLdw== -insecure

5. Confirm the Elastic-Agent is correctly sending data (both integration and monitoring)

6. Go to Dev Tools and set the monitoring runtime to otel

Go to Dev Tools and run the request below. Don't forget to edit the Policy ID to match yours.

PUT kbn:/api/fleet/agent_policies/<POLICY ID>
{
   "name": "Test policy",
    "namespace": "default",
    "overrides": {
         "agent": {
              "monitoring": {
                "_runtime_experimental": "otel"
              }
            }
    }
}

7. Run elastic-agent status and confirm the TLS settings are not valid in otel mode:

root@archlinux /home/vagrant/elastic-agent-9.2.0-SNAPSHOT-linux-x86_64 % elastic-agent status                
┌─ fleet
│  └─ status: (HEALTHY) Connected
└─ elastic-agent
   ├─ status: (DEGRADED) 1 or more components/units in a degraded state
   ├─ beat/metrics-monitoring
   │  ├─ status: (DEGRADED) DEGRADED
   │  └─ beat/metrics-monitoring
   │     └─ status: (DEGRADED) Elasticsearch request failed: tls: failed to verify certificate: x509: certificate signed by unknown authority
   ├─ filestream-monitoring
   │  ├─ status: (DEGRADED) DEGRADED
   │  └─ filestream-monitoring
   │     └─ status: (DEGRADED) Elasticsearch request failed: tls: failed to verify certificate: x509: certificate signed by unknown authority
   └─ http/metrics-monitoring
      ├─ status: (DEGRADED) DEGRADED
      └─ http/metrics-monitoring
         └─ status: (DEGRADED) Elasticsearch request failed: tls: failed to verify certificate: x509: certificate signed by unknown authority
root@archlinux /home/vagrant/elastic-agent-9.2.0-SNAPSHOT-linux-x86_64 % 

8. Collect the diagnostics and look at otel-merged.yaml

You'll see the CA trusted fingerprint configuration (or any configuration to accept custom certificates) is missing:

exporters:
    elasticsearch/_agent-component/monitoring:
        api_key: <REDACTED>
        batcher:
            enabled: true
            max_size: 50
            min_size: 0
        compression: gzip
        compression_params:
            level: 1
        endpoints:
            - https://elasticsearch:9200
        idle_conn_timeout: 1m0s
        logs_dynamic_id:
            enabled: true
        mapping:
            mode: bodymap
        retry:
            enabled: true
            initial_interval: 1s
            max_interval: 1m0s
            max_retries: 3
        timeout: 1m30s
        tls:
            max_version: "1.3"
            min_version: "1.2"

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

This should be fixed by #9633 CC @khushijain21 this is a good test to do once you have fixed that.

elastic-package provisions self-signed certificates everywhere and clearly that is missing in the existing translation. I will mark this as a sub-issue of the one above.

[cmacknz]

Reworded the title because the otelconsumer output does not make outgoing HTTP calls and is just used to glue the beat receiver to the collector pipeline.

[khushijain21]

What was the original elastic-agent.yml where you enabled agent.monitoring._runtime_experimental: "otel" because unless you explicitly pass certificates it is not going to be rendered in es-exporter config.

[belimawr]

What was the original elastic-agent.yml where you enabled agent.monitoring._runtime_experimental: "otel" because unless you explicitly pass certificates it is not going to be rendered in es-exporter config.

It was a Fleet-Managed Elastic Agent, so the output configurations were correct there.

Here is an example policy exported from Fleet UI (elastic-package):

id: c22538ac-19b9-41fd-a0b0-7c9edab6924b
revision: 4
outputs:
  default:
    type: elasticsearch
    hosts:
      - https://elasticsearch:9200
    ssl.ca_trusted_fingerprint: A69BACBAAE1B323E95223A37D36224F45607B9E5D415EA11A9D462A043F9441C
    preset: latency
fleet:
  hosts:
    - https://fleet-server:8220
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.cloud_defend-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.cloudbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.cloudbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat_input-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat_input-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_elastic_collector-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_elastic_symbolizer-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.pf_host_agent-default
          privileges:
            - auto_configure
            - create_doc
        - names:
            - logs-elastic_agent.status_change-default
          privileges:
            - auto_configure
            - create_doc
    _elastic_agent_checks:
      cluster:
        - monitor
    b775375c-91bf-40fb-b027-17dec14443fc:
      indices:
        - names:
            - logs-filestream.generic-default
          privileges:
            - auto_configure
            - create_doc
agent:
  download:
    sourceURI: https://artifacts.elastic.co/downloads/
  monitoring:
    enabled: true
    use_output: default
    logs: true
    metrics: true
    traces: true
    namespace: default
    _runtime_experimental: otel
  features: {}
  protection:
    enabled: false
    uninstall_token_hash: KY0f7s9BYLUsKjDTT6yxla+K4MR89uHSbqjW/1z5DTU=
    signing_key: >-
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE3ZZ6CfevZs4dLAoMcEBxIdqUcu+z/+pwwt0scHetWu1KjLw5dv5qqIQHvtXDDK74K9wCIQvV1CnHZxpeIAxFxg==
inputs:
  - id: filestream-filestream-b775375c-91bf-40fb-b027-17dec14443fc
    name: filestream-1
    revision: 1
    type: filestream
    use_output: default
    meta:
      package:
        name: filestream
        version: 1.2.0
    data_stream:
      namespace: default
    package_policy_id: b775375c-91bf-40fb-b027-17dec14443fc
    streams:
      - id: filestream-filestream.generic-b775375c-91bf-40fb-b027-17dec14443fc
        data_stream:
          dataset: filestream.generic
          type: logs
        paths:
          - /tmp/flog.log
        prospector.scanner.recursive_glob: true
        prospector.scanner.exclude_files:
          - \.gz$
        clean_inactive: -1
        parsers: null
        file_identity.fingerprint: null
        prospector.scanner.fingerprint.enabled: true
        prospector.scanner.fingerprint.offset: 0
        prospector.scanner.fingerprint.length: 1024
signed:
  data: >-
    <secret>
  signature: >-
    <secret>
secret_references: []
namespaces:
  - default

[khushijain21]

oh, I see we only enable ssl.ca_trusted_fingerprint which is unsupported upstream. This should be fixed by
#9633