Skip to content

[AWS Firehose] Non-S3 logs are incorrectly classified as aws.s3access #15157

@MichaelKatsoulis

Description

@MichaelKatsoulis

Non-S3 logs are incorrectly classified as aws.s3access due to token count fallback logic.

This log

{
  "_time": 1756320965.344,
  "message": "W0903 18:05:03.633758      13 logging.go:55] [core] [Channel #3891283 SubChannel #3891284]grpc: addrConn.createTransport failed to connect to {Addr: \"172.16.32.16:12379\", ServerName: \"172.16.32.16:12379\", }. Err: connection error: desc = \"transport: authentication handshake failed: context canceled\"",
  "cloud": {
    "region": "us-west-2",
    "provider": "aws",
    "account": {
      "id": "000000000"
    }
  },
  "data_stream": {
    "type": "logs",
    "dataset": "awsfirehose"
  },
  "aws": {
    "firehose": {
      "arn": "arn:aws:firehose:us-west-2:000000000:deliverystream/uvp-logs-nonprod",
      "request_id": "33330164-8b2a-44cc-9293-16689b788447"
    },
    "cloudwatch": {
      "log_group": "/aws/eks/enmi-uat-uat2/cluster",
      "log_stream": "kube-apiserver-baa6c29217ce4caed1c7dd8d9aa6ef9a"
    },
    "kinesis": {
      "name": "uvp-logs-nonprod"
    }
  },
  "event": {
    "id": "33330164-8b2a-44cc-9293-16689b788447"
  }
}

is classified as s3access because the message has 25 tokens although it is not.
We need to think where should messages like this be routed.
aws.cloudwatch dataset could be a solution

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions