processor/lsmintervalprocessor: investigate using OTel/Arrow for intermediate representation

[axw]

A columnar encoding should be a good fit for the lsminterval processor's intermediate representation stored in Pebble, since it should be more amenable to aggregation and compaction. We should investigate using OTel/Arrow for this. The general idea would be:

• ConsumeMetrics will convert pdata to OTel/Arrow on the way in
• Merges will operate entirely on OTel/Arrow
• On harvest, convert OTel/Arrow back to pdata

Merges should then be able to operate directly on the low-level Arrow data structures.

[lahsivjar]

I have been working towards a PoC for this since Tuesday and I think this is going to take a lot more time. The current state of otel-arrow is still quite far from the proposed phase-2 where OTAP (OpenTelemetry Arrow Protocol) based OTel pipelines (written in rust) will be introduced.

I have attempted a few things to build a PoC and the major blocker I have come across is working with the columnar layout of the OTel data for merging (see current Arrow data model here). Performing merges has been possible but inefficient so far in my attempts because of the way the schema is designed - it has batch local IDs for various entities (resource, scope, datapoints) and are designed for the purpose of conversion to and from pmetric.Metrics which we couldn't use to perform merges on Arrow schema directly.

An alternative approach I tried was to build a custom view on top of the arrow records as is done here. The idea was to directly build a map to allow us to query for resources, scopes, and datapoints by a hash representing their identity (ignoring hash-collisions for the PoC) but it required moving the arrow representation more towards pmetric.Metrics structure defeating the purpose. In addition, the current APIs in otel-arrow repository make it difficult to re-serialize the data after performing the merges as the data representation during serialization and de-serialization are different (example: RelatedData during serialization and de-serialization).

I have tried a few more approaches but everytime came to a conclusion that we would need some major changes in the otel-arrow package to make things possible. My current recommendation is to work with the otel-arrow group to investigate the phase-2 of the Arrow project and get back to this once we have made some progress there - I have also started a Slack conversation with the folks working towards this.

CC: @axw @felixbarny

[felixbarny]

My current recommendation is to work with the otel-arrow group to investigate the phase-2 of the Arrow project and get back to this once we have made some progress there

Makes sense to me. What are some points we would like to make sure they consider during that phase? Like things that would enable us to do aggregations based on the arrow representation.

[axw]

My current recommendation is to work with the otel-arrow group to investigate the phase-2 of the Arrow project and get back to this once we have made some progress there - I have also started a Slack conversation with the folks working towards this.

Sounds sensible to me too. Thank you for starting that conversation, seems like the otel-arrow SIG folks are in agreement that this is a valid approach, but it'll take a little while to get there.

[lahsivjar]

What are some points we would like to make sure they consider during that phase?

Ideally, the Arrow-friendly form that will be exposed to the LSM interval processor should have the following properties:

1. O(1) check and retrieval for resource, scope, metric, and datapoints - it could be great if we can do the same check and retrival with a composite key build from pmetric model directly in addition to being able to do this using another Arrow friendly from.
2. Being able to build on top of the Arrow-friendly form - i.e. append data to it.
3. Being able to reserialize the form and put it to disk.

There are also a few things we need to look into with regards to how to persist the data to disk. We have an option to convert Arrow to proto and persist as well using Parquet files directly.

[raultorrecilla]

Removing this from intake-services board and moving it to managed otlp input board as post-ga task.
Thanks @lahsivjar for the investigation 🚀

[bturquet]

Removing the Ungated parent issue as it's not part of this workstream anymore