Add opt-in LogRecordProcessor.Enabled

[pellared]

Fixes #4363

Towards #4208 (uses Severity Level passed via Logger.Enabled)

Towards stabilization of OpenTelemetry Go Logs API and SDK.

Use cases

Below are some use cases where the new functionality can be used:

1. Bridge features like LogLevelEnabled in log bridge/appender implementations. This is needed for all (but one) currently supported log bridges in OTel Go Contrib.
2. Configure a minimum log severity level for a certain log processor.
3. Filter out log and event records when they are inside a span that has been sampled out (span is valid and has sampled flag of false).
4. Efficiently support high-performance logging destination like Linux user_events and ETW (Event Tracing for Windows).
5. Bridge Logs API to a language-specific logging library (the other way than usual).

Changes

Add Enabled opt-in operation to the LogRecordProcessor.

I created an OTEP first which was a great for having a lot of discussions and evaluations of different proposals:

• OTEP: Logger.Enabled #4290

Most importantly from #4290 (comment):

Among Go SIG we were evaluating a few times an alternative to provide some new "filter" abstraction which is decoupled from the "processor". However, we faced more issues than benefits going this route (some if this is described here, but there were more issues: open-telemetry/opentelemetry-go#5825 (comment) . With the current opt-in Processor.Enabled we faced less issues so far.
We also do not want to replicate all features from the logging libraries. If someone prefer the log4j (or other) filter design then someone can always use a bridge and use log4j for filtering. Enabled callback hook is the simplest design (yet very flexible) which makes it easy to implement in the SDKs. This design is inspired from the design of the two most popular Go structured logging libraries: https://pkg.go.dev/log/slog (standard library) and https://pkg.go.dev/go.uber.org/zap.

It is worth to adding that Rust design is similar and it also has an Enabled hook. See #4363 (comment). Basically we want to add something like https://docs.rs/log/latest/log/trait.Log.html#tymethod.enabled to the LogRecordProcessor and allow users to implement Enabled in the way that it will meet their requirements.

I also want to call out form https://github.com/open-telemetry/opentelemetry-specification/blob/main/oteps/0265-event-vision.md#open-questions:

How to support routing logs from the Logs API to a language-specific logging library

To support this we would need a log record processor which bridges the Logs API calls to given logging library. For such case we would need an Enabled hook in Processor to efficiently bridge Logger.Enabled calls. A filterer design would not satisfy such use case.

I decided to name the new operation Enabled as:

1. this name is already used in logging libraries in many languages: Add opt-in LogRecordProcessor.Enabled #4439 (comment)
2. it matches the name of the API call (for all trace, metrics and logs APIs).

I was also considering OnEnabled to have the same pattern as for Emit and OnEmit. However, we already have ForceFlush and Shutdown which does not follow this pattern so I preferred to keep the simple Enabled name. For OnEmit I could also imagine OnEmitted (or OnEmitting) which does something after (or just before like we have OnEnding in SpanProcessor) OnEmit on all registered processors were called. Yet, I do not imagine something similar for Enabled as calling Enabled should not have any side-effects. Therefore, I decided to name it Enabled.

I want to highlight that a processor cannot assume Enabled was called before OnEmit, because of the following reasons:

1. Backward compatibility – Existing processors may already perform filtering without relying on Enabled. For example: Add Advanced Processing to Logs Supplementary Guidelines #4407.
2. Self-sufficiency of OnEmit – Since Enabled is optional, OnEmit should be able to handle filtering independently. A processor filtering events should do so in OnEmit, not just in Enabled.
3. Greater flexibility – Some processors, such as the ETW processor, don’t benefit from redundant filtering. ETW already filters out events internally, making an additional check unnecessary.
4. Performance considerations – Calling Enabled from OnEmit introduces overhead, as it requires converting OnEmit parameters to match Enabled's expected input.
5. Avoiding fragile assumptions – Enforcing constraints that the compiler cannot validate increases the risk of introducing bugs.

This feature is already implemented in OpenTelemetry Go:

• sdk/log: Add FilterProcessor and EnabledParameters opentelemetry-go#6317
  We have one processor in Contrib which takes advantage of this functionality:
• https://pkg.go.dev/go.opentelemetry.io/contrib/processors/minsev

This feautre (however with some differences) is also avaiable in OTel Rust; #4363 (comment):

OTel Rust also has this capability. Here's an example where it is leveraged to improve performance by dropping unwanted log early. https://github.com/open-telemetry/opentelemetry-rust/blob/88cae2cf7d0ff54a042d281a0df20f096d18bf82/opentelemetry-appender-tracing/benches/logs.rs#L78-L85

[jack-berg]

I disagree with it as even the spec says "log bridge/appender" in a few places.

I'm telling you the historical context as someone who was deeply involved of the original effort to design and stabilize the log API and SDK. The API and SDK were designed as simple tools to facilitate the primary use case of buffering logs in a queue to send to a OTLP network location. SDK extension points like processor and exporter were added for symmetry to traces. There was an explicit goal to not require OpenTelemetry language implementation to recreate the rich ecosystems of existing log frameworks, with long lists of appenders, encoders, etc. See that goal reflected in this text:

Existing logging libraries generally provide a much richer set of features than what is defined in OpenTelemetry. It is NOT a goal of OpenTelemetry to ship a feature-rich logging library.

If OpenTelemetry log appenders were supposed to be the implementation of log APIs, we'd have a much different / richer set of capabilities. With built-in capabilities, the log SDK is pretty much only good for buffering logs before sending to an OTLP network location and that is not an accident / coincidence.

There is no built-in zap feature like that.

From the docs:

Extending zap to support a new encoding (e.g., BSON), a new log sink (e.g., Kafka), or something more exotic (perhaps an exception aggregation service, like Sentry or Rollbar) typically requires implementing the zapcore.Encoder, zapcore.WriteSyncer, or zapcore.Core interfaces.

This would be analog to a new log sink like the kafka example the zap authors give.

We need not to forget that Logs API can also be used directly.

You're right, but this is a separate issue. We would still want to make sure go log appenders are implemented at the intended level of abstraction else go users will have a very different experience than other OpenTelemetry languages.

[pellared]

to not require OpenTelemetry language implementation to recreate the rich ecosystems of existing log frameworks

I get it and this is why I do not want to add any sophisticated components (like filters in log4j) but add the simplest yet flexible feature that can support multiple use cases (more in PR description). At last, it is added as a MAY. SDKs are not required to support it. For OTel Go such feature is a must have (this is not "my" opinion, but a statement that all @open-telemetry/go-maintainers agree with).

This would be analog to a new log sink like the kafka example the zap authors give.

This is can be only done by implementing a zapcore.Core and this is exactly what we are doing. This is not something which you understand as a log appender. slog and logr share similar design.

[jack-berg]

At last, it is added as a MAY. SDKs are not required to support it.

I'm not making arguments for / against this PR. This is tangential, but to me is an very important realization that helps explain the "why" behind so many things that I couldn't quite understand previously.

[jack-berg]

Just to complete this conversation, I've sketched out an idea of how go log appenders might be adjusted to more closely match the log appender concept in other languages: open-telemetry/opentelemetry-go-contrib#6964

I don't want to distract from this PR's content anymore, so let's take any followup to that draft PR or an issue.

[jsuereth]

@pellared - I do think #4439 (comment) may be something we should specify. It is easy to do, but something we should probably provide OOTB.