[Research] Cost Management Receiver

[constanca-m]

This issue is to open the discussion on how we could get data for:

• Resources costs
• Savings recomendations
• Forecast

The receiver (note: open to be a different component or to be integrated in an already existent component, but throughout this description I will write it as a receiver), would be a bit like:

receiver:
  costmanagement:
     # we could have different providers,
     # each would be implementing the same
     # interface
     provider: aws

Focusing a bit more on AWS, there are a few functions available on AWS SDK that we could use:

For the AWS use case in specific, it would have support for:

• GetCostAndUsage function
• GetCostForecast function
• GetSavings*Recommendation functions

To help visualize, let's imagine the configuration for the function GetCostAndUsage and GetCostForecast:

receiver:
  costmanagement:
    provider: aws
    aws:
      # I did no research do see which fields are in common
      # between all three features, so I will only consider
      # the fields in common between forecast and usage under
      # this block.    
      billing: [optional] # the billing view arn    
      filters:
        - [list of expressions]
      granularity: DAILY
      poll_interval: 24h # when to make next request
      
      # This part applies especifically to the forecast.
      # Each common field can be overwritten if necessary.
      # See https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_GetCostForecast.html.
      forecast:
        filters:
          - [add extra filters]     
        metric: METRIC
        time:
          start: START
          end: END
          
      # This part applies especifically to the usage.
      # Each common field can be overwritten if necessary.
      # See https://docs.aws.amazon.com/aws-cost-management/latest/APIReference/API_GetCostAndUsage.html
      usage:
        metrics:
          - [list of metrics]
        filters:
          - [add extra filters]
        group_by:
          - [add groups]
        time:
          start: START
          end: END

Example use case for us: we deploy this receiver in a collector (let's think of MOtel):

• We have the receiver running and get data on how much the tagged hosted-otel-collector resources use.
• We send this to elasticsearch exporter.
• We could optionally then configure an alert on the Elasticsearch side.

Alternatives: There is another option to AWS Cost Explorer. We could use the AWS Pricing API. This is just the pricing of the costs, so for services that require payment according to the usage, we would have to get the usage metrics through cloudwatch metrics (usage * price per usage). This way, the delay wouldn't be so long (AWS Cost explorer is up to 24h delay), but this is much more work to build. Then it would also have no savings recommendation, I believe that's unique to the cost explorer.

What about the current AWS Cloudwatch metrics?: For AWSCloudwatch to get estimated costs, it requires: first to set up billing alarms, can only split by service, no recommendation for savings possible. So it is much more limited.

What about adding this as a functionality that is part of the ECF? I can see a few issues with this:

• Big con: It would force every person that wants to use the cost receiver, to also deploy ECF. ECF might have functionalities that don't matter at all for this use case.
• We would have to manually configure the report, I think.
• There is no reason to deploy in AWS lambda, if we can authenticate through AWS SDK. Then we can deploy this receiver wherever we want.

How could we then use this with ECF if we wanted?
I think we could do it this way:

flowchart LR
    subgraph "Cost Management Pipeline"
        A[Cost Receiver] --> B[Processor] --> C[S3 Exporter]
    end

    C -. "put object notifies" .-> D[ECF Receiver]

    subgraph "Event Processing Pipeline"
        direction TB
        D --> E[Elasticsearch Exporter]
    end

Loading

[smith]

Related: opencost/opencost#3046

[axw]

What about adding this as a functionality that is part of the ECF? I can see a few issues with this:

Big con: It would force every person that wants to use the cost receiver, to also deploy ECF. ECF might have functionalities that don't matter at all for this use case.

I don't understand this con. How is that different to the proposal to make it a receiver? The receiver would be one of many in an OTel Collector - so the same thing applies doesn't it?

We would have to manually configure the report, I think.

You can create a Data Export using with CloudFormation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-bcmdataexports-export.html. When you define a Data Export, you get to specify the columns you're interested in - see the data dictionary at https://docs.aws.amazon.com/cur/latest/userguide/dataexports-table-dictionary.html

There is no reason to deploy in AWS lambda, if we can authenticate through AWS SDK. Then we can deploy this receiver wherever we want.

Both can be true: you might want to run a long-lived OTel Collector, or you might want to have it triggered by an S3 object.

[constanca-m]

Big con: It would force every person that wants to use the cost receiver, to also deploy ECF. ECF might have functionalities that don't matter at all for this use case.

I don't understand this con. How is that different to the proposal to make it a receiver? The receiver would be one of many in an OTel Collector - so the same thing applies doesn't it?

My issue with this is that cloud exporter features would be coupled with the lambda receiver (I will be calling it lambda receiver for a bit, I think it's a bit more clear in this context). I don't see advantages in this approach, unfortunately. Why would we be doing this: cost explorer -> s3 -> lambda receiver -> exporter, when we could just as well do cost explorer -> exporter. The only reason I see it is for error handler, but even then...

Then we are also forced to deploy this in AWS lambda. What if the user does not want that? We can deploy collectors in other resources as well: EC2, ECS, EKS... So why add such a limitation from the start?

You can create a Data Export using with CloudFormation

I don't see a reason why to have so much work. We can just call the API of the cost explorer.

Both can be true: you might want to run a long-lived OTel Collector, or you might want to have it triggered by an S3 object.

But in that case, don't we fall in the case covered in the description? We have a collector that has the cost receiver that sends events to S3 bucket, and this S3 bucket triggers lambda.

---

I think if it is so important for you that ECF handles this functionality, we could think of having this as an extension instead of a receiver. The extension could start and launch a go routine that runs every X hours. But I don't know either if this makes much sense... @axw

Edit: one of the things I like the most about open telemetry components is how decoupled they are. I think we should keep taking advantage of that 😄

[axw]

Then we are also forced to deploy this in AWS lambda. What if the user does not want that?

Nobody is forced to do that. You can alternatively use the S3 receiver. The only "limitation" is that the reports must be exported to S3. As I'll describe below, this isn't actually necessarily a limitation at all.

You can create a Data Export using with CloudFormation

I don't see a reason why to have so much work. We can just call the API of the cost explorer.

I'd like to know why you think this is a lot of work. You can have a CloudFormation template that sets everything up, just like you can have a collector config that does everything.

Beside that, think about it from the other way: many bigger users will already be using the Data Export feature, i.e. they will already have their reports in S3. If the only way that they can get the cost data into Elasticsearch is by configuring a receiver then we're forcing them to duplicate things.

Data Exports also has support for FOCUS 1.0, which aims to be an industry standard for cloud billing: https://aws.amazon.com/blogs/aws-cloud-financial-management/data-exports-for-focus-1-0-is-now-generally-available/. So if we have support for transforming this into OTLP metrics (e.g. imagine "focusmetricsencodingextension"), we would be able to process these from other CSPs too.

I don't see a reason why to have so much work. We can just call the API of the cost explorer.

It's not "just" calling the API of the cost explorer. It also implies you're now on the hook for running a collector, and making sure it keeps running, because if you don't you're going to stop ingesting cost metrics.

But in that case, don't we fall in the case covered in the description? We have a collector that has the cost receiver that sends events to S3 bucket, and this S3 bucket triggers lambda.

I don't follow. In which collector is the cost receiver running? To be clear, this is what I have in mind:

flowchart LR
    subgraph "Cost Management Pipeline"
        A[Data Export] --> B[S3]
    end

   B -. "put object notifies" .-> C[ECF Receiver]

    subgraph "Event Processing Pipeline"
        direction TB
        C --> D[Elasticsearch Exporter]
    end

Loading

I think if it is so important for you that ECF handles this functionality, we could think of having this as an extension instead of a receiver. The extension could start and launch a go routine that runs every X hours. But I don't know either if this makes much sense...

I don't think it does, Lambdas are event-driven. You could run on a schedule with EventBridge, but I don't think that's any simpler.

[constanca-m]

Then we are also forced to deploy this in AWS lambda.

Nobody is forced to do that.

What do you mean? By ECF you mean the lambda receiver, right? Why would someone deploy the lambda receiver other than in the AWS lambda?

[axw]

@constanca-m I'm not saying it should only be in ECF. We can add an encoding extension in opentelemetry-collector-contrib that can be used with either ECF, or with the S3 receiver.

[constanca-m]

Ok, I will try to take a look at both pipelines today or on the upcoming days. Then I think it will be more clear how this should be done. Another question before I do that: do you prefer cloud formation to terraform? I know cloudformation is from AWS.

[axw]

I prefer CloudFormation for AWS because it's tightly integrated with the AWS CLI and console, and enables https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-create-stacks-quick-create-links.html

We can always wrap it with Terraform.

[constanca-m]

It's not "just" calling the API of the cost explorer. It also implies you're now on the hook for running a collector, and making sure it keeps running, because if you don't you're going to stop ingesting cost metrics.
[..]
I'm not saying it should only be in ECF. We can add an encoding extension in opentelemetry-collector-contrib that can be used with either ECF, or with the S3 receiver.

I understand this now. I thought you wanted to be using AWS SDK on ECF, because that was what I was assuming for the cost explorer receiver. So now what we have to do is:

1. Create cloud formation to configure data exports. Send the export to S3, with the options:

2. Create the encoding format for this (this will take some time, I see we can have 113 columns).

3. Then this encoding extension can be used in any component that supports the receiver.

[axw]

@constanca-m yes, that's right. If/when we do this, we should aim to implement an encoding extension that can parse the FOCUS 1.0 schema. You would set this up in a Data Export like this:

By parsing FOCUS 1.0, we could then ingest billing data for GCP, Azure, Oracle Cloud, and Tencent -- see https://focus.finops.org/get-started/