Grafana Oncally Integration Dynamic Label creation error

[DownWithThisSortOfThing]

Terraform Version

1.11

Terraform Grafana Provider Version

3.25.2 -> 3.25.7

Grafana Version

Grafana Cloud IRM Version 0.37.2

Affected Resource(s)

No response

Terraform Configuration Files

resource "grafana_oncall_integration" "alertmanager" {
  name = "alertmanager"

  type    = "alertmanager"
  team_id = data.grafana_oncall_team.on_call.id

  dynamic_labels = [data.grafana_oncall_label.alertmanager_customer]

  templates {
    grouping_key = "{{ payload.customer }}"
  }

  default_route {
    escalation_chain_id = grafana_oncall_escalation_chain.default.id
  }
}

resource "grafana_team" "on_call_team" {
  name    = var.on_call_rotation.team_name
}

data "grafana_oncall_team" "on_call" {
  name = grafana_team.on_call_team.name
}

data "grafana_oncall_label" "alertmanager_customer" {
  key   = "customer"
  value = "{{ payload.get('customer', 'unknown') }}"
}

resource "grafana_oncall_escalation_chain" "default" {
  name = var.schedule_name

  team_id = data.grafana_oncall_team.on_call.id
}

Expected Behavior

The dynamic labels will be created in accordance with the plan (Example plan provided). ID's have been X'd out just to not leak information 😅

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.an_other_schedule.grafana_oncall_integration.alertmanager will be updated in-place
  ~ resource "grafana_oncall_integration" "alertmanager" {
      ~ dynamic_labels = [
          + {
              + "id"    = "customer"
              + "key"   = "customer"
              + "value" = "{{ payload.get('customer', 'unknown') }}"
            },
        ]
        id             = "XXXXXXXXX"
        name           = "pgt-cmp-schedule-alertmanager"
        # (4 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

  # module.schedule.grafana_oncall_integration.alertmanager will be updated in-place
  ~ resource "grafana_oncall_integration" "alertmanager" {
      ~ dynamic_labels = [
          + {
              + "id"    = "customer"
              + "key"   = "customer"
              + "value" = "{{ payload.get('customer', 'unknown') }}"
            },
        ]
        id             = "XXXXXXXX"
        name           = "pgt-cops-on-call-schedule-alertmanager"
        # (4 unchanged attributes hidden)

        # (2 unchanged blocks hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

Actual Behavior

The first one or two applies worked, but now a 500 is returned from the provider every time. Here is an excerpt from the TF_LOG=debug dump to show that. This was tested with provider version 3.25.2 -> 3.25.7 (latest)

2025-07-14T10:31:13.163+0200 [WARN]  Provider "registry.terraform.io/grafana/grafana" produced an invalid plan for module.schedule.grafana_oncall_integration.alertmanager, but we are toler
ating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .templates[0].resolve_signal: planned value cty.StringVal("") for a non-computed attribute
      - .templates[0].source_link: planned value cty.StringVal("") for a non-computed attribute
      - .templates[0].acknowledge_signal: planned value cty.StringVal("") for a non-computed attribute
2025-07-14T10:31:13.163+0200 [WARN]  Provider "registry.terraform.io/grafana/grafana" produced an invalid plan for module.an_other_schedule.grafana_oncall_integration.alertmanager, but we are t
olerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .templates[0].acknowledge_signal: planned value cty.StringVal("") for a non-computed attribute
      - .templates[0].resolve_signal: planned value cty.StringVal("") for a non-computed attribute
      - .templates[0].source_link: planned value cty.StringVal("") for a non-computed attribute
module.cmp_schedule.grafana_oncall_integration.alertmanager[0]: Modifying... [id=CNQJK834XDSFH]
module.schedule.grafana_oncall_integration.alertmanager[0]: Modifying... [id=C4GQDTZRZJ9DF]
2025-07-14T10:31:13.163+0200 [INFO]  Starting apply for module.cmp_schedule.grafana_oncall_integration.alertmanager[0]
2025-07-14T10:31:13.163+0200 [INFO]  Starting apply for module.schedule.grafana_oncall_integration.alertmanager[0]
2025-07-14T10:31:13.163+0200 [DEBUG] module.cmp_schedule.grafana_oncall_integration.alertmanager[0]: applying the planned Update change
2025-07-14T10:31:13.163+0200 [DEBUG] module.schedule.grafana_oncall_integration.alertmanager[0]: applying the planned Update change
2025-07-14T10:31:13.164+0200 [DEBUG] provider.terraform-provider-grafana_v3.25.2: 2025/07/14 10:31:13 [DEBUG] PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/C4GQDTZR
ZJ9DF/
2025-07-14T10:31:13.164+0200 [DEBUG] provider.terraform-provider-grafana_v3.25.2: 2025/07/14 10:31:13 [DEBUG] PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/CNQJK834
XDSFH/
2025-07-14T10:31:13.797+0200 [DEBUG] provider.terraform-provider-grafana_v3.25.2: 2025/07/14 10:31:13 [DEBUG] PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/CNQJK834
XDSFH/ (status: 500): retrying in 1s (4 left)
2025-07-14T10:31:13.831+0200 [DEBUG] provider.terraform-provider-grafana_v3.25.2: 2025/07/14 10:31:13 [DEBUG] PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/C4GQDTZR
ZJ9DF/ (status: 500): retrying in 1s (4 left)
2025-07-14T10:31:15.530+0200 [DEBUG] provider.terraform-provider-grafana_v3.25.2: 2025/07/14 10:31:15 [DEBUG] PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/C4GQDTZR
ZJ9DF/ (status: 500): retrying in 2s (3 left)
2025-07-14T10:31:15.530+0200 [DEBUG] provider.terraform-provider-grafana_v3.25.2: 2025/07/14 10:31:15 [DEBUG] PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/CNQJK834
XDSFH/ (status: 500): retrying in 2s (3 left)
2025-07-14T10:31:18.192+0200 [DEBUG] provider.terraform-provider-grafana_v3.25.2: 2025/07/14 10:31:18 [DEBUG] PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/C4GQDTZR
ZJ9DF/ (status: 500): retrying in 4s (2 left)
2025-07-14T10:31:18.192+0200 [DEBUG] provider.terraform-provider-grafana_v3.25.2: 2025/07/14 10:31:18 [DEBUG] PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/CNQJK834
XDSFH/ (status: 500): retrying in 4s (2 left)
2025-07-14T10:31:22.801+0200 [DEBUG] provider.terraform-provider-grafana_v3.25.2: 2025/07/14 10:31:22 [DEBUG] PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/CNQJK834
XDSFH/ (status: 500): retrying in 8s (1 left)
2025-07-14T10:31:22.801+0200 [DEBUG] provider.terraform-provider-grafana_v3.25.2: 2025/07/14 10:31:22 [DEBUG] PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/C4GQDTZR
ZJ9DF/ (status: 500): retrying in 8s (1 left)
module.cmp_schedule.grafana_oncall_integration.alertmanager[0]: Still modifying... [id=CNQJK834XDSFH, 00m10s elapsed]
module.schedule.grafana_oncall_integration.alertmanager[0]: Still modifying... [id=C4GQDTZRZJ9DF, 00m10s elapsed]
2025-07-14T10:31:31.508+0200 [ERROR] provider.terraform-provider-grafana_v3.25.2: Response contains error diagnostic: @caller=github.com/hashicorp/terraform-plugin-go@v0.26.0/tfprotov5/intern
al/diag/diagnostics.go:58 @module=sdk.proto diagnostic_detail="" tf_proto_version=5.8 tf_provider_addr=registry.terraform.io/grafana/grafana tf_resource_type=grafana_oncall_integration tf_rpc
=ApplyResourceChange diagnostic_severity=ERROR diagnostic_summary="PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/CNQJK834XDSFH/ giving up after 5 attempt(s)" tf_req
_id=045bf0c3-3c48-f78f-b2c7-248ddc7e1769 timestamp="2025-07-14T10:31:31.508+0200"
2025-07-14T10:31:31.509+0200 [ERROR] provider.terraform-provider-grafana_v3.25.2: Response contains error diagnostic: @module=sdk.proto diagnostic_detail="" diagnostic_severity=ERROR tf_provi
der_addr=registry.terraform.io/grafana/grafana tf_req_id=c7d92996-b79f-fabd-b4a6-5faf5dddad1f tf_resource_type=grafana_oncall_integration tf_rpc=ApplyResourceChange @caller=github.com/hashico
rp/terraform-plugin-go@v0.26.0/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_summary="PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/C4GQDTZRZJ9DF/ giving up
after 5 attempt(s)" tf_proto_version=5.8 timestamp="2025-07-14T10:31:31.509+0200"
2025-07-14T10:31:31.511+0200 [DEBUG] State storage *remote.State declined to persist a state snapshot
2025-07-14T10:31:31.511+0200 [ERROR] vertex "module.schedule.grafana_oncall_integration.alertmanager[0]" error: PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/C4GQDT
ZRZJ9DF/ giving up after 5 attempt(s)
2025-07-14T10:31:31.511+0200 [DEBUG] State storage *remote.State declined to persist a state snapshot
2025-07-14T10:31:31.511+0200 [ERROR] vertex "module.cmp_schedule.grafana_oncall_integration.alertmanager[0]" error: PUT https://oncall-prod-eu-west-0.grafana.net/oncall/api/v1/integrations/CN
QJK834XDSFH/ giving up after 5 attempt(s)
2025-07-14T10:31:31.512+0200 [DEBUG] states/remote: state read serial is: 133; serial is: 133
2025-07-14T10:31:31.512+0200 [DEBUG] states/remote: state read lineage is: 31c453e6-d0b8-7f78-b03c-a77802924135; lineage is: 31c453e6-d0b8-7f78-b03c-a77802924135

Steps to Reproduce

Run a terraform apply with the above configuration. It worked a couple of times but now consistently fails and gives 500 errors back

Important Factoids

It worked a couple of times (Saturday evening 12th July 2025) but from Sunday morning (13th July 2025) onwards it consistantly fails. If working 2 times in a row is all it can do 🤷 or if the dates are important I don't know, you get both.

References

No response