BUG: `BROKER_URL` is not set in Sentry configmap if `externalRedis.existingSecret` or `redis.auth.existingSecret` is used

[jiriks74]

Issue submitter TODO list

• I've searched for an already existing issues here

Describe the bug (actual behavior)

I wanted to migrate from the chart provided Redis instance to my own external Redis.

redis.enable is set to false and the externalRedis.host, externalRedis.port (default in values), externalRedis.existingSecret, externalRedis.existingSecretKey, externalRedis.dp externalRedis.ssl options are set to correct values.

Deployments connecting to 127.0.0.1 (from logs):

• sentry-post-process-forward-errors
• sentry-ingest-consumer-events
• sentry-cron
• sentry-web
• sentry-worker
• sentry-worker-events
• sentry-worker-transactions

All of the above try to connect to 127.0.0.1 even though all the settings tell them not to - see Additional context.

After installing dnsutils in the pod (I set the command to sleep for debugging purposes) I checked that the hostname is resolved correctly which it is. Connecting and authenticating with the externalRedis host using telnet works without issues so it something with the workers and probably just those 2.

Expected behavior

The consumers connecting to the externalRedis.host host.

values.yaml

values.yaml

  ingestConsumerEvents:
    enabled: true
    replicas: 1
    # concurrency: 4
    # env: []
    resources: {}
    #  requests:
    #    cpu: 300m
    #    memory: 500Mi
    affinity: {}
    nodeSelector: {}
    securityContext: {}
    containerSecurityContext: {}
    # tolerations: []
    # podLabels: {}
    # maxBatchSize: ""
    # logLevel: "info"
    # inputBlockSize: ""
    # maxBatchTimeMs: ""

    # it's better to use prometheus adapter and scale based on
    # the size of the rabbitmq queue
    autoscaling:
      enabled: false
      minReplicas: 1
      maxReplicas: 3
      targetCPUUtilizationPercentage: 50
    sidecars: []
    topologySpreadConstraints: []
    # volumes: []
    livenessProbe:
      enabled: true
      initialDelaySeconds: 5
      periodSeconds: 320
    # volumeMounts: []
    autoOffsetReset: "earliest"
    # noStrictOffsetReset: false

externalRedis:
  ## Hostname or ip address of external redis cluster
  ##
  host: "external-sentry-redis"
  port: 6379
  ## Just omit the password field if your redis cluster doesn't use password
  # password: redis
  existingSecret: redis-secret
  ## set existingSecretKey if key name inside existingSecret is different from redis-password'
  existingSecretKey: REDIS_PASSWD
  ## Integer database number to use for redis (This is an integer)
  db: 0
  ## Use ssl for the connection to Redis (True/False)
  ssl: false

Helm chart version

26.22.0

Steps to reproduce

1. Have Sentry running with chart's Redis
2. Disable chart's Redis redis.enabled: false
3. Set host, secret, db and ssl under externalRedis
4. helm upgrade sentry sentry/sentry --version 26.22.0 --timeout 20m

Screenshots

No response

Logs

sentry.ingest.consumer.processors.Retriable: Error 111 connecting to 127.0.0.1:6379. Connection refused.

Additional context

I see the correct host in the Sentry ConfigMap is set correctly:

    SENTRY_OPTIONS["redis.clusters"] = {
      "default": {
        "hosts": {
          0: {
            "host": "external-sentry-redis",
            "password": os.environ.get("REDIS_PASSWORD", ""),
            "port": "6379",
            "db": "0"
          }
        }
      }
    }

The BROKER_URL ENV is set correctly as well:

redis://:$(HELM_CHARTS_SENTRY_REDIS_PASSWORD_CONTROLLED)@external-sentry-redis:6379/0

From what I can tell it's just ignoring the Redis settings.

[jiriks74]

Progress: I patched a file where I found the localhost URL it was using instead of the one provided.

sed -i 's|BROKER_URL = "redis://127.0.0.1:6379"|BROKER_URL = f"redis://:{os.getenv('REDIS_PASSWORD')}@external-sentry-redis:6379"|g'  /usr/src/sentry/src/sentry/conf/server.py

Now I get NameError: name 'REDIS_PASSWORD' is not defined. Seems like running under Python3 doesn't copy all environment variables.

[jiriks74]

When the password is directly written to the config file it works:

sed -i "s|BROKER_URL = \"redis://127.0.0.1:6379\"|BROKER_URL = f\"redis://:${REDIS_PASSWORD}@external-sentry-redis:6379\"|g" /usr/src/sentry/src/sentry/conf/server.py

But this requires manipulating the startup commands so it's not really a solution and I will not be using it to run external Redis.

EDIT: I was able to make Sentry work using this bodge in my testing environment, so it's just this stupid thing.

[jiriks74]

I maybe found the issue. The sentry.conf.py is missing these lines:

rabbitmq_host = None
if rabbitmq_host:
    BROKER_URL = "amqp://{username}:{password}@{host}/{vhost}".format(
        username="guest", password="guest", host=rabbitmq_host, vhost="/"
    )
else:
    BROKER_URL = "redis://:{password}@{host}:{port}/{db}".format(
        **SENTRY_OPTIONS["redis.clusters"]["default"]["hosts"][0]
    )

Adding

    BROKER_URL = "redis://:{password}@{host}:{port}/{db}".format(
        **SENTRY_OPTIONS["redis.clusters"]["default"]["hosts"][0]
    )

is sufficient to make Sentry work with Redis instead of RabbitMQ. As this seems to have fixed the issue for me I'll do some further tests and try to make a PR if everything works as it should.

[jiriks74]

So I'm thinking of 2 possible solutions.

1. Just do BROKER_URL = os.getenv("BROKER_URL") and leave everything else as normal
2. Rip out the BROKER_URL ENV and generate the settings directly into sentry.conf.py (basically ending up with what upstream has).

I'll probably go with the first one.

[jiriks74]

Found out why it works when using the chart's Redis but not the external one. It's because the template doesn't set the URL when Redis password is set using existingSecret.

charts/charts/sentry/templates/sentry/_helper-sentry.tpl

Lines 207 to 213 in 480a2a3

	{{- if or (.Values.rabbitmq.enabled) (.Values.rabbitmq.host) }}
	BROKER_URL = os.environ.get("BROKER_URL", "amqp://{{ .Values.rabbitmq.auth.username }}:{{ .Values.rabbitmq.auth.password }}@{{ template "sentry.rabbitmq.host" . }}:5672/{{ .Values.rabbitmq.vhost }}")
	{{- else if $redisPass }}
	BROKER_URL = os.environ.get("BROKER_URL", "{{ $redisProto }}://:{{ $redisPass }}@{{ $redisHost }}:{{ $redisPort }}/{{ $redisDb }}")
	{{- else if and (not .Values.externalRedis.existingSecret) (not .Values.redis.auth.existingSecret)}}
	BROKER_URL = os.environ.get("BROKER_URL", "{{ $redisProto }}://{{ $redisHost }}:{{ $redisPort }}/{{ $redisDb }}")
	{{- end }}