Cannot connect to Kafka via kcat

[Khodesaeed]

Hi folks, I have deployed a Kafka cluster with the help of the Strimzi operator, but when I want to check the connectivity with the kcat binary, I get an error of Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512.

I use a test pod in the same namespace and run this command:

 kcat -b sentry-kafka-cluster-kafka-bootstrap.sentry-test-system.svc:9092 -L \
-X security.protocol=SASL_PLAINTEXT \
-X sasl.mechanisms=SCRAM-SHA-512 \
-X sasl.username="sentry-user" \
-X sasl.password="q5Gno2OyRmYvTvfCR2xkasqyDcGfXFqA"

output log:

%3|1755699718.439|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://sentry-kafka-cluster-kafka-bootstrap.sentry-te]: sasl_plaintext://sentry-kafka-cluster-kafka-bootstrap.sentry-test-system.svc:9092/bootstrap: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 (after 313ms in state AUTH_REQ)
%3|1755699719.026|FAIL|rdkafka#producer-1| [thrd:sasl_plaintext://sentry-kafka-cluster-kafka-bootstrap.sentry-te]: sasl_plaintext://sentry-kafka-cluster-kafka-bootstrap.sentry-test-system.svc:9092/bootstrap: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 (after 318ms in state AUTH_REQ, 1 identical error(s) suppressed)
% ERROR: Failed to acquire metadata: Local: Broker transport failure (Are the brokers reachable? Also try increasing the metadata timeout with -m <timeout>?)

Here is the Kafka cluster configuration:

metadata:
  name: sentry-kafka-cluster
  namespace: sentry-test-system
spec:
  cruiseControl:
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: metrics-config.yml
          name: sentry-kafka-cluster-cruise-control-metrics
  entityOperator:
    topicOperator: {}
    userOperator: {}
  kafka:
    authorization:
      type: simple
    config:
      connections.max.idle.ms: '600000'
      default.replication.factor: '3'
      log.retention.bytes: '209715200'
      log.retention.check.interval.ms: '900000'
      log.segment.bytes: '207374182'
      log.segment.ms: '604800000'
      message.max.bytes: '10485760'
      min.insync.replicas: '2'
      num.io.threads: '6'
      num.network.threads: '6'
      offsets.topic.replication.factor: '3'
      replica.lag.time.max.ms: '10000'
      socket.receive.buffer.bytes: '2097152'
      socket.send.buffer.bytes: '2097152'
      transaction.state.log.min.isr: '2'
      transaction.state.log.replication.factor: '3'
    listeners:
      - authentication:
          type: scram-sha-512
        name: plain
        port: 9092
        tls: false
        type: internal
      - authentication:
          type: scram-sha-512
        name: internal
        port: 9093
        tls: true
        type: internal
    livenessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 20
    metadataVersion: '4.0'
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          key: kafka-metrics-config.yml
          name: sentry-kafka-cluster-metrics
    readinessProbe:
      initialDelaySeconds: 15
      timeoutSeconds: 20
    resources:
      limits:
        cpu: 6000m
        memory: 12Gi
      requests:
        cpu: 200m
        memory: 2Gi
    template:
      pod: {}
    version: 4.0.0
  kafkaExporter:
    resources:
      limits:
        cpu: 500m
        memory: 128Mi
      requests:
        cpu: 10m
        memory: 64Mi

And here is my KafkaUser.spec:

spec:
  authentication:
    type: scram-sha-512
  authorization:
    acls:
      - operation: All
        resource:
          name: '*'
          patternType: literal
          type: topic
      - operation: All
        resource:
          name: '*'
          patternType: literal
          type: group
    type: simple

[im-konge]

Hey,
did you copied the password correctly from the KafkaUser's Secret? Was it decoded? Do you have sentry-user created?
You shared YAMLs with removed metadata part. Should it be just mechanism instead of mechanisms inside sasl.mechanisms=SCRAM-SHA-512 ? (I never worked with kcat, so that's why I'm asking)