Fail to deploy KEDA - read udp x.x.x.x:46084->y.y.y.y:53: i/o timeout on MacOS (Apple Silicon)

[mhcastro]

General information

• OS: macOS (Apple Silicon)
• Did you run crc setup before starting it? Yes.
• Running CRC on: Laptop

CRC version

CRC version: 2.9.0+589ab2cd
OpenShift version: 4.11.3
Podman version: 4.2.0

CRC status

DEBU CRC version: 2.9.0+589ab2cd
DEBU OpenShift version: 4.11.3
DEBU Podman version: 4.2.0
DEBU Running 'crc status'
DEBU Checking file: /Users/myuser/.crc/machines/crc/.crc-exist
DEBU Checking file: /Users/myuser/.crc/machines/crc/.crc-exist
DEBU Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1
DEBU Using ssh private keys: [/Users/myuser/.crc/machines/crc/id_ecdsa /Users/myuser/.crc/cache/crc_vfkit_4.11.3_arm64/id_ecdsa_crc]
DEBU SSH command results: err: <nil>, output: 32737570816 13706608640 /sysroot
CRC VM:          Running
OpenShift:       Running (v4.11.3)
Podman:
Disk Usage:      13.71GB of 32.74GB (Inside the CRC VM)
Cache Usage:     36.85GB
Cache Directory: /Users/myuser/.crc/cache

CRC config

- consent-telemetry                     : no

Host Operating System

ProductName:	macOS
ProductVersion:	12.6
BuildVersion:	21G115

Steps to reproduce

1. Deployed KEDA using helm, per documendation: helm install keda kedacore/keda --namespace keda
2. Containers going into ErrImagePull.

keda-operator-metrics-apiserver-75d6895d6c-75j5r   0/1     ErrImagePull        0          33s
keda-operator-698598c4d-74k9j                      0/1     ErrImagePull        0          33s

3. Getting the following error:

Events:
Type     Reason          Age   From               Message
----     ------          ----  ----               -------
Normal   Scheduled       44s   default-scheduler  Successfully assigned keda/keda-operator-metrics-apiserver-75d6895d6c-75j5r to crc-j2d48-master-0 by crc-j2d48-master-0
Normal   AddedInterface  41s   multus             Add eth0 [z.z.z.z/23] from openshift-sdn
Normal   Pulling         41s   kubelet            Pulling image "ghcr.io/kedacore/keda-metrics-apiserver:2.8.1"
Warning  Failed          11s   kubelet            Failed to pull image "ghcr.io/kedacore/keda-metrics-apiserver:2.8.1": rpc error: code = Unknown desc = pinging container registry ghcr.io: Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io on x.x.x.x:53: read udp y.y.y.y:46084->x.x.x.x:53: i/o timeout
Warning  Failed          11s   kubelet            Error: ErrImagePull
Normal   BackOff         11s   kubelet            Back-off pulling image "ghcr.io/kedacore/keda-metrics-apiserver:2.8.1"
Warning  Failed          11s   kubelet            Error: ImagePullBackOff

4. Direct image pull using Podman works fine: podman pull ghcr.io/kedacore/keda-metrics-apiserver:2.8.1

There is an issue in Linux (Issue #253) that sounds similar, but it is in Federa and the resolution may be different.

[praveenkumar]

Direct image pull using Podman works fine: podman pull ghcr.io/kedacore/keda-metrics-apiserver:2.8.1

You mean it works inside the VM? You can ssh to the VM using https://github.com/code-ready/crc/wiki/Debugging-guide document.

[mhcastro]

@praveenkumar I mean that I can successfully pull the image directly through Podman, outside CRC. But in CRC, it is failing to pull the image with that dial tcp: look up .... i/o timeout issue.

[mhcastro]

This issue is not only for Keda. It is the same even for the simple NGINX deployment.

oc apply -f https://k8s.io/examples/application/deployment.yaml

Same error to pull the image on MacOS (Apple M1 Max chip).

It must be something in OpenShift CRC as it works in minikube.

[praveenkumar]

@mhcastro I tried same but didn't have issue pulling the image, have you tried following?

$ crc delete
$ crc cleanup
$ ps aux | grep crc <= make sure there is no crc daemon process
$ crc setup
$ crc start

[mhcastro]

Yes, I have tried that. It works in minikube, but not in CRC.

[praveenkumar]

@mhcastro Are you trying to run CRC and minikube in parallel?

[mhcastro]

@praveenkumar - no, I am not.

[mhcastro]

Hi - I noticed that the error message is issued by ../crc/vendor/github.com/containers/image/v5/docker/docker_client.go, line 759, but I am using podman, not docker. Docker is not even installed. Is there a configuration issue where I need to explicitly configure CRC to use podman instead of docker?

[praveenkumar]

@mhcastro no, this is just the file name which have logic around pulling the image, I am still not able to reproduce it :( , check #2597 one, looks similar.

[mhcastro]

@praveenkumar This seems to be a known, legitimate and recurrent issue, which is also referred by #3373. The trick seems to be in the ability to reproduce.

[mhcastro]

This may be relevant info ....

{
  "ARP": {
    "DisabledPacketsReceived": 0,
    "MalformedPacketsReceived": 0,
    "OutgoingRepliesDropped": 0,
    "OutgoingRepliesSent": 0,
    "OutgoingRequestBadLocalAddressErrors": 0,
    "OutgoingRequestInterfaceHasNoLocalAddressErrors": 0,
    "OutgoingRequestsDropped": 0,
    "OutgoingRequestsSent": 7,
    "PacketsReceived": 1,
    "RepliesReceived": 1,
    "RequestsReceived": 0,
    "RequestsReceivedUnknownTargetAddress": 0
  },
  "BytesReceived": 78582484,
  "BytesSent": 7419755,
  "DroppedPackets": 0,
  "ICMP": {
    "V4": {
      "PacketsReceived": {
        "ICMPv4PacketStats": {
          "DstUnreachable": 569,
          "EchoReply": 0,
          "EchoRequest": 0,
          "InfoReply": 0,
          "InfoRequest": 0,
          "ParamProblem": 0,
          "Redirect": 0,
          "SrcQuench": 0,
          "TimeExceeded": 0,
          "Timestamp": 0,
          "TimestampReply": 0
        },
        "Invalid": 0
      },
      "PacketsSent": {
        "Dropped": 0,
        "ICMPv4PacketStats": {
          "DstUnreachable": 0,
          "EchoReply": 0,
          "EchoRequest": 0,
          "InfoReply": 0,
          "InfoRequest": 0,
          "ParamProblem": 0,
          "Redirect": 0,
          "SrcQuench": 0,
          "TimeExceeded": 0,
          "Timestamp": 0,
          "TimestampReply": 0
        },
        "RateLimited": 0
      }
    },
    "V6": {
      "PacketsReceived": {
        "ICMPv6PacketStats": {
          "DstUnreachable": 0,
          "EchoReply": 0,
          "EchoRequest": 0,
          "MulticastListenerDone": 0,
          "MulticastListenerQuery": 0,
          "MulticastListenerReport": 0,
          "NeighborAdvert": 0,
          "NeighborSolicit": 0,
          "PacketTooBig": 0,
          "ParamProblem": 0,
          "RedirectMsg": 0,
          "RouterAdvert": 0,
          "RouterSolicit": 0,
          "TimeExceeded": 0
        },
        "Invalid": 0,
        "RouterOnlyPacketsDroppedByHost": 0,
        "Unrecognized": 0
      },
      "PacketsSent": {
        "Dropped": 0,
        "ICMPv6PacketStats": {
          "DstUnreachable": 0,
          "EchoReply": 0,
          "EchoRequest": 0,
          "MulticastListenerDone": 0,
          "MulticastListenerQuery": 0,
          "MulticastListenerReport": 0,
          "NeighborAdvert": 0,
          "NeighborSolicit": 0,
          "PacketTooBig": 0,
          "ParamProblem": 0,
          "RedirectMsg": 0,
          "RouterAdvert": 0,
          "RouterSolicit": 0,
          "TimeExceeded": 0
        },
        "RateLimited": 0
      }
    }
  },
  "IGMP": {
    "PacketsReceived": {
      "ChecksumErrors": 0,
      "IGMPPacketStats": {
        "LeaveGroup": 0,
        "MembershipQuery": 0,
        "V1MembershipReport": 0,
        "V2MembershipReport": 0
      },
      "Invalid": 0,
      "Unrecognized": 0
    },
    "PacketsSent": {
      "Dropped": 0,
      "IGMPPacketStats": {
        "LeaveGroup": 0,
        "MembershipQuery": 0,
        "V1MembershipReport": 0,
        "V2MembershipReport": 0
      }
    }
  },
  "IP": {
    "DisabledPacketsReceived": 0,
    "Forwarding": {
      "Errors": 0,
      "ExhaustedTTL": 0,
      "ExtensionHeaderProblem": 0,
      "HostUnreachable": 0,
      "LinkLocalDestination": 0,
      "LinkLocalSource": 0,
      "NoMulticastPendingQueueBufferSpace": 0,
      "PacketTooBig": 0,
      "UnexpectedMulticastInputInterface": 0,
      "UnknownOutputEndpoint": 0,
      "Unrouteable": 0
    },
    "IPTablesForwardDropped": 0,
    "IPTablesInputDropped": 0,
    "IPTablesOutputDropped": 0,
    "IPTablesPostroutingDropped": 0,
    "IPTablesPreroutingDropped": 0,
    "InvalidDestinationAddressesReceived": 0,
    "InvalidSourceAddressesReceived": 0,
    "MalformedFragmentsReceived": 0,
    "MalformedPacketsReceived": 0,
    "OptionRecordRouteReceived": 0,
    "OptionRouterAlertReceived": 0,
    "OptionTimestampReceived": 0,
    "OptionUnknownReceived": 0,
    "OutgoingPacketErrors": 9,
    "PacketsDelivered": 52301,
    "PacketsReceived": 52301,
    "PacketsSent": 38271,
    "ValidPacketsReceived": 52301
  },
  "NICs": {
    "DisabledRx": {
      "Bytes": 0,
      "Packets": 0
    },
    "MalformedL4RcvdPackets": 0,
    "Neighbor": {
      "UnreachableEntryLookups": 2
    },
    "Rx": {
      "Bytes": 77849358,
      "Packets": 52300
    },
    "Tx": {
      "Bytes": 7420007,
      "Packets": 38269
    },
    "TxPacketsDroppedNoBufferSpace": 0
  },
  "TCP": {
    "ActiveConnectionOpenings": 1069,
    "ChecksumErrors": 0,
    "CurrentConnected": 47,
    "CurrentEstablished": 46,
    "EstablishedClosed": 302,
    "EstablishedResets": 18,
    "EstablishedTimedout": 0,
    "FailedConnectionAttempts": 4,
    "FailedPortReservations": 0,
    "FastRecovery": 0,
    "FastRetransmit": 0,
    "InvalidSegmentsReceived": 0,
    "ListenOverflowAckDrop": 0,
    "ListenOverflowInvalidSynCookieRcvd": 0,
    "ListenOverflowSynCookieRcvd": 0,
    "ListenOverflowSynCookieSent": 0,
    "ListenOverflowSynDrop": 0,
    "PassiveConnectionOpenings": 1,
    "ResetsReceived": 13,
    "ResetsSent": 2752,
    "Retransmits": 0,
    "SACKRecovery": 0,
    "SegmentSendErrors": 0,
    "SegmentsAckedWithDSACK": 0,
    "SegmentsSent": 34569,
    "SlowStartRetransmits": 0,
    "SpuriousRTORecovery": 0,
    "SpuriousRecovery": 0,
    "TLPRecovery": 0,
    "Timeouts": 0,
    "ValidSegmentsReceived": 47197
  },
  "UDP": {
    "ChecksumErrors": 0,
    "MalformedPacketsReceived": 0,
    "PacketSendErrors": 0,
    "PacketsReceived": 3711,
    "PacketsSent": 3702,
    "ReceiveBufferErrors": 0,
    "UnknownPortErrors": 118
  }
}

[mhcastro]

@praveenkumar perhaps one additional piece of information. I didn't have issues with CRC in MacOS using Intel-chip. I started having the problems only in the Apple M1 Max silicon.