OKD 4.19 (4.19.0-okd-scos.6) cannot start new worker nodes in AWS region outside of us-east-1

[wzur-r3]

Describe the bug

I have a fresh installation of OKD 4.19 in AWS region eu-west-1 (Ireland).

After installation and I presume updating to 4.19.0-okd-scos.6 it cannot create new worker nodes.

MachineSet for worker nodes is set to use ami-0c4afb3332ab4602d, which does exist in us-east-1 (Virginia) AWS region, but not in Ireland.

The installer copied that image to our AWS region, and initial worker nodes have been created from the copy of that AMI.

On top of everything that problem sometimes happens at installation, and there is no worker nodes at all.

coreos stream returns AMI just for a single region:

openshift-install coreos print-stream-json | jq '.architectures.x86_64.images.aws'

{
  "regions": {
    "us-east-1": {
      "release": "9.0.20250510-0",
      "image": "ami-0c4afb3332ab4602d"
    },
    "us-gov-west-1": {
      "release": "9.0.20250510-0",
      "image": "ami-036058fe73aaa4263"
    }
  }
}

Version

4.19.0-okd-scos.6, IPI with custom VPC

Reproducibility

100% rate

Log Bundle

ClusterID: 274f61ce-e6f1-40ae-bad0-f00f071f8173
ClientVersion: 4.19.0-okd-scos.6
ClusterVersion: Stable at "4.19.0-okd-scos.6"
ClusterOperators:
All healthy and stable

[wzur-r3]

One more thing - openshift-installer complains about AWS region I'm in, and says only us-east-1 is supported.

[wzur-r3]

After gathering some more information I came across #2077 and specifically #2077 (comment) comment, which sheds more light into my problems.

As far as I understand, the AWS image used by OKD 4.19 is available in us-east-1 as per output of openshift-install coreos print-stream-json - hence installer complains about my regions (eu-west-1), but it is still unclear to me how it is supposed to work in this state outside us-east-1 region.

As I mentioned in my initial report, Openshift installer does make a copy of ami-0c4afb3332ab4602d image and uses it for the installation. Subsequently MachineSet uses a filter in aws field to find that new image, but after OKD installation is completed that is replaced by the machine-controller with a reference of ami-0c4afb3332ab4602d. After that any workers cannot be created as that AMI does not exist in my AWS region.

I'm still very new to Openshift - I cannot figure out how to convince that machine controller to use a different AMI. From the controller source code I saw it uses SCOS stream, but I'm not sure where it comes from and if I can modify it locally.

[chriskeAlerant]

Thanks for the detailed write-up—based on that I already knew where to look once you’d identified the cross-region issue. I ran into the same problem after an update. The solution is as follows:

Use a command like this with the AWS CLI to copy the AMI:

bash
Másolás
Szerkesztés
aws ec2 copy-image
--source-image-id ami-0c4afb3332ab4602d
--source-region us-east-1
--region eu-central-1
--name "SCOS"
--description "Copied from public AMI"

When it finishes, you’ll see the AMI appear under AMIs in your AWS account. Keep in mind it will get a new ID! Now all that remains is to set this in your OKD/OpenShift MachineSet. Unfortunately, with IPI-installed clusters we can’t modify the built-in MachineSets—but we can create a new one. If you create a new MachineSet modeled on the existing ones but with your new AMI ID, it will work. That’s how I resolved the issue. Thanks for confirming that the AMI exists in another region!

[diwilli]

I'm also seeing this on the 4.19.0-okd-scos.4 release. However uploading a new machine set (based on the built in worker one) seems to override the ami id, back to the one point at us-east-1.

Although rather than copying a new image, I was pointing to the one that was already copied in as the relevant ami used for the masters.

[wzur-r3]

@chriskeAlerant as far as I can see in relevant source code, the AMI id is always replaced with a value taken from the COREOS stream, and if a region is not listed in that stream, it takes the value for us-east-1 so I cannot see how your solution can work at all.

The OKD installer already made a copy of the AMI from us-east-1 region, and uses that copy to create master and worker nodes. Unfortunately as soon as these nodes are updated to the latest version, and to be more specific, machine-api is updated to the latest version, the mechanism I described above kicks in and it is not possible to create new nodes.

[chriskeAlerant]

@wzur-r3 you are correct, the original machineSets cannot be modified. But you can create new ones. Those will create machines with the exact AMI ID you provide.

[davidecelano]

Hi, this is a pretty big issue. I've installed on AWS eu-west-1

4.19.0-okd-scos.10
built from commit 249d7428c51fe85cabdab6394c4aed86fc24d398
release image quay.io/okd/scos-release@sha256:6e7ced1ae1293519afeddf563b9738eb03a139092a29cccfc673813910dfe9f9
release architecture multi
default architecture amd64

{
  "regions": {
    "us-east-1": {
      "release": "9.0.20250510-0",
      "image": "ami-0c4afb3332ab4602d"
    },
    "us-gov-west-1": {
      "release": "9.0.20250510-0",
      "image": "ami-036058fe73aaa4263"
    }
  }
}

and the default worker machineset doesn't come with any ami.id value, but only

      ami:
        filters:
          - name: 'tag:Name'
            values:
              - CLUSTERID-ami-eu-west-1

which exists, but when scaling up the worker nodes the process fails with this error

Generated from machineset_controller
failed to sync machines: admission webhook "validation.machine.machine.openshift.io" denied the request: providerSpec.ami: Required value: expected providerSpec.ami.id to be populated