[Feature Request] Introduce OpenStackClusterIdentity Resource for Centralized Credential Management

[bnallapeta]

/kind feature

Describe the solution you'd like

The current implementation of identityRef in OpenStackCluster requires secrets containing credentials (e.g., clouds.yaml) to reside in the same namespace as the OpenStackCluster resource. This limitation poses challenges in environments with namespace-based access control, where sensitive credentials need to remain centralized and secure while still being usable by resources in other namespaces.

To address this, I propose introducing an OpenStackClusterIdentity custom resource, similar to AWSClusterStaticIdentity or AzureClusterIdentity. This resource would allow for centralized credential management while providing fine-grained access control across namespaces.

Proposed Implementation:
1. New CRD: Create OpenStackClusterIdentity to reference credentials stored in a secret, allowing usage across namespaces with access controls.
2. Identity Reference: Update OpenStackCluster.spec.identityRef to reference the new resource instead of secrets directly.
3. Access Control: Implement validation in the CAPO controller to restrict usage based on namespaces or roles.

[mdbooth]

Even implementing the ability to reference cross-namespace credentials raises architectural issues in CAPO akin to requiring a setuid bit in a posix application. Despite what anybody else may have done, my instinct is to actively avoid doing this. It requires us to robustly implement and enforce our own RBAC scheme. I'm not sure we're up for that.

IOW, far from protecting secrets, I believe this feature would open a potential hole where a bug might expose any secret in the cluster.

However, I'm very interested in the use case. Can you explain why the credentials cannot live in the target namespace alongside the CRs which reference them?

[s3rj1k]

@mdbooth Can you please elaborate on "CAPO akin to requiring a setuid bit in a posix application"?

Assuming that I understand use case correctly, to have what other major CAPI providers have, to have similar API (both AWS and Azure have this).

[wkonitzer]

Hi @mdbooth Your points about the potential complexity of enforcing a robust RBAC scheme and the risk of inadvertently exposing secrets are well taken.

The use case for cross-namespace credentials stems from the need to centralize credential management, especially in environments with a high number of clusters and frequent credential rotations. Here are a few key reasons why having credentials live in a single namespace is advantageous:

• Centralized Management: Storing credentials in one namespace allows for better oversight and governance. For organizations managing dozens or hundreds of clusters, centralization ensures consistency, simplifies rotation, and reduces the likelihood of credential sprawl.
• Improved Security: While I understand your concern about enforcing RBAC, centralizing credentials reduces the overall surface area of sensitive data. Rather than duplicating secrets across namespaces, centralization allows tighter access control and audit capabilities.
• Operational Efficiency: Rotating credentials across multiple namespaces adds administrative overhead and increases the likelihood of human error, which could lead to downtime or configuration drift. A single source of truth for credentials ensures that changes propagate seamlessly without manual intervention.

I hope this helps clarify the rationale behind the feature request.

[lentzi90]

Thank you for the feature request!
I can see the usefulness of this and also how it is more secure than direct cross namespace references to secrets.
I do share @mdbooth 's concern about bugs in the code though. This would add to CAPO's responsibilities so we do need to be careful.

[mdbooth]

For posterity I'll summarise the behaviour of AWSClusterStaticIdentity.

AWSClusterStaticIdentity is a cluster-scoped, non-namespaced custom resource. I assume that users are therefore not expected to create them. It looks like users are not even strictly required to be able to read them, merely know the name of one which will permit their access. They are presumably expected to be created and managed by a service provider.

AWSClusterStaticIdentity includes allowedNamespaces in its spec.

Credentials are stored in a secret in the same namespace the controller is running in.

CAPA has a method isClusterPermittedToUsePrincipal which is responsible for enforcing that only clusters in an allowed namespace may use the credentials: https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/4afd74ff9f41ef714af378adabe0f5e4fa66723e/pkg/cloud/scope/session.go#L418-L459

If this method returns true, buildAWSClusterStaticIdentity will return a AWSStaticPrincipalTypeProvider containing the credentials from the secret.

I can see how this addresses the issue of centrally-managed namespaces.

At a minimum, in this context the methods buildAWSClusterStaticIdentity and isClusterPermittedToUsePrincipal are both 'security critical', meaning that an error here could allow a tenant to use another tenant's credentials.

[mdbooth]

Have you considered using a tool which automatically copies secrets from a central namespace to target namespace, and updates them automatically in line with the source credentials? That solves the maintenance problem while greatly reducing the potential for security bugs in code.

But I guess you don't want to expose the OpenStack credentials to your users at all? What's the thinking here?

[s3rj1k]

Have you considered using a tool which automatically copies secrets from a central namespace to target namespace, and updates them automatically in line with the source credentials

This is what I was mentioning in Slack, this is exactly what I would personally like to avoid, given that ClusterIdentity is common practice in CAPI

[bnallapeta]

@mdbooth Yes, we want to avoid this as this exposes the cluster secret to all the other namespaces where we would want to provision a cluster.

[bnallapeta]

@mdbooth so, based on the conversation here and on Slack, are you now convinced that ClusterIdentity should be the way to address this? Please let me know. I am happy to take it up and work on it.

We could discuss further if we reach a conclusion here.

[richardcase]

Yes, we want to avoid this as this exposes the cluster secret to all the other namespaces where we would want to provision a cluster.

We have this functionality in CAPA (i am a maintainer) and this is a valid concern. In CAPA on our equivalent to this (AWSClusterIdentitySpec) we have the ability to lock down using the identity from specific namespaces via a AllowedNamespaces field. This could be a good way around this concern.

[mdbooth]

Yes, we want to avoid this as this exposes the cluster secret to all the other namespaces where we would want to provision a cluster.

@bnallapeta Can you explain this further? As I mentioned before we have a custom solution to this in OpenShift, but looking around it seems to have been solved a few times. Here's an example that looks very much like the sort of thing I'd expect to use for this: https://medium.com/lonto-digital-services-integrator/why-we-developed-own-kubernetes-controller-to-copy-secrets-e46368ae6db9

In this model:

• Credentials are centrally managed
• Credentials are copied only to the namespaces which need them
• Updating a centrally managed credential updates it everywhere
• Deleting a centrally managed credential deletes it everywhere

It also requires no code in CAPO which is directly responsible for mediating access to Secrets.

I also have to consider that we would need to add support for cross-namespace Secret references to ORC. I was hoping to be able to switch ORC, and maybe even CAPO, to a per-namespace controller model, which would have the benefit of load sharding in addition to greatly reduced RBAC requirements for each controller. This is a secondary consideration, but I bring it up because it's on my mind.

I want to be 100% clear that I'm not rejecting this proposal¹, btw. AWSClusterStaticIdentity is a much safer model than 'bare' cross-namespace secret references. I especially like the control that users (presumably) can't create them. Unlike cross-namespace references, I can see a path to personally approving of this, especially as it seems to be an established pattern. However, it would represent a significant change in secure coding requirements in CAPO, and I hope we would not choose to do that without a complete understanding of why we rejected the alternatives. In particular, I would very much like to understand the concerns around using a secret controller. I don't yet understand this.

Footnotes

1. For what that would be worth. There are 2 other maintainers who are welcome to disagree with me! ↩