Determine scoping of Cluster registry types

[pmorie]

I realized today that despite our numerous discussions on the subject we don't have an issue dealing with what the scope of the cluster registry API should be.

Options:

• Only cluster-scoped
• Only namespace-scoped
• Both

/sig multicluster

[pmorie]

I believe that at a minimum, we should do a namespaced-scoped resource. We know there to be use-cases for namespacing with the cluster registry; example: a multitenant cluster where different tenants know and care about different clusters. This implies that we should make a namespace-scoped resource.

A common theme of discussion has been why we shouldn't make a cluster scoped resource and have everyone operate their own cluster registry (ie, run your own in your own namespace). I don't think that's a good solution because we do not have the any to support for tenant-by-tenant API aggregation or CRDs or ways to make kubectl usable in that scenario. One of the big value adds of API aggregation is that you can use a single client configuration to talk to multiple API groups. If I have an API aggregation, and then a new random API server that I can't use gracefully with the rest of my kube APIs, that outlier API isn't going to work nicely with the rest of the system. For example, do I have to plumb the coordinates of the API server into references from API groups that are aggregated?

To me, the question is whether we should also do a cluster-scoped resource. I don't personally think a cluster-scoped resource is absolutely necessary, but I don't want to prevent people from having it available to them if they want it. Cluster operators can make a choice about which API groups they allow users to use. Likewise, people who want to operate their own copies of the cluster registries if they choose to do so.

I would be in favor of doing both ns- and cluster- scoped in order to move forward on this subject. I don't think everyone needs to accept the same solution. We shouldn't imagine that doing so would be free of cost. Some factors to consider:

• We should ensure both scopes stay in sync
• If we build two different resources, we're effectively forcing APIs that work with the cluster registry to make a choice about whether to implement integrations with either or both resources

I will say that in another community project, the service-catalog, we made an early design choice to make certain resources cluster-scoped, and that seems to have been viewed as a great limitation due to issues around access control. We are currently working on adding ns-scoped versions of these resources (kubernetes-retired/service-catalog#1826 if people are interested). Namespaces are the fundamental unit of access control currently implemented in Kubernetes - we should use them. Our users want to.

Another theme that's been discussed several times is finding new ways to solve the problems of access control in order to provide more flexibility for cluster-scoped resources using ACL filtering. I'm supportive of developing better approaches to this problem in Kubernetes, but I don't think we should block on that here. For the record, ACL filtering on list/watch cannot be implemented in a consistent, ordered manner because:

• Not all authorizers have the ability to provide a list of resources a subject has access to
• Authorizers may have their state changed at any time out of band to kubernetes, making it impossible to implement ACL-filtered list/watch from a particular resource version

[pmorie]

We had a call to this discuss this today and arrived at a consensus to make the cluster resource namespace scoped.

As a next step, we will determine a conventional namespace to hold globally available Clusters that will have a permissive RBAC configuration for GET (one example of such a policy would be anyone in system:authenticated group could GET the cluster resource from this namespace).

@thockin mentioned that this would be the third such special namespace that he's aware of and that we should consider discussing the problem of 'special' namespaces in SIG architecture.

[perotinus]

Thanks for the summary, @pmorie!

To add a bit more color to the conventional/"special" namespace discussion, we discussed a few potential (not mutually exclusive) ways we can move forward:

• A single namespace that is defined to not be shared among control planes, that is commonly accepted as holding a “global” set of clusters, and that is never replicated to other control planes.
• A naming convention that specifies which namespaces are private to a specific control plane.
  • This could be a sig-multicluster or a Kubernetes-level convention.
• An annotation (that could potentially become a field) on the namespace object that specifies that the namespace is “special” and should be treated differently from other namespaces. As a strawman example, this could be an IsSystem annotation that multi-cluster tools understand to imply that the namespace should never be duplicated between clusters.

The addition of a special namespace is intended to address the concern that having cluster namespaces means that users who simply want to have a list of clusters, or controllers that want to read a list of clusters, won't know where to find that list of clusters. We believe that defining a conventional namespace in which they live is an acceptable alternative to a cluster-scoped object, and having only one object definition is better in the long run.

[pmorie]

So - now that this is determined, can we close the issue?

[perotinus]

Yes, I believe so.

Just to close the loop, the decision was to go with a namespaced cluster object and introduce a "global" namespace for cluster registry records that can be used in lieu of the cluster scoped objects.