Allow KafkaRoller connect to controllers

[tinaselenge]

Type of change

Select the type of your PR

• Enhancement / new feature

Description

• Add a method for creating admin client for controllers with BOOTSTRAP_CONTROLLERS_CONFIG set.
• Make KafkaRoller create admin client against controllers nodes before attempting to roll a controller node
• Tidy up: remove ceShouldBeFatal option as it's never set to true.

There will be a follow up PR to dynamically apply configurations for controllers using the admin client instead of how we currently restart controllers when any config changes.

Closes #9692

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Reference relevant issue(s) and close them after merging
• Update CHANGELOG.md
• Supply screenshots for visual changes, such as Grafana dashboards

[scholzj]

I think this looks good. How risky do you think it is? Can we merge it shortly before new release if it passes the other reviews as well?

[scholzj]

/gha run pipeline=regression,upgrade

[github-actions]

⏳ System test verification started: link

[scholzj]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[scholzj]

/azp run upgrade

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[github-actions]

❌ System test verification failed: link

[github-actions]

🎉 System test verification passed: link

[tinaselenge]

I think this looks good. How risky do you think it is? Can we merge it shortly before new release if it passes the other reviews as well?

Given that I had to revert this change once already (due to an AdminClient bug that is fixed now), and resulted in delaying the release , I am happy for it to get merged after the release :D