Skip to content

OCPBUGS-42810: actively move bootstrap member lead #1369

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

OCPBUGS-42810: actively move bootstrap member lead #1369

wants to merge 5 commits into from
+534 −39

Conversation

tjungblu
Copy link
Contributor

This PR will actively try to move the leadership away from the bootstrap member to another healthy member.

/hold

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 19, 2024
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 19, 2024
@openshift-ci-robot
Copy link

@tjungblu: This pull request references Jira Issue OCPBUGS-42810, which is invalid:

  • expected the bug to target the "4.18.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This PR will actively try to move the leadership away from the bootstrap member to another healthy member.

/hold

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from dusk125 and Elbehery November 19, 2024 13:22
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 19, 2024
@tjungblu tjungblu force-pushed the OCPBUGS-42810_moveleader branch 2 times, most recently from 5720e20 to a747960 Compare November 20, 2024 13:08
@tjungblu
Copy link
Contributor Author

/payload-aggregate periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade 10

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Nov 20, 2024

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ce14d7c0-a741-11ef-9f14-d84ba44f8a80-0

@tjungblu tjungblu force-pushed the OCPBUGS-42810_moveleader branch from a747960 to 0c031aa Compare November 20, 2024 13:25
@tjungblu
Copy link
Contributor Author

/retest

1 similar comment
@tjungblu
Copy link
Contributor Author

/retest

@tjungblu tjungblu force-pushed the OCPBUGS-42810_moveleader branch from 0c031aa to 369b1cd Compare November 28, 2024 13:06
@tjungblu
Copy link
Contributor Author

/cherry-pick release-4.18

@openshift-cherrypick-robot
Copy link

@tjungblu: once the present PR merges, I will cherry-pick it on top of release-4.18 in a new PR and assign it to you.

In response to this:

/cherry-pick release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@tjungblu
Copy link
Contributor Author

/payload-aggregate periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade 10

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Nov 28, 2024

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/01c7f930-ada0-11ef-8674-28fc71df2147-0

@tjungblu
Copy link
Contributor Author

/retest-required

@tjungblu
Copy link
Contributor Author

/payload-aggregate periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade 10

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Nov 28, 2024

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/e6fa4930-adab-11ef-8910-6373b3a73b44-0

@tjungblu
Copy link
Contributor Author

tjungblu commented Dec 2, 2024

/payload-aggregate periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade 10

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Dec 2, 2024

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/37cd8410-b0a6-11ef-8f86-550f1a2ac5ba-0

@tjungblu
Copy link
Contributor Author

tjungblu commented Dec 3, 2024

/retest

/payload-aggregate periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade 10

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Dec 3, 2024

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/63892fc0-b180-11ef-99d0-e245fa792cd0-0

lance5890
lance5890 reviewed Dec 6, 2024
View reviewed changes
pkg/operator/bootstrapteardown/bootstrap_teardown_controller.go Outdated
err = c.removeBootstrap(timeoutCtx, safeToRemoveBootstrap, hasBootstrap, bootstrapID)
if hasBootstrap {
if err := c.ensureBootstrapIsNotLeader(ctx, bootstrapMember); err != nil {
klog.Errorf("error while ensuring bootstrap is not leader: %v", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we return err here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really, I wouldn't want the controller to degrade when that operation fails once. Mind you, this happens very very rarely and affects our CI pass rates in very few cases of hundreds.

@vrutkovs
Copy link
Member

vrutkovs commented Jan 8, 2025

/retest

@vrutkovs
Copy link
Member

vrutkovs commented Jan 8, 2025

/test e2e-metal-ovn-ha-cert-rotation-shutdown

@vrutkovs
Copy link
Member

/test e2e-metal-ovn-ha-cert-rotation-shutdown e2e-metal-ovn-sno-cert-rotation-shutdown

@dusk125
Copy link
Contributor

dusk125 commented Aug 29, 2025

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 29, 2025
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Aug 29, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dusk125, tjungblu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tjungblu
fix naming
43a24c5
@tjungblu
OCPBUGS-42810: actively move bootstrap member lead
994ce3f 
This PR will actively try to move the leadership away from the bootstrap member to another member.

Signed-off-by: Thomas Jungblut <[email protected]>
@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 1, 2025
@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 1, 2025

New changes are detected. LGTM label has been removed.

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 1, 2025

@tjungblu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-metal-assisted 994ce3f link true /test e2e-metal-assisted
ci/prow/e2e-aws-disruptive-ovn 994ce3f link false /test e2e-aws-disruptive-ovn
ci/prow/e2e-metal-ovn-ha-cert-rotation-shutdown 994ce3f link false /test e2e-metal-ovn-ha-cert-rotation-shutdown
ci/prow/e2e-metal-ovn-sno-cert-rotation-shutdown 994ce3f link false /test e2e-metal-ovn-sno-cert-rotation-shutdown
ci/prow/e2e-azure-ovn-etcd-scaling 994ce3f link false /test e2e-azure-ovn-etcd-scaling
ci/prow/e2e-aws-disruptive 994ce3f link false /test e2e-aws-disruptive
ci/prow/unit 994ce3f link true /test unit
ci/prow/e2e-gcp 994ce3f link false /test e2e-gcp
ci/prow/e2e-gcp-disruptive 994ce3f link false /test e2e-gcp-disruptive
ci/prow/e2e-gcp-disruptive-ovn 994ce3f link false /test e2e-gcp-disruptive-ovn
ci/prow/e2e-aws-etcd-recovery 994ce3f link false /test e2e-aws-etcd-recovery
ci/prow/e2e-aws-cpms 994ce3f link true /test e2e-aws-cpms
ci/prow/e2e-aws-ovn-etcd-scaling 994ce3f link true /test e2e-aws-ovn-etcd-scaling
ci/prow/e2e-aws-ovn-serial 994ce3f link true /test e2e-aws-ovn-serial
ci/prow/e2e-vsphere-ovn-etcd-scaling 994ce3f link false /test e2e-vsphere-ovn-etcd-scaling
ci/prow/e2e-metal-ovn-two-node-fencing 994ce3f link false /test e2e-metal-ovn-two-node-fencing
ci/prow/e2e-gcp-ovn-etcd-scaling 994ce3f link false /test e2e-gcp-ovn-etcd-scaling

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@tjungblu
Copy link
Contributor Author

tjungblu commented Sep 2, 2025

/retest-required

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 2, 2025

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/896d0f70-87e2-11f0-9ad1-290c2e9d6120-0

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 2, 2025

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.18-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/8cd00ac8-87e2-11f0-90f9-15a444d87075-0

@tjungblu
Copy link
Contributor Author

tjungblu commented Sep 2, 2025

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade 10

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 2, 2025

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9aae7ad0-87e2-11f0-8e36-9df58d5d7d07-0

@tjungblu
Copy link
Contributor Author

tjungblu commented Sep 3, 2025

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade 10

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 3, 2025

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/d13fd050-888a-11f0-8776-036a5a213b71-0

@tjungblu
Copy link
Contributor Author

tjungblu commented Sep 3, 2025

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade 10

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 3, 2025

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/4b6983f0-88c0-11f0-9736-1b2d461ebfab-0

@tjungblu
Copy link
Contributor Author

tjungblu commented Sep 4, 2025

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade 10

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 4, 2025

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/69083890-8954-11f0-9223-cf9f15e3c9ba-0

@tjungblu
Copy link
Contributor Author

tjungblu commented Sep 4, 2025

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade 20

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 4, 2025

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/7e345990-897e-11f0-8451-b91fd76131ee-0

@tjungblu
Copy link
Contributor Author

tjungblu commented Sep 5, 2025

/payload-aggregate periodic-ci-openshift-release-master-ci-4.21-e2e-aws-ovn-upgrade 20

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 5, 2025

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.21-e2e-aws-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1a7dde20-8a3d-11f0-9f46-8138e528642f-0

@tjungblu
Copy link
Contributor Author

tjungblu commented Sep 9, 2025

/payload-aggregate periodic-ci-openshift-release-master-ci-4.21-e2e-azure-ovn-upgrade 20

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 9, 2025

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.21-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/94d60770-8d75-11f0-953c-decb559445d5-0

@tjungblu
Copy link
Contributor Author

/payload-aggregate periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade 20

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 10, 2025

@tjungblu: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-upgrade

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9c92dad0-8e3b-11f0-99df-a62f065d382a-0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dusk125 dusk125 Awaiting requested review from dusk125

@Elbehery Elbehery Awaiting requested review from Elbehery

1 more reviewer

@lance5890 lance5890 lance5890 left review comments

Reviewers whose approvals may not affect merge requirements
Assignees

@dusk125 dusk125

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@tjungblu @openshift-ci-robot @openshift-cherrypick-robot @vrutkovs @openshift-bot @dusk125 @lance5890 @openshift-merge-robot