Backup/Restore issue with Multibranch plugin

[blucas]

I'm using Jenkins Operator v0.2.0 and backup-pvc v0.0.6

When restoring a Bitbucket Team job (Organization Folder) using the cloudbees-bitbucket-branch-source:2.4.6 plugin, the operator fails to "fully" restore the folder and jobs. After the restore completes and when you navigate to the Bitbucket Team job in the Jenkins UI, it displays as if it has yet to scan the Bitbucket Team and all of that team's repositories / branches.

I checked the backup file. It does contain the builds for each repository/branch and did successfully restore them on disk. The only way to get the builds to display after restore is to issue a new "Scan Organization Folder Now" operation on the Organization Folder. But this creates further issues as that scan assumes it's the first and so resets nextBuildNumber to 1.

The issue is that the plugin or framework expects that the job's config.xml file is also restored. If I remove --exclude jobs/*/config.xml from backup.sh and trigger a restore, then everything displays as expected in the UI. Jenkins knows what the nextBuildNumber should be for all jobs under the Organization Folder.

I would like to suggest that exclusion of the config.xml file become a configurable setting. For example add an env_var EXCLUDE_CONFIG_XML=true|false whereby the default is true and one can disable that functionality on the backup container by providing a false value when configuring the container's environment variables

[tomaszsek]

Hi @blucas

How you create your BitBucket jobs? Do you run groovy scripts or Configuration as Code?
You can force Jenkins to reload configuration from disk by running groovy script Jenkins.instance.reload().

Cheers

[blucas]

Hi @tomaszsek

We create a single job using JobDSL via the seed-job agent. The DSL is below with some sections omitted. I am aware of the reload functionality. I did issue that command, but it still didn't display the folders/jobs under my team/organization folder.

organizationFolder('my-bitbucket-team') {
    description('This contains branch source jobs for Bitbucket Team: My Bitbucket Team')
    displayName('My Bitbucket Team')
    triggers {
        periodic(30) // minutes
    }

    organizations {
        bitbucket {
            // Credentials MUST be of type USERNAME-PASSWORD
            credentialsId('bitbucket-credentials')
            repoOwner('my-bitbucket-team')
            traits {
            }
        }
    }

    // We need to configure this stuff by hand until JobDSL gets some support (https://github.com/jenkinsci/git-plugin/pull/595)
    configure { node ->
        def traits = node / navigators / 'com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator' / traits
        traits << 'com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait' {
            strategyId(1)
        }
        traits << 'com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait' {
            strategyId(1)
        }
        // Clone repositories using SSH
        traits << 'com.cloudbees.jenkins.plugins.bitbucket.SSHCheckoutTrait' {
            credentialsId('my-key')
        }
    }
}

[tomaszsek]

I suggest:

• create my-bitbucket-team job throughout seed job
• run my-bitbucket-team job in groovy script
• wait for the my-bitbucket-team to complete
• run Jenkins.instance.reload() groovy script

[blucas]

create my-bitbucket-team job throughout seed job

That's what we do (see above DSL)

run my-bitbucket-team job in groovy script

I'm not too sure what you mean by this. the seed job automatically runs this the first time it gets created.

wait for the my-bitbucket-team to complete

This is what causes the nextBuildNumber to reset

run Jenkins.instance.reload() groovy script

This does nothing other than pickup the wrong build number because the previous step has reset it.

For example, lets say My Bitbucket Team has the following repositories in it:

• repo-1
• repo-2
• repo-3

Each repo has a master branch with a Jenkinsfile in it.

1. Install Operator and Jenkins CR with a seed job which picks up the DSL I mentioned previously.
2. Seed-agent will create job Organization Folder 'my-bitbucket-team' and trigger an "Scan Organization Folder" operation
3. Scan will pick up the repos above and create my-bitbucket-team/repo-N folders in Jenkins
4. Each repo-N folder will trigger a "Multibranch Pipeline Scan" to detect all branches with a Jenkinsfile
5. The scan will generate a 'master' pipeline job for each repo-N
6. The scan will trigger a build of each of the jobs in step 5.
7. Suppose time passes, and repo-1/master job has 5 builds, repo-2/master has 6 builds and repo-3/master has 10 builds. (confirm builds are in backup file)
8. Trigger a restore (for example install a plugin on the Jenkins CR yaml file).
9. Jenkins Operator forces a new Jenkins Master to be created and restores backup file.
10. Login to Jenkins.

You will see that the seed-job has been triggered again and created the 'my-bitbucket-team' Organization Folder. Click on that folder and you'll be presented with a page similar to the one below. If the restore had worked properly, it would list repo-1, repo-2 and repo-3 instead and if you click on them, they would list a master job for each of those repos. Each master job would list each of its builds 5, 6, and 10 respectively.

Check the jenkins-master disk and you will see the jobs and their respective builds. You can to http://<jenkins-url>:<port>/script and run Jenkins.instance.getItemByFullName("my-bitbucket-team/repo-1/master").getNextBuildNumber() and it will fail as in-memory, Jenkins doesn't now about that job.

If you trigger a reload-from-disk (Jenkins.instance.reload()) and the UI still displays the "This folder is empty" page and the script above will still fail. Now, if you trigger a "Scan Organization Folder Now" (see screenshot) Jenkins will re-scan bitbucket and find all those repos/branches and re-trigger builds on those branch job. BUT the problem with this is Jenkins assumes it's the first build of these jobs and resets nextBuildNumber on disk. Jenkins will silently (unless you look at the Jenkins log) fail to build these jobs as there are already builds for them. At this point your configuration on disk is broken as the "Scan Organization Folder Now" functionality has reset the build number on disk.

I hope this helps clear up any confusion.

[tumevoiz]

Hi, @blucas

I apologize for late response, but I have a fix for your problem.

Please put this to your groovy scripts, and everything should work.

def jobName = "my-bitbucket-team"
def job = Jenkins.instance.getItem(jobName)

job.scheduleBuild()
sleep 10000
job.doReload()

If this will solve your problem, please close issue.

Cheers

[blucas]

I haven't had time to test this out, but I don't see how this will solve the problem. The nextBuildNumber, in theory, will still be reset to 1.

[tomaszsek]

Hi @blucas

job.scheduleBuild() will trigger the scan and job.doReload() will read the nextBuildNumber from the disk(restored by the operator).

Cheers

[blucas]

I still don't see how this will help, not to mention it feels more like a hack than a solution (the sleep NNNN).

scheduleBuild() will trigger a scan, that scan will reset the nextBuildNumber on disk to 2.
sleep N makes me assume we have to wait for the scan to finish, this could take seconds, or hours for larger organizations
doReload() will pickup the wrong build number on disk because of scheduleBuild().

[pawelprazak]

I've hit a different issue with restore and the multibranch jobs, the "Build with parameters" button no longer appears. And scanning/reindexing doesn't help like it used to.

In general I've hit multiple problems with incompatibility of multibranch plugin with configuration as code and job dsl plugins.

If at all possible, I would advice to reconsider using multibranch plugin at all.

[pawelprazak]

After I excluded jobs/*/branches/*/config.xml I could see the "Build with parameters" button (after reindexing).

And now I can confirm @blucas 's issue, I was able to reproduce it.

#104 (comment) wont work because the jobs won't be loaded, in a best case scenario it will cause a race condition.

[pawelprazak]

On the other hand if we start to include jobs/*/config.xml I'm afraid it would cause even more problems with "rotten state" that the whole "immutable configuration as code" thing we have going one in the operator tries to solve.

I wonder are there any documented issues regarding job dsl plugin and job state backups...