JUnit report sent to wrong build

[jawnsy]

Dear maintainer,

Thanks so much for creating this excellent plugin and sharing it for others to use!

I have multiple workflows defined, and it seems that the check is added to the first workflow, rather than the one that actually ran the test suite. Is there some way to associate the report with the correct workflow definition? Ideally I'd like the report to be below the test suite (e.g. job that runs tests, publishes report, should have the report appended as a check below it or nearby)

[mikepenz]

Hi @jawnsy

This is sadly a problem of GitHub, which you can find more about here:
https://github.community/t/github-actions-status-checks-created-on-incorrect-check-suite-id/16685

Hoping them providing a fix for that soon

[pbrisbin]

Hi there- unfortunately, this issue makes this Action pretty unusable for us since the check run output is the only place our test suite failures are visible and when they get added to some other "random" Workflow, they're impossible to find.

I waded through some of the related issues and there seems to be a potential workaround:

JUnit reporter github action should update existing check instead of creating new one.

Get check which triggered action:
https://docs.github.com/en/rest/reference/checks#get-a-check-run

Update this check with provided information about test results:
https://docs.github.com/en/rest/reference/checks#update-a-check-run

If I understand correctly, this would add all of your annotations to the actual Job where I'm using the step (our tests), instead of as a new check that is getting misplaced in some other Workflow.

Besides being a workaround for this issue, this is actually exactly where I was trying to find these annotations to begin with. I think it's a much more useful place to put them. Even if the check was showing up under the correct Workflow, I think users would still be looking for failures in the existing, failed "Test Job" check and not some separate "JUnit Reporter" check (I know we could configure the name to be more obvious, but still).

Am I missing anything? Would you consider implementing such functionality?

[mikepenz]

@pbrisbin thank you very much for the added notes.

It sure sounds like an opportunity to use those APIs and make it so it will only update the current run. Would need to look into it.
Do you maybe know an action already using this approach?

[pbrisbin]

Hmm, I can't say that I do. A lot of Actions add annotations to the Job's check where they're run, but they do it by outputting messages with a problem-matcher attached, not by updating the Check. I suppose that's another alternative you could consider.

Looking at the docs a bit more closely, I think it would look something like,

const ref = head_sha
const check_name = github.context.job // assumes check names are job.<job-id>
const filter = 'latest'

const checks = await octokit.rest.checks.listForRef({
  ...github.context.repo
  ref,
  check_name,
  filter
});

// assumes a check for our job exists if we're running, probably want some error-handling
const check_run_id = checks[0].run_id

// as per docs: "Each time you update the check run, annotations are 
// appended to the list of annotations that already exist for the check run."
oktokit.rest.checks.update({ owner, repo, check_run_id, ... })

Oh, and I noticed you're currently only sending your first 50 annotations. I suppose it's reasonable to expect a test suite in a to never have more than 50 test failures in a given run, but you could send everything if you call this .update() multiple times, each with fewer than 50.

The Checks API limits the number of annotations to a maximum of 50 per API request. To create more than 50 annotations, you have to make multiple requests to the Update a check run endpoint. Each time you update the check run, annotations are appended to the list of annotations that already exist for the check run

[mikepenz]

@pbrisbin there's a version coming up using the outlined APIs. Sadly those lead to a different result than anticipated, and due to the check still being running at the time of updating the result, it won't keep the failure state for example.

Just for reference, there is still no API to create a check in a specific suite / workflow sadly:

https://github.community/t/specify-check-suite-when-creating-a-checkrun/118380/9
https://github.community/t/associate-a-check-run-with-a-workflow/122540
https://github.community/t/associating-a-check-run-with-a-specific-action-run/126682

[pbrisbin]

Awesome!

due to the check still being running at the time of updating the result, it won't keep the failure state for example

Let me make sure I understand...

So, if I use this in a app-test job for example, it runs tests then does the JUnit thing, the Job will pass/fail based on the tests and not the JUnit processing? That seems... desired? I kind of think the only reason the JUnit action needs to control the failure at all is because it's doing that separate-check thing. When it's part of the test's check, the tests setting the check result just makes sense (to me).

Am I missing something?