[Possible issue] TEST-junit-jupiter.xml replaced when using the rerunFailingTestsCount from maven-surefire-plugin

[eliasnogueira]

Hi all! 👋🏼

I'm facing a problem (won't say it's a bug because I don't know if it's by design) and need your help figuring out whether it's by design or a bug.

The problem

When the <rerunFailingTestsCount> configuration from the maven-surefire-plugin. is in use, the following happens when I look at the contents of TEST-junit-jupiter.xml:

• First run: generates all the data, meaning all the failures, errors, and successful tests
• N runs (number of retries): shows only the retried ones, ignoring the successful ones from the first run

Why this is a problem

Even though I can see the individual TEST-*.xml files in the target/surefire-reports and it shows the tests regardless its run status the TEST-junit-jupiter.xml shows only the failed ones.

In terms of reporting, it's bad as I can't see the successful ones.

My specific case

My case is that I enrich each <testcase> entry in the TEST-junit-jupiter.xml generated file by adding new <property> entries. So I'm missing all the successful executions.

Experiment

It only happens when I use the <rerunFailingTestsCount> configuration from the maven-surefire-plugin.
When I develop a custom retry extension by implementing the handleTestExecutionException from TestExecutionExceptionHandler, it works as expected, meaning I see all tests.

Confirmation I want

It seems to be a "problem" with the maven-surefire-plugin, but I would like any input/experience/etc. for anyone to understand whether it would be acceptable behaviour.

I'm open to help with anything.
Thank you!

[marcphilipp]

TEST-junit-jupiter.xml sounds like it's written by JUnit's LegacyXmlReportGeneratingListener, not Surefire's XML reporting support. Is that correct?

If so, how are you using that? Could you please share a reproducer project?

[eliasnogueira]

@marcphilipp I apologize for the delay in my response.

Here's the project I created to reproduce it: https://github.com/eliasnogueira/junit-xml-report-tryout
There's a full explanation in the README.md file, along with some observations.

Thank you for the help!

[mpkorstanje]

The observed behaviour is a consequence of using Surefire to rerun failed JUnit tests while using JUnit to write reports.

From JUnits perspective, the second rerun execution is identical to the previous execution but but for a smaller selection of tests. So what actually happens is that two reports were written. One report is written for the full test run, then a second report is written for the rerun.

You can either adapt your GlobalLegacyXmlListener to create a new directory for each report or rely on Surefire to generate reports.

[eliasnogueira]

Thanks @mpkorstanje!

My main take, and I can be really wrong on this, is that the correct behaviour should be showing all the tests in the report, instead of only the rerun ones, and I know that this implementation live on Surefire via JUnit4Provider for Legacy (I think) and JUnitCoreProvider for JUnit 5.

I know I can create an extension using, for example, the InvocationInterceptor to create the retry strategy and "fix" it for me.

I also seek feedback from you guys to understand what would be the "correct" behaviour.

[mpkorstanje]

For JUnit the "correct" behaviour is that the reports JUnit generates overwrite each other as they are created by two distinct executions. Only Surefire knows that it is rerunning tests, so only Surefire can create the correct the report. In other words, for correct reports you have to use Surefire's XML reporting support, not JUnits.

[eliasnogueira]

Thank you, @mpkorstanje!