Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core] Kotest tests aren't picked up by surefire #5009

Closed
oowekyala opened this issue May 12, 2024 · 3 comments · Fixed by #5022
Closed

[core] Kotest tests aren't picked up by surefire #5009

oowekyala opened this issue May 12, 2024 · 3 comments · Fixed by #5022
Assignees
@adangel
Labels
a:bug PMD crashes or fails to analyse a file. in:testing About tests of pmd, eg the module pmd-lang-test or pmd-test
Milestone
7.2.0

Comments

@oowekyala
Copy link
Member

Many tests written in kotlin for Kotest are not being executed by Maven.

For instance see this log:
https://github.com/pmd/pmd/actions/runs/9036543035/job/24833638646#step:7:6447

[INFO] Running net.sourceforge.pmd.lang.java.types.internal.infer.CaptureInferenceTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.178 s -- in net.sourceforge.pmd.lang.java.types.internal.infer.CaptureInferenceTest
[INFO] Running net.sourceforge.pmd.lang.java.types.internal.infer.CtorInferenceTest
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.100 s -- in net.sourceforge.pmd.lang.java.types.internal.infer.CtorInferenceTest

Those are these files: CaptureInferenceTest and CtorInferenceTest

The tests that are picked up show "Tests run: 0", and many other tests are not picked up (search BranchingExprsTestCases in the log for instance, it's not there).

I tried to upgrade kotest but that makes surefire hang completely... Not sure what happens. I noticed this because I can actually run all those tests from within Intellij, and one test case in BranchingExprsTestCases has been failing for a while... but the Maven build succeeds.

Related issue (maybe) #4236
@oowekyala oowekyala added a:bug PMD crashes or fails to analyse a file. in:testing About tests of pmd, eg the module pmd-lang-test or pmd-test labels May 12, 2024
@adangel
Copy link
Member

adangel commented May 16, 2024

IIRC kotest depends on a specific junit5 version, which needs to match and need to be updated at the same time. Back then in #4236 it was actually a display issue, the tests were executed.

Surefire uses a pattern to detect test classes which is the reason, why test classes need to end with "Test" - maybe it is as simple as renaming BranchingExprsTestCases to BranchingExprTest?
(Not sure which component actually detects test cases, whether it's surefire or a junit5 component. IDE usually have there own implementation for that).

@adangel adangel added this to the 7.2.0 milestone May 16, 2024
@adangel
Copy link
Member

adangel commented May 16, 2024

Btw. I'd consider this as a blocker - if we don't execute all the tests, we don't know if the software works as intended...

@adangel adangel self-assigned this May 16, 2024
@adangel
Copy link
Member

adangel commented May 16, 2024

I can confirm that if BranchingExprTestCases is renamed to BranchingExprTests it is being picked up by surefire (and it fails). We use the default include patterns and "TestCases" is not among the default patterns.

For the tests the report "Tests run: 0", they are either test suites (e.g. AllJavaAstTreeDumpTest or RenderersTests) or they incidentally are all subclasses of our ProcessorTestSpec. However, they seem to be executed (verified by making one test explicitly fail), but just not reported correctly.

So, it's not as bad as it sounds: Only one test class is not executed (BranchingExprTestCases).

I'll try, if we can improve the reporting for these ProcessorTestSpecs and for the suites (the individual tests of a suite are reported correctly).

@adangel adangel mentioned this issue May 17, 2024
Merged
adangel added a commit to adangel/pmd that referenced this issue May 17, 2024
@adangel
[doc] Update release notes (pmd#5009)
3be6df2
@adangel adangel mentioned this issue May 17, 2024
Merged
4 tasks
@adangel adangel mentioned this issue May 30, 2024
Open
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@adangel adangel

Labels
a:bug PMD crashes or fails to analyse a file. in:testing About tests of pmd, eg the module pmd-lang-test or pmd-test
Projects
None yet
Milestone
7.2.0
Development

Successfully merging a pull request may close this issue.

[core] Fix Kotest based tests adangel/pmd
2 participants
@adangel @oowekyala