Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Still problems with test results? #3171

Open
LorenzoBettini opened this issue Aug 24, 2024 · 2 comments
Open

Still problems with test results? #3171

LorenzoBettini opened this issue Aug 24, 2024 · 2 comments

Comments

@LorenzoBettini
Copy link
Contributor

On my fork, I recently saw a failure in one of the builds: https://github.com/LorenzoBettini/xtext/actions/runs/10456433264/job/28957424248

I see this:

image

While there are several links with raw output on that page, I still don't see, in the summary, any direct link to the failed tests. So I don't understand what's flake and what's really failed.

Since -fae has been added, the raw output from the build is not easy to navigate, because, it fails at the end, and not on the project with the failures:

image

Unless I miss something, I still found this new test report unusable, and in general it makes things harder

cc @cdietrich @szarnekow @HannesWell

@HannesWell
Copy link
Member

While there are several links with raw output on that page, I still don't see, in the summary, any direct link to the failed tests. So I don't understand what's flake and what's really failed.

From your screenshot it actually looks like expected to me. For example in m2e it looks the same:
https://github.com/eclipse-m2e/m2e-core/runs/29202595998
In the comment on a PR you don't get each failure listed, you only get that from the overview. In the comment you should only get the link to the overview as described here:
#3144 (comment)

To me the only problem left seems to be that flaky tests are falsely detected as persistent errors. Maybe we are lucky and EnricoMi/publish-unit-test-result-action#539 gets more drive now and is fixed soon.

Alternatively, could the flaky tests be fixed to work on the first run?

@LorenzoBettini
Copy link
Contributor Author

I still don't have a direct and easy access to the failed test: in the output, besides flakes, it's easy to spot the test that finally failed, while now there's no way to see the failed test immediately, at least, I don't find an easy one.

For example, in the linked URL https://github.com/LorenzoBettini/xtext/actions/runs/10456433264/job/28957424248 can you tell which test(s) finally failed?

If flaky tests could be easily fixed, it would be a better world ;)
we never found a way...

I'd tend to remove at least -fae. @szarnekow @cdietrich what do you think?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants