@pzygielo In this specific project, tests are actually allowed to fail, as they indicate the project which they test against is faulty and the vendor should fix the issue. The test is not broken, but for that test Payara is.

It agree that is does make it difficult to see if new tests fail or not. We've all been trained (me too) to strive for passing builds, but this project is thus a little different. @Ignore is unfortunately not correct here.

I think it would be better to have such test in vendor(s) suite, and not in samples. From samples I'd expect to pass rather than show faults in used components (regardless of ci-job disclaimer).

With failing test, one always has to check the reason for failure. And having two failing just on start (that are not announced to be known to fail by default in any way in code), although somehow educational, isn't quite nice experience to begin with.

@Ignore is not marker of broken test (Probably I should have written Build was broken instead of Tests were broken).