Make xpass a failure again

[RonnyPfannschmidt]

What's the problem this feature will solve?

Some years ago we accepted a change which made xpass not a failure

I recently hit multiple cases where it was missed that a test was fixed

That's not an acceptable default

Describe the solution you'd like

Alternative Solutions

Additional context

[nicoddemus]

This means changing the default of the strict parameter to True?

[RonnyPfannschmidt]

That's would be the starting point, In the long run it should go away

[TanyaAgarwal28]

Hi, How we can test this scenario, I can't see xpass as a marker in pytest.

[RonnyPfannschmidt]

@TanyaAgarwal28 xpass is a test outcome for when a xfail marked test passed "unexpectedly"

[TanyaAgarwal28]

Yeah Got it so what we are facing issue here?
Isin't a feature that if xfail testcase is passed then it will came as xpassed in outcome?
import pytest

@pytest.mark.xfail
def test_example_expected_fail():
assert 3 + 3 == 6

[RonnyPfannschmidt]

@TanyaAgarwal28 the problem is that its a misfeature - it hides important change for a non-flaky testsuite - fixing a previously failing test by accident

when non-strict xfail was introduced, it was a major mistake not to use a new mark

[TanyaAgarwal28]

So basically we have to revert this change and if xfail testcase is by mistakly fixed then the outcome should be passed??
And for non_strict_xfail we should have new mark??

[TanyaAgarwal28]

Overview of 4 scenarios for xfail and non_strict_xfail
xfail testcase which is expectedly fail:
import pytest

@pytest.mark.xfail
def test_example_expected_fail():
assert 3 + 3 == 7 # This test is marked as expected fail

xfail testcase which is passed:
import pytest

@pytest.mark.xfail
def test_example_expected_fail():
assert 3 + 3 == 6 # This test is marked as expected fail

non_strict_xfail testcase is passed:
import pytest

@pytest.mark.non_strict_xfail
def test_example_expected_fail():
assert 3 + 3 == 6 # This test is marked as expected fail

[nicoddemus]

So basically we have to revert this change and if xfail testcase is by mistakly fixed then the outcome should be passed??
And for non_strict_xfail we should have new mark??

IMHO what should be done:

1. pytest.mark.xfail has a strict parameter, which defaults to False today. We want to issue a warning that this default will change to True in the future, but only if the strict flag has not been passed explicitly.
2. A few major releases later, we change the default to True and remove the warning.

Is this what you have in mind @RonnyPfannschmidt ?

[RonnyPfannschmidt]

Correct, once the default is switched I'd like to ensure xpass is either a warning or a failure in any case

[TanyaAgarwal28]

okay Got it
Thanks