Py.test with xdist fails tests parameterized with random values
Has anyone noticed the following strange behavior for pytest and xdist.
If you try to run a test parameterized with some randomly selected values, the test fails. The same test runs without any problem if xdist is not used.
You can use the following code to reproduce this code.
import pytest
import random
PARAMS_NUMBER = 3
PARAMS = []
for i in range(PARAMS_NUMBER):
PARAMS.append(random.randrange(0, 1000))
@pytest.mark.parametrize('rand_par', PARAMS)
def test_random_param(rand_par):
assert 500 > rand_par
Without xdists, this works great.
No test runs with xdist with the following output
============================= test session starts =============================
platform win32 -- Python 2.7.3 -- py-1.4.24 -- pytest-2.6.2
plugins: xdist
gw0 [3] / gw1 [3] / gw2 [3] / gw3 [3]
scheduling tests via LoadScheduling
============================== in 1.93 seconds ===============================
The versions I am using:
- python 2.7.3
- pytest 2.6.2
- pytest-xdist 1.11
Note:
With some older versions (xdist 1.8 and pytest 2.4.X or 2.5.X don't remember exactly) xdist stopped on assertion in dsession.py
assert collection == col
Thanks in advance for any help on how to solve it or at least a workaround :)
source to share
A workaround would be to parameterize with reproducible (but meaningless) values, like all numbers between 0 and PARAMS_NUMBER-1
. Each test can then individually select a random value when it runs. To find out which random variable it chose (to reproduce in case of failure), each test must first print it. At least that's what I do with some non-xdist tests; I hope it works with xdist too, i.e. The prints are correctly propagated to the parent process and only show up if an individual test fails (or py.test is run with -s).
source to share
So here is the code after Armins hint :)
import pytest
import random
PARAMS_NUMBER = 3
PARAMS = []
for i in range(PARAMS_NUMBER):
PARAMS.append(1000)
@pytest.mark.parametrize('rand_par', PARAMS)
def test_random_param(rand_par):
par_val = random.randrange(0, rand_par)
assert 500 > par_val
And it runs the test 3 times with a randomly selected value.
Update: I created an issue for the xdist project and it is resolved in the sense of returning reasonable information to the user.
More details can be found here py.test with xdist does not execute tests parameterized with random values
source to share