"python -m doctest" ignores files with the same name in different directories
Let's assume we have two files:
crash /stuff.py
"""
>>> True
False
"""
skip /stuff.py
"""
>>> True
True
"""
Then we run both of them in the doctest section:
python -m doctest fail/stuff.py pass/stuff.py
As expected, we see an error from fail/stuff.py
. But if we run them in reverse order:
python -m doctest pass/stuff.py fail/stuff.py
Then he passes!
Is there some fundamental reason why the Python import system is unable to handle loading tests from both files, or is the doctie just destroyed?
source to share
Is there some fundamental reason why the Python import system is unable to handle loading tests from both files, or is the doctie just destroyed?
Doctests are mostly just broken. There is no underlying reason why the Python import system cannot handle this scenario. Ergo, there is no underlying reason why a test runner cannot handle this scenario. I doubt they thought too much when writing the command line interface because most people would not use doctrine as a runner directly (rather, integrating library code with a more fully featured runner and using the doctest plugin would be much more common).
You can just use the best test runner. For example, a nose (among others) will not have this problem:
$ nosetests pass/stuff.py fail/stuff.py --with-doctest
.F
======================================================================
FAIL: Doctest: stuff
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/doctest.py", line 2199, in runTest
raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for stuff
File "/private/tmp/fail/stuff.py", line 0, in stuff
----------------------------------------------------------------------
File "/private/tmp/fail/stuff.py", line 2, in stuff
Failed example:
True
Expected:
False
Got:
True
----------------------------------------------------------------------
Ran 2 tests in 0.004s
FAILED (failures=1)
source to share
You can only have one top-level module with the same name in a Python process, and it python -m doctest
handles all files passed to it as top-level modules. Import pass/stuff.py
fills the module cache entry for the module stuff
, and when doctest tries to import the second file you specified, the import system finds the entry for pass/stuff.py
instead of loading fail/stuff.py
.
There are ways they can try to get around this. For example, it is possible to import a module from a file without populating its entry sys.modules
, but this may still fail if these files with the same name also have the same name, different dependencies, or if any circular imports are involved. They could try to drop sys.modules
into a pre-test state after running a test, which would be more reliable but less efficient, and could still fail with modules that don't play well on reboot. Another option is to create a separate sub-process for each test, which would be the most reliable but least efficient.
source to share