What else can I do to troubleshoot a package not importing into python, but importing into ipython, but into virtualenv?
I'm a Python enthusiast who decided to get serious about making some of my own reusable packages.
Summary of my problem:
- The call
python test_dummy.py
does not allow the package to be importedeforest
. ImportError is thrown. -
import eforest
in the python interpreter and in the ipython interpreter both throw exceptions. - Calling
run test_dummy.py
in the ipython interpreter does not throw an exception. - All cases are executed inside a virtual file.
virtualenv has nothing to do with my problem. I asked a new question: a script that adds packages to import sys.path as expected when run from ipython, but throws an exception when the script is run from python
A similar question is in Unable to import package from virtualenv
I read the documentation on virtualenv
and virtualenvwrapper
and installed them both on an Ubuntu 14.04.1 LTS system with Python 2.7.6.
I created a virtualenv:
mkvirtualenv module_troubleshooting -a . -r requirements.txt
My requirements.txt
file contains:
Django==1.6.6
EasyProcess==0.1.6
PyVirtualDisplay==0.1.5
South==1.0
argparse==1.2.1
ipython==2.2.0
lxml==3.3.5
selenium==2.42.1
wsgiref==0.1.2
I have activated my virtualenv with workon module_troubleshooting
My virtualenv was active: (module_troubleshooting)dmmmd@ubuntuG5:
I figured out from the virtualenvwrapper documentation that I could add packages to sys.path using the command add2virtualenv
.
Using add2virtualenv
, I added my very simple packages that I created. I have confirmed that indeed the pth file contains these packages.
/home/dmmmd/development/python/singleton /home/dmmmd/development/python/display /home/dmmmd/development/python/browser /home/dmmmd/development/python/eforest /home/dmmmd/development/python/dummy
I ran ipython
to see if I could import these packages. I could. No mistakes. Examination sys.path
in ipython showed that the above paths were present.
For my own guidance, I created the following module called test_dummy.py:
import browser
import display
import singleton
import eforest
I run the following command in ipython : run test_dummy.py
.
No exceptions in ipython .
Then I exited ipython and ran the following command to make sure the global python was NOT called and to check if python would run the script:/home/dmmmd/.virtualenvs/module_troubleshooting/bin/python test_dummy.py
Traceback (most recent call last):
File "tests/test_dummy.py", line 4, in <module>
import eforest
ImportError: No module named eforest
The other three packages, browser, display and singleton were imported as expected and an exception was thrown in import eforest
I first assumed that eforest might contain some kind of exception. So I ran /home/dmmmd/.virtualenvs/module_troubleshooting/bin/python
and had the expected prompt and no import errors when importing eforest:
Python 2.7.6 (default, Mar 22 2014, 22:57:26)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import eforest
>>> print eforest
<module 'eforest' from 'eforest/__init__.pyc'>
>>>
So, I assumed the sys.path for ipython and python might be different. I saved the sys.path file for python to a file and edited it so that the list could be imported from sys.path
:
/home/dmmmd/.virtualenvs/module_troubleshooting/bin/python -c "import sys; print sys.path" > path.py
I edited path.py to from path import path
be the list sys.path
from the call /home/dmmmd/.virtualenvs/module_troubleshooting/bin/python
.
Then I started ipython and typed the following commands:
In [1]: from path import path
In [2]: path # this is the sys.path from calling /home/dmmmd/.virtualenvs/module_troubleshooting/bin/python
Out[2]:
['',
'/home/dmmmd/development/python/singleton',
'/home/dmmmd/development/python/eforest',
'/home/dmmmd/development/python/dummy',
'/home/dmmmd/development/python/display',
'/home/dmmmd/development/python/browser',
'/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7',
'/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7/plat-powerpc-linux-gnu',
'/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7/lib-tk',
'/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7/lib-old',
'/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7/lib-dynload',
'/usr/lib/python2.7',
'/usr/lib/python2.7/plat-powerpc-linux-gnu',
'/usr/lib/python2.7/lib-tk',
'/home/dmmmd/.virtualenvs/module_troubleshooting/local/lib/python2.7/site-packages',
'/home/dmmmd/.virtualenvs/module_troubleshooting/lib/python2.7/site-packages']
To see if ipython is differentsys.path
, I typed:
In [5]: import sys
In [6]: len(sys.path)
Out[6]: 19
In [7]: len(path)
Out[7]: 16
In [8]: [item for item in sys.path if item not in path]
Out[8]:
['/home/dmmmd/.virtualenvs/module_troubleshooting/bin',
'/home/dmmmd/.virtualenvs/module_troubleshooting/local/lib/python2.7/site-packages/IPython/extensions',
'/home/dmmmd/.ipython']
The last two elements of ipython sys.path are related to ipython . One other item: "/home/dmmmd/.virtualenvs/module_troubleshooting/bin".
Can anyone suggest any other troubleshooting advice or explain me how to have '/home/dmmmd/.virtualenvs/module_troubleshooting/bin' in sys.path in ipython allows eforest to import as expected in ipython when is called run test_dummy.py
?
eforest package:
eforest/
├── CustomElementFilters.py
├── CustomEtrees.py
├── CustomEtrees.pyc
├── __init__.py
└── __init__.pyc
Again, import the import as expected in ipython and with a python interpreter, but it is not imported when the script is passed as an argument /home/dmmmd/.virtualenvs/module_troubleshooting/bin/python test_dummy.py
.
The only possible eforest exception that comes to my mind, which is different from other packages, is that it imports lxml. Can lxml depend on having '/home/dmmmd/.virtualenvs/module_troubleshooting/bin' in sys.path?
Any suggestions for troubleshooting are greatly appreciated. I first noticed the problem when I was using py.test. I thought it was related to py.test, but as it turns out, the test scripts I wrote were not executed with python either. I decided to stop there and see if I could help.
Excluded virtualenv as source of problem
I have executed the following script with python test_dummy.py
with virtuenv deactivated .
import sys
import os
HOME = os.path.expanduser('~')
PYTHON = os.path.join(HOME, 'development/my_python')
PACKAGES = [
'browser',
'display',
'singleton',
'eforest'
]
for package in PACKAGES:
package = os.path.join(PYTHON, package)
if os.path.exists(package) is True:
if package not in sys.path:
print "loading package '{0}'".format(package)
sys.path.append(package)
else:
print "package '{0}' already in sys.path".format(package)
else:
raise
import browser
import display
import singleton
import eforest
With virtualenv deactivated, this eforest
is the only package that is not imported when the command is run python test_dummy.py
.
Result:
Traceback (most recent call last):
File "my_django/automated_browsing/tests/test_dummy.py", line 16, in <module>
import eforest
ImportError: No module named eforest
source to share
I found a solution, although the solution does not address the issue of why certain package paths added ad hoc to sys.path
are not imported when the script is executed from python test_dummy.py
.
I originally ran into this problem while trying to use py.test. In the py.test documentation, I saw this tip about "managing your project with virutalenv, pip and editable mode". I ignored it because I thought it was too advanced for my level of Python knowledge.
I decided to try the tip after reading about making my own python project .
In pwd, I created a file setup.py
with the following code:
from setuptools import setup, find_packages
setup(name='import_troubleshooting', version='1.0')
packages = find_packages(exclude=[
'my_django',
'fake*',
'tests*'
])
Then I ran the following code on the command line:
pip install -e . # the pip way (which just calls "setup.py develop")
Then I ran the following code with the expected results:
$ python tests/test_dummy.py
# output
package '/home/dmmmd/development/my_python/browser' already in sys.path
package '/home/dmmmd/development/my_python/display' already in sys.path
package '/home/dmmmd/development/my_python/singleton' already in sys.path
loading package '/home/dmmmd/development/my_python/test_a'
loading package '/home/dmmmd/development/my_python/hello_world'
loading 'browser'
loading 'display'
loading 'singleton'
loading 'test_a'
loading 'hello_world.hello'
Hello, world!
None of this explains to me why I couldn't add package paths in a special way. However, I'm glad I learned something about installing and packaging python. test_dummy.py
The script now works as expected when called from python tests/test_dummy.py
.
Now I am confused as to why three packages are already in sys.path
when the test_dummy.py
script is run :
package '/home/dmmmd/development/my_python/browser' already in sys.path
package '/home/dmmmd/development/my_python/display' already in sys.path
package '/home/dmmmd/development/my_python/singleton' already in sys.path
I don't remember doing anything that could add them to sys.path
, but it is possible since I am dealing with various tutorials. They are not sys.path
in any other environment where I have not run the created one setup.py
.
My confusion is a symptom of Python's complexity, not the reason!
Thanks everyone for their input.
NB: This is the output python test_dummy.py
after the new virtualenv:
$ pip install -e .
# install output unremarkable
$ python tests/test_dummy.py
# output
loading package '/home/dmmmd/development/my_python/automated_browsing/browser'
loading package '/home/dmmmd/development/my_python/automated_browsing/display'
loading package '/home/dmmmd/development/my_python/automated_browsing/singleton'
loading package '/home/dmmmd/development/my_python/automated_browsing/hello_world'
loading package '/home/dmmmd/development/my_python/automated_browsing/test_a'
loading 'browser'
loading 'display'
loading 'singleton'
loading 'hello_world.hello'
Hello, world!
loading 'test_a'
source to share