Mix Python virtualenv packages with distribution packages?
What is a good way to deal with a situation where you are using Python virtualenv but want some of the packages to be installed through the distribution manager?
Let's say you need lxml but because you can't get it pip install lxml
to work with Ubuntu. and you really don't want to waste time on this, so just follow through apt-get install python-lxml
.
You can now build virtaulenv with --system-site-packages
and now access the shared pre-compiled lxml installed . But you will also drag and drop all the other system packages you don't need! And yes, there will be quite a few packages that will be installed outside of virtualenv, either through sudo pip ...
or sudo apt-get python-...
, so no "just keep your system clean and install whatever you can in virtualenvs so you --system-site-packages
don't drag too many packages around with you. Is not form for me.
So is there a way to only install some of the system site packages?
source to share
I always use pip install --user packagename
- it doesn't require sudo.
If you don't already have it pip
, or the OS pip
or OS easy_install
isn't working properly, first install setuptools
to the user's home directory and then use that easy_install
for pip
.
$ wget https://bootstrap.pypa.io/ez_setup.py -O - | python3 - --user
$ easy_install-3.4 --user pip
On the other hand, if you choose to use pip
inside virtualenv, don't use the option --user
for the obvious reason.
Doing as you described
For example, I want to import a module curl
that is present in the OS.
$ pip3 install --user virtualenv
$ python3 -m virtualenv myvenv
$ cd myvenv
$ source bin/activate
(myvenv)$ python3
>>> import curl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named curl
There is no such module inside the virtual one. Just to make sure I looked at what's in sys.path
.
>>> import sys
>>> sys.path
['',
'/home/username/myvenv/lib/python27.zip',
'/home/username/myvenv/lib/python2.7',
'/home/username/myvenv/lib/python2.7/lib-dynload',
'/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2',
'/home/username/myvenv/lib/python2.7/site-packages']
Note that not /usr/lib/python3.4/site-packages
as you wanted, but /usr/lib/python3.4
present. If that doesn't work for you, use it --always-copy
when creating the virtual file.
In my further steps, I create symbolic links to the module curl
and its dependencies.
>>> ^D # Press Control-D
(myvenv)$ ln -s /usr/lib/python3.4/site-packages/curl lib/python3.4/site-packages/
Note that there is no forward slash in the second argument (the file or directory the symlink points to), but there is a forward slash in the third argument (where the symbolic link should be created). This is because I am creating a symbolic link inside the directory lib/python3.4/site-packages
. Without the trailing slash, it will try to replace the directory.
Check if it works.
(myvenv)$ python3
>>> import curl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/username/myvenv/lib/python3.4/site-packages/curl/__init__.py", line 9, in <module>
import sys, pycurl
ImportError: No module named 'pycurl'
Nope. curl
depends on pycurl
. We need to go deeper.
>>> ^D
(myvenv)$ ls -1 /usr/lib/python3.4/site-packages/pycurl*
/usr/lib/python3.4/site-packages/pycurl-7.19.3.1-py3.4.egg-info
/usr/lib/python3.4/site-packages/pycurl.cpython-34m.so
(myvenv)$ ln -s /usr/lib/python3.4/site-packages/pycurl-7.19.3.1-py3.4.egg-info lib/python3.4/site-packages/
(myvenv)$ ln -s /usr/lib/python3.4/site-packages/pycurl.cpython-34m.so lib/python3.4/site-packages/
(myvenv)$ python3
>>> import curl
>>> curl
<module 'curl' from '/home/username/myvenv/lib/python3.4/site-packages/curl/__init__.py'>
Finally.
This is what you need?
Personally, I categorically reject you from this, because, firstly, it is a perversion.
Also, I doubt this method works for everything. The depth of dependencies can be scary.
Secondly, I see no reason to avoid --system-site-packages
, which works like a charm in all the cases I have learned about.
Packages installed pip
in the user's home directory take precedence over system-wide packages, so they are taken first on execution import
. In virtualenv they are only imported if the virtualenv was created with the option --system-site-packages
. Packages installed pip
inside virtualenv have an even higher priority.
Honestly, I only wrote this answer to ask: why do you want to screw it all up instead of working pip
. I can't write comments on SO yet, so I wrote a complete answer. :)
But you will also drag and drop all the other system packages you don't need!
No you will not. You load the module into memory only at runtime import
. If you mean the package will be present in the virtualenv - again, it won't. If you looked inside myvenv/lib/python3.4/site-packages/
, you would see that there are only two packages inside - pip
and setuptools
(the last one is split into several directories). The Python interpreter inside virtualenv simply loads the system-wide modules like a regular interpreter does.
PS The above code also applies to Python 2.7. Just replace pip3
with pip2
, python3
for python2
, 3.4
for 2.7
, etc.
PPS Instead of symbolic linking, you can copy the modules with their dependencies in virtualenv to their respective directories. It makes sense when used --always-copy
, so that your virtualenv is portable.
PPPS If you decide to use virtualenv with --system-site-packages
and pip
, and let's say you want a requests
newer package than the one already installed in /usr/lib
, use pip install -I
as described here .
source to share