Why is easy_install modifying the Python module download path correctly, but pip and .tar.gz source not?

I am using Apple factory-installed Python 2.7.5 distribution on a Mac laptop running OS X 10.9.4.

This particular distribution comes with factory-loaded versions of several popular Python add-ons such as numpy , many of which live in a directory /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python

.

I find that I can update the version numbers of these preloaded packages, but only if I use the Apple factory-installed, update utility /usr/bin/easy_install

. If I instead try to update using either pip or directly from the .tar.gz source using python setup.py install

, then the installation will fail in a very specific way, which I will detail below. I would like to fix this problem so that I can update Python packages using whatever method I like.

First, a description of what I think is "correct": when I update numpy with easy_install, it creates a new subdirectory numpy-1.9.0-py2.7-macosx-10.9-intel.egg

underneath /Library/Python/2.7/site-packages

, for example:

$> ls -l /Library/Python/2.7/site-packages | grep numpy
drwxr-xr-x   4 root  wheel     136 Sep 11 10:21 numpy-1.9.0-py2.7-macosx-10.9-intel.egg

      

        
        
        
      

    

and then the easy_install utility also modifies the Python module load path so that a "fresh" numpy install takes precedence over the default factory version; for example, inside python, it looks like this:

In [1]: from pprint import pprint

In [2]: import sys

In [3]: pprint(sys.path)
['',
 '/usr/local/bin',
 '/Library/Python/2.7/site-packages/PyVCF-0.6.7-py2.7-macosx-10.9-intel.egg',
 '/Library/Python/2.7/site-packages/distribute-0.7.3-py2.7.egg',
 '/Library/Python/2.7/site-packages/setuptools-3.3-py2.7.egg',
 '/Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.9-intel.egg',
 '/Library/Python/2.7/site-packages/wheel-0.24.0-py2.7.egg',
 '/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg',
 '/Library/Python/2.7/site-packages/numpy-1.9.0-py2.7-macosx-10.9-intel.egg',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 '/Library/Python/2.7/site-packages',
 '/Library/Python/2.7/site-packages/IPython/extensions',
 '/Users/stachyra/.ipython']

In [4]: import numpy

In [5]: numpy.__file__
Out[5]: '/Library/Python/2.7/site-packages/numpy-1.9.0-py2.7-macosx-10.9-intel.egg/numpy/__init__.pyc'

In [6]: numpy.__version__
Out[6]: '1.9.0'

      

        
        
        
      

    

If I uninstall numpy, for example using sudo pip uninstall numpy

(I couldn't figure out how to do uninstall with easy_install), it removes numpy from /Library/Python/2.7/site-packages

, and the factory-installed "default" is /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python

still there.

To observe the "wrong" installation behavior, I can re-update the wpy70 version loaded by the factory, this time with sudo pip install -U numpy

. This creates a new pair of directories in /Library/Python/2.7/site-packages

, for example:

$> ls -l /Library/Python/2.7/site-packages | grep numpy
drwxr-xr-x  35 root  wheel    1190 Sep 11 10:24 numpy
drwxr-xr-x   8 root  wheel     272 Sep 11 10:24 numpy-1.9.0.dist-info

      

        
        
        
      

    

However, when I look at the download path in python and then load numpy and look at the version number and file for numpy, I get the following output:

In [1]: from pprint import pprint

In [2]: import sys

In [3]: pprint(sys.path)
['',
 '/usr/local/bin',
 '/Library/Python/2.7/site-packages/PyVCF-0.6.7-py2.7-macosx-10.9-intel.egg',
 '/Library/Python/2.7/site-packages/distribute-0.7.3-py2.7.egg',
 '/Library/Python/2.7/site-packages/setuptools-3.3-py2.7.egg',
 '/Library/Python/2.7/site-packages/PyYAML-3.11-py2.7-macosx-10.9-intel.egg',
 '/Library/Python/2.7/site-packages/wheel-0.24.0-py2.7.egg',
 '/Library/Python/2.7/site-packages/pip-1.5.6-py2.7.egg',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
 '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
 '/Library/Python/2.7/site-packages',
 '/Library/Python/2.7/site-packages/IPython/extensions',
 '/Users/stachyra/.ipython']

In [4]: import numpy

In [5]: numpy.__file__
Out[5]: '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/numpy/__init__.pyc'

In [6]: numpy.__version__
Out[6]: '1.6.2'

      

        
        
        
      

    

As you can see, based on the above, it looks like pip does not update the sys.path variable with the new updated version of numpy, but instead loads the latest one (numpy 1.9.0), python instead loads the older default factory version, 1.6.2.

Apart from the above two methods, I can also install numpy directly from the sourceforge .tar.gz files using sudo python setup.py install

, in which case I get very similar output as when using pip.

So my question is, in the first place, what exactly is being violated here? Is this the default Apple factory version of Python to install pre-loaded Python packages to some other non-standard directory scope ( /System/Library/Frameworks/Python.framework/Versions/2.7

) besides the normal default ( /Library/Python/2.7/site-packages

)? Or is it pip and source .tar.gz files if you weren't able to reverse the default Apple factory installation correctly?

Also, the second part of my question is, how do I fix this problem so that I can use pip and .tar.gz in the future if I want to? And FWIW, if it turns out in the answer that it includes uninstalling the Apple factory by default, then I would rather not go into the directory /System/Library/Frameworks

and just start manually deleting things as that looks like a potential recipe for hacking things.

One more thing: I know I could work around this whole problem simply by using another alternative distribution like Anaconda or Enthought Canopy , however it seems cleaner and more elegant to me to fix the problem "correctly" if possible.