Sharing python modules between apps with git or svn: externals helper modules
We use subversive activities in our company. We have different python modules (native and third party) in different versions used. The different applications we develop have different dependencies on the version of the shared modules.
One possibility is to use the virtualenv
installation of modules from the local pypi server. So on every initial check, we need to create a virtualenv, activate it, and install the dependent modules from requirements.txt.
Disadvantages:
- A relatively complex operation for a simple task such as checking and running
- Your way to skip creating virtualenv and working with modules installed in site packages
- Necessity for local pypi server (ok, otherwise you can use urls pointing to your vcs)
So, we came up with another solution, and I ask for your opinion: On the way to the application, we use svn: externals (aka git subodules) to "link" to the specified module (from it the release path and with the specified revision number to save it only for read), so the module will be placed locally in the application path. "Import mylib" will work as it was installed in python sites or virtualenv. This can be extended to even release a release of wx, numpy and other commonly used libraries to our repository and link them locally.
Benefits:
- After initial verification, your readiness to launch (really important point to me)
- depend on version (for example, .txt requirements)
Topical question: Are there projects on github / sorceforge using this scheme? Why is everyone using virtualenv instead of this (apparently) simpler schema? I've never seen a solution like this, so maybe we missed the point?
PS: I have posted this already on the pypa-dev mailing list, but this seems to be the wrong place for a question like this. Please excuse this post.
source to share
On the path to the application, we use svn: externals (aka git subodules) to "link" to the specified module (from there the release path and with the specified revision number to keep it read-only), so the module will be hosted locally in the application path.
This is the more traditional method of managing package dependencies, and the simpler of the two software options that is only used internally. Concerning...
After initial verification, you are ready to run
... this is not strictly true. If one of your dependencies is a Python library written in C, it needs to be compiled first.
We tried it with git subodule function, but it is not possible to get the subpath of the repository (for example
/source/lib
)
It's pretty easy to get around if you checkout the entire repository in your outside location PYTHONPATH
and then just link links to the required files or directories inside yours PYTHONPATH
, although this requires you to use filesystems that support symbolic links.
For example with layout like ...
myproject
|- bin
| |- myprogram.py
|
|- lib
| |- mymodule.py
| |- mypackage
| | |- __init__.py
| |
| |- foopackage -> ../submodules/libfoo/lib/foopackage
| |- barmodule
| |- __init__.py -> ../../submodules/libbar/lib/barmodule.py
|
|- submodules
|- libfoo
| |- bin
| |- lib
| |- foopackage
| |- __init__.py
|
|- libbar
|- bin
|- lib
| barmodule.py
... you my_project/lib
only need to have in PYTHONPATH
, and everything should import correctly.
Are there projects on github / sourceforge using this schema?
The submodules information is simply stored in a file called .gitmodules
and quickly googled for "site: github.com.gitmodules" returns quite a few results.
Why is everyone using virtualenv instead of this (apparently) simpler schema?
For packages published to PyPI and installed with pip
, this is arguably easier in terms of dependency management.
If your software has a relatively simple dependency graph like ...
myproject
|- libfoo
|- libbar
... it doesn't matter, but when it gets bigger ...
myproject
|- libfoo
| |- libsubfoo
| |- libsubsubfoo
| |- libsubsubsubfoo
| |- libsubsubsubsubfoo
|- libbar
|- libsubbar1
|- libsubbar2
|- libsubbar3
|- libsubbar4
... you may not want to take responsibility for developing which versions of all these subpackages are compatible if you need to update libbar
for whatever reason. You can delegate this responsibility to the package developer libbar
.
In your particular case, the decision as to whether your solution is correct will depend on the answers to the questions: -
- Are all the external modules you need to use actually available from the repositories
svn
? - Are these repositories properly used
svn:externals
to include compatible versions of any dependencies they need, or if not, are you willing to take responsibility for managing those dependencies yourself?
If the answer to both questions is yes, then your solution is probably right for your case.
source to share