Is it safe to cache dependencies generated from source in Travis CI

This question is about Travis CI's caching capabilities.

My project ( quamash ) has some pretty strange dependencies. It has to be tested 6 times, once with PySide, once with PyQt4, and once with PyQt5 in Python 3.3 and Python 3.4, in order to really test the whole code. (I also need to test on the * nix platform and the Windows platform, since each uses different asynchronous backends, but that's not really important.)

PySide is easy because it can be installed using pip. I only have the wheels ready (they are now stored on github, but they shouldn't be, maybe cleaner, so they won't)

I can get binary packages for Qt4 (from exact) and Qt5 (from ppa). The PyQt4 package (python3-qt4), which is exact, is for python 3.2, so it's useless. And PyQt5 is completely unavailable.

Installing a global package and then using it anyway.

PyQt4 / 5 have a common dependency in SIP, which is a Python binding generator for C ++. (all 3 are developed by river bank calculations)

With PySide alone, my tests finish in less than a minute, but PyQt takes ~ 5 minutes to build (time 2 for both versions). My guess is that a 10 minute build isn't that bad, but it just seems like a waste since it's mostly spent on dependencies.

Is it safe to cache PyQt and SIP build directories and then just run sudo make install

from them every time? Should I create a .deb file (can I? And then install on virtualenv?)