Reproducible test frames
I'm looking for a test or integration framework that supports lengthy expensive tests for correctness. Tests should only be rerun if the code that affects the test has changed.
Ideally the test framework would be
- find the test code
- creates a hash from it,
- run the code and write to output file with hash as name
- or skip if it already exists.
- provide a simple overview of which tests were successful and which failed.
It would be nice if the test should specify the modules and files it depends on.
Python would be ideal, but the problem can be high enough for other languages ββto work too.
Perhaps there is already an integrated test or build environment that I can adapt to accommodate this behavior?
source to share
Basically you need to keep track of what the test is so you can check if it has changed.
Python code can be traced with sys.settrace(tracefunc)
. There is a module trace
that can help with it.
But if it's not just Python code - if the tests are carried out other programs, test input files, etc., and you also need to keep track of the changes you will need to track at the operating system level, for example strace
, dtrace
, dtruss
.
I created a small demo prototype of a simple test environment that only runs tests that have changed since the last run: https://gist.github.com/messa/3825eba3ad3975840400 It uses a module trace
. It works like this:
- collect tests, each test is identified by name
- Load fingerprints from JSON file (if available)
- for each test:
- If the fingerprint matches the current bytecode of the features listed in the fingerprint, the test is skipped
- execute test otherwise
- track it while it is running, record all called functions
- create test fingerprint with function names and MD5 bytecode hashes of each recorded function
- save updated test fingerprints in JSON file
But there is one problem: it is slow . Running code when traced with is trace.Trace
about 40 times slower than without tracing. So, you might be better off running all tests without tracing :) But if the tracer is implemented in C, for example, it's in the coverage
/ a> module it should be faster. (Python module is trace
not in C.)
Maybe some other tricks can help with speed. Maybe you are only interested in the top-level function, whether they have changed or not, so you don't need to keep track of all the function calls.
Have you considered other ways to speed up expensive tests? Like parallelization, ramdisk (tmpfs) ... For example, if you are testing against a database, do not use "system" or "one", but start a special instance of the database with light configuration (no prealloc, no journal ...) from tmpfs ... If possible, of course - some tests should be performed with a configuration similar to production.
Some test frameworks (or their plugins) can only run the tests that failed the last time - this is different, but kind of similar functionality.
source to share
This may not be the most efficient way to do it, but it can be done using a Python module pickle
.
import pickle
At the end of the file, save it as a pickle.
myfile = open('myfile.py', 'r') #Your script
savefile = open('savefile.pkl', 'w') #File the script will be saved to
#Any file extension can be used but I like .pkl for "pickle"
mytext = myfile.readlines()
pickle.dump(mytext, savefile) #Saves list from readlines() as a pickle
myfile.close()
savefile.close()
And then at the beginning of your script (after you've pickled it already once) add a bit of code that checks it for pickle.
myfile = ('myfile.py', 'r')
savefile = ('savefile.pkl', 'r')
mytext = myfile.readlines
savetext = pickle.load(savefile)
myfile.close()
savefile.close()
if mytext == savetext:
#Do whatever you want it to do
else:
#more code
This should work. It's a little longer, but it's pure python and should do what you're looking for.
source to share