Clever Geek Handbook

Monitor / recover for complex schedule of batch processing tasks

I have a set of batch tasks that I need to run automatically on a weekly basis. The weekly basic part is not hard: I use cron to throw things away. Tasks are anything that can be initiated using a shell script. The tricky part is that my tasks have a non-trivial dependency graph (B depends on A, C depends on B, D depends on C and B, etc.). I'm interested in monitoring tasks so that I can be notified when something goes wrong with a task. Better yet, I would like to be able to easily resume the task schedule from the point of failure without restarting the whole thing.

Now I can imagine something by itself for all this to happen, but I have to imagine that someone has already experienced this problem. Are there existing libraries or frameworks that make this task easier? Ideally I am looking for something to work in a linux / unix environment.

+2


source to share


1 answer


I would suggest a Makefile to solve your problems.

You will need to have token files for tasks indicating when they were last completed and work out some dependencies for the "root" tasks (no internal dependencies).



A : 
    <A tasks>
    touch A

B : A
    <B tasks>
    touch B

C : B
    <C tasks>
    touch C

D : B C
    <D tasks>
    touch D

Now you just need to decide why A needs to be redone and you can cron

make that make file and only the relevant bits are redone.

+1


source






More articles:


All Articles