Capture sqoop output in oozie
I have an Oozie workflow that needs to run every X minutes. It reads the value from the HBase table. Then a Sqoop (incremental) action is performed based on the value read from HBase in the previous step. To make the workflow I need to somehow grab the new -last value from the Sqoop Oozie activity, which will be written back to HBase and the next time the workflow starts reading it again ... etc.
How can I do this, or maybe a better way?
Jonas
source to share
I think the blog http://www.tanzirmusabbir.com/2013/05/chunk-data-import-incremental-import-in.html can give you some hints.
Basically, it stores startindex and chunksize in job.properties and startindex is used where the condition is in the Sqoop job, after which it changes startindex through the shell script after the sqoop job.
source to share