How to make the actions of the coordinator materialize at a certain frequency?
I would like to know if this is possible / how to force the coordinator to materialize or create a workflow at regular intervals, even if the previous instance of the workflow has not been executed yet.
Let me explain: I have a simple coordinator that looks like this:
<coordinator-app name="myApp" frequency="${coord:hours(3)}" start="2015-01-01T0:00Z" end="2016-01-01T00:00Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.4">
<action>
<workflow>
<app-path>${myPath}/workflow.xml</app-path>
</workflow>
</action>
</coordinator-app>
The frequency is set to 3 hours. Every 3 hours, I expect the coordinator to "materialize" a new instance / job of the workflow.
Here's my problem: when the execution of a workflow lasts more than 3 hours, the coordinator does not materialize a new instance of the workflow, but waits for the current current workflow to finish. Then it will create the following workflow. Worker processes started by the coordinator are queued if they are greater than frequency.
How do I get the coordinator to start a new job every 3 hours no matter what? Thanks you
source to share
You have to use the property concurrency
. By default this is one, so you have queue problems. Install it as much as you think is reasonable.
<coordinator-app name="[NAME]" frequency="[FREQUENCY]"
start="[DATETIME]" end="[DATETIME]" timezone="[TIMEZONE]"
xmlns="uri:oozie:coordinator:0.1">
<controls>
<concurrency>[CONCURRENCY]</concurrency>
</controls>
From the docs:
Concurrency: The coordinator task can specify the concurrency for its coordinator actions, that is, how many coordinator actions are allowed to run simultaneously (RUNNING status) before the coordinator engine starts throttling them.
source to share