Force airflow backfill command
Is there a way that I can do the reverse load sequentially without multitasking? For example, if I run backfill with multiple dates such as Airflow backfill fill [dag] -s "2017-07-01" -e "2017-07-10" is there a way to end each miss before starting the next day ? He now completes all days of each task before moving on to the next task.
Thank.
source to share
You can set max_active_runs
your DAG parameter to 1, which will make sure that only one DAG for that dag starts at a time. https://pythonhosted.org/airflow/code.html?highlight=concurrency#models
If you need your entire dag to complete before moving forward, you can add ExternalTaskSensor
collection tasks at the beginning of your DAG and collection tasks DummyOperator
at the end. Then set the ExternalTaskSensor to run in DummyOperator at the end of the previous run.
dag = DAG(dag_id='dag')
wait_for_previous_operator = ExternalTaskSensor(\
task_id='wait_for_previous',
external_dag_id='dag',
external_task_id='collection',
execution_delta=schedule_interval,
dag=dag)
collection_operator = DummyOperator(\
task_id='collection',
dag=dag)
wait_for_previous_operator.set_downstream(your_other_tasks_list)
collection_operator.set_upstream(your_other_tasks_list)
source to share