Clever Geek Handbook

Dynamic task creation in DAG Airflow

I want to use Airflow to create client reports, I would like to have one DAG that will go through all clients and run a task to generate its report. The customer list is collected by the first task in the DAG and cannot be hardcoded.

Basically I have a task that uses a database hook in a python statement to create a list. Then, for each item in the list, I would like to perform a task using a python statement and that item is passed as an argument to the python function. Is there some pattern I can use to achieve this?

+3


source to share


1 answer


Trigger_dag concept

Let the task using the database hook in the python statement to create the list be a task in the dag controller and pass each item to the trigger_dag list in the params section.



You will find the link in the examples folder in your airflow installation

Good luck!

0


source






More articles:


All Articles