How to get JobID to start airflow?
When we do dagrun, in the Airflow UI, in Graphics View, we get the details of each job.
The JobID is something like "schedule__2017-04-11T10: 47: 00" .
I need this JobID to track and generate logs, in which I keep track of the times when each task / dagrun was executed.
So my question is , how can I get the JobID within the same dag that is being executed .
Thanks, Chetan
source to share
This value is actually named run_id
and can be accessed through context or macros.
In a python statement this is accessed through the context, and in a bash statement it is accessed using the jinja template in the field bash_command
.
More information on what's available in macros:
https://airflow.incubator.apache.org/code.html#macros
More info on jinja:
https://airflow.incubator.apache.org/concepts.html#jinja-templating
from airflow.models import DAG
from datetime import datetime
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
dag = DAG(
dag_id='run_id',
schedule_interval=None,
start_date=datetime(2017, 2, 26)
)
def my_func(**kwargs):
context = kwargs
print(context['dag_run'].run_id)
t1 = PythonOperator(
task_id='python_run_id',
python_callable=my_func,
provide_context=True,
dag=dag
)
t2 = BashOperator(
task_id='bash_run_id',
bash_command='echo {{run_id}}',
dag=dag)
t1.set_downstream(t2)
Use this dag as an example and check the log for each statement, you should see run_id
it printed in the log.
source to share