Skip to main content

Common pitfalls

DAGs with a dynamic start date​

Creating DAGs with a dynamic start date (f.e. datetime.now()) often results in confusing behaviors, and in general it is recommended against. Unless you have a very strong reason why your DAG needs a dynamic start date, you should probably initialize your DAG with a fixed start date.

If you are sure that the pattern you want to implement requires a dynamic start date, you should take the following into account. The Airflow scheduler will trigger the run of a task once the period for that task closes. A period is considered closed when the end of the period (the value for data_interval_end) is in the past, but also that the start time of the period (the value for data_interval_start) is after the start date of the DAG.

For example, if you create a DAG with a schedule of @daily, the period for its tasks will have a duration of 24 hours (one day). This means that a task from this DAG will only be considered closed if the start date of the DAG is at least 24 hours in the past. To avoid issues regarding scheduler latency (tasks are typically not started at the exact moment that the period closes), and potential longer or shorter periods when DST changes, it is recommended to place the start date further back in time than the duration of the period by at least 2 hours.

Creation of DAG runs after pausing​

When a DAG is paused, no new DAG runs will be created. When a DAG with a fixed start date is resumed, the Airflow scheduler will detect that there are missing runs (more specifically, runs for which the period is closed). It will then proceed with creating those runs, ensuring your schedule is complete.

For DAGs with a dynamic start date, this is not the case. When resuming the DAG, the scheduler will not consider the runs that were not created as missing, given that the start date has moved, and their periods are not considered closed. No tasks will be created for the time when your DAG was paused.

Depending on your design, this might be a desirable feature or not, but it is important to be aware of this behavior.

Airflow FAQ​

If you did not find your answer in the sections above, you can also refer to the Airflow FAQ for possible explanations.