Backfilling a DAG with Conveyor

Using a backfill DAG

If you want to perform a backfill, we recommend to create a backfill DAG with a fixed start and end date. This way you will have better visibility on the state of the backfill.

Using the Conveyor CLI

If you do not want to deploy a separate DAG for backfilling, you can mimic the same behavior through conveyor run. In order to achieve this, you can refer to the following shell script example for Linux:

start=2024-01-01T00:01:00Z
end=2024-01-10T00:01:00Z
while ! [[ $start > $end ]]; do
    echo $start
    conveyor run --env <environment> --skip-dag-validation --dag <dag-name> --task <task-name> --execution-date $start
    start=$(date --date="$start + 1 days" + "%Y-%m-%dT%H:%M:%SZ")
done

note

In the provided script we increment the start with one day, but you can change this to the frequency you like.

note

If you are using macOS, the date command should look as follows: $(date -u -j -v +1d -f "%Y-%m-%dT%H:%M:%SZ" "$start" +"%Y-%m-%dT%H:%M:%SZ").