Skip to main content

Can I use the Airflow CLI with Conveyor?

We do not support using the Airflow CLI directly in Conveyor.

The most popular Airflow CLI command is the backfill command, for which the next section describes the alternatives.

How to perform a backfill with Conveyor?​

Using a backfill DAG​

If you want to perform a backfill, we recommend to create a backfill DAG with a fixed start and end date. This way you will have better visibility on the state of the backfill.

Using the Conveyor CLI​

If you do not want to deploy a separate DAG for backfilling, you can mimic the same behavior through conveyor run. In order to achieve this, you can refer to the following shell script example for Linux:

start=2024-01-01T00:01:00Z
end=2024-01-10T00:01:00Z
while ! [[ $start > $end ]]; do
    echo $start
    conveyor run --env <environment> --skip-dag-validation --dag <dag-name> --task <task-name> --execution-date $start
    start=$(date --date="$start + 1 days" +"%Y-%m-%dT%H:%M:%SZ")
done
note

In the provided script we increment the start with one day, but you can change this to the frequency you like.

note

If you are using macOS, the date command should look as follows: $(date -u -j -v +1d -f "%Y-%m-%dT%H:%M:%SZ" "$start" +"%Y-%m-%dT%H:%M:%SZ").