Skip to main content

Airflow Configuration

Description​

Using Airflow in Conveyor is pretty hands-off, almost everything related to configuring and running Airflow is managed by the Conveyor team to provide the best out-of-the box experience. However, sometimes you might require different configuration options for Airflow, hence we support configuring a couple of settings. This is typically not needed, but you might have specific requirements for certain environments where increasing the parallelism makes sense.

Configuration options​

Airflow has a long list of configuration options, described here. At the moment we only support changing 2 properties in the core section of the Airflow configuration and 1 property of the Airflow webserver.

  • Parallelism [core]: The parallelism field limits how much tasks Airflow will run in parallel. The default value is 128, but some busy environments might need a higher value. The value can be configured between 16 and 512.

  • Max active tasks per DAG [core]: The setting limits how much tasks can be run in parallel on a single DAG. Always set this value lower than the parallelism setting. Otherwise, one DAG is able to occupy all running slots, causing hindrance for other DAGs. We recommend setting the value to 0.25 * parallelism. The default value is 32, but some busy environments might need a higher value. The value can be configured between 4 and 128.

  • Navigation bar color [webserver]: This setting allows you to change the color of the Airflow navigation bar. This can be useful to distinguish different environments. The production environments could for example be given a distinct color, such that users are visually aware that they are working on production.

Secrets backend​

Airflow allows you to configure an external secrets backend for storing variables and connections. How to configure this in Conveyor is described in the following page.

How to configure them​

All Airflow properties can be changed from the UI as well as in the CLI and Terraform.

UI​

Select an environment and go to the settings tab, which should give you the following page:

Here you can change the Airflow settings.

CLI​

The configuration settings can also be updated through the cli, through the use of the conveyor env update command. For full details take a look at the cli docs

Terraform​

The configuration settings can also be specified in the environment resource in Terraform. For full details take a look at the terraform docs