Airflow Configuration
Description
Using Airflow in Conveyor is pretty hands-off, almost everything related to configuring and running Airflow is managed by the Conveyor team to provide the best out-of-the box experience. However, sometimes you might require different configuration options for Airflow, hence we support configuring a couple of settings. This is typically not needed, but you might have specific requirements for certain environments where increasing the parallelism makes sense.
Configuration options
Airflow has a long list of configuration options, described here.
At the moment we only support changing 2 properties in the core
section of the Airflow configuration and 1 property of the Airflow webserver.
-
Parallelism [core]: The parallelism field limits how much tasks Airflow will run in parallel. The default value is
128
, but some busy environments might need a higher value. The value can be configured between 16 and 512. -
Max active tasks per DAG [core]: The setting limits how much tasks can be run in parallel on a single DAG. Always set this value lower than the
parallelism
setting. Otherwise, one DAG is able to occupy all running slots, causing hindrance for other DAGs. We recommend setting the value to0.25 * parallelism
. The default value is32
, but some busy environments might need a higher value. The value can be configured between 4 and 128. -
Navigation bar color [webserver]: This setting allows you to change the color of the Airflow navigation bar. This can be useful to distinguish different environments. The production environments could for example be given a distinct color, such that users are visually aware that they are working on production.
Secrets backend
Airflow allows you to configure an external secrets backend for storing variables and connections. How to configure this in Conveyor is described in the following page.
How to configure them
All Airflow properties can be changed from the UI as well as in the CLI and Terraform.
UI
Select an environment and go to the settings tab, which should give you the following page:
Here you can change the Airflow settings.
CLI
The configuration settings can also be updated through the cli, through the use of the conveyor env update
command.
For full details take a look at the cli docs
Terraform
The configuration settings can also be specified in the environment
resource in Terraform.
For full details take a look at the terraform docs