Installation details
The scheduler and the web component are running on Kubernetes, using a custom Airflow container image. Our image consists of the following components:
- An Airflow installation for a specific version
- The Conveyor plugin that enables using our custom operators, sensors, and macros
- A number of additional Airflow providers
We make sure that our plugin as well as the additional providers, which extend the functionality of Airflow as described
here, are compatible with the deployed Airflow version.
As a consequence, when writing Airflow DAGs, you are limited to the specified providers, and we do not support extending this.
Since Conveyor runs every task in its own container, you can always package your code in a container and run it with our ConveyorContainerOperator.
This way we separate the responsibilities of the Conveyor platform from functionality that users want to run using Conveyor. The Conveyor team can still perform upgrades and only needs to manage a single Airflow version as all environments run the same one. Users can still write code in their preferred language and package and run it by using a custom Docker image.