Environments
A Conveyor environment is where projects live. There can only exist one active build for every project in a given environment.
At most of our customers, we see two different ways in which Conveyor environments are used:
- Companies operate several long-living environments such as a development, testing as well as a production environment.
- Short-lived developer or team environments to test new features in complete isolation from everyone else. This is encouraged by Conveyor because we make it very easy and quick to spin up and destroy environments.
Technically, a Conveyor environment corresponds to a namespace on the underlying Kubernetes cluster. Every Conveyor environment comes with its own, separate Airflow server. For more details see here
Conveyor environments only exist within Conveyor. If you want jobs that run in a certain Conveyor environment to operate on different datasets, you will still have to provide support for this in your code. You can do this by configuring dags to supply different CLI arguments to your program (e.g. docker container) as well as use different IAM permissions depending on the environment you operate in.
Creating a new environment
There are two ways to create new environments in Conveyor:
- Conveyor CLI: run the following command:
conveyor env create --name <some-name> - Conveyor UI: on the environment list page, you can click on
Create a new environment
When creating an environment, you can specify 2 options:
- enable deletion protection: make sure the environment cannot be deleted by accident.
- instance lifecycle: indicates whether the airflow components (web/scheduler) should run either on on-demand or on spot nodes.