Clusters

Conveyor executes batch and streaming jobs on a Kubernetes cluster that resides in the customers cloud environment (AWS or Azure). In order to further separate the infrastructure used by different workloads, Conveyors support the creation of multiple Kubernetes clusters for the same customer on AWS. (Multiple clusters are not yet supported on Azure, contact us to learn more).

This can be used to separate production from non-production workloads and eliminate the possibility of a wrongly configured job running on a non-production cluster to impact resources used for production jobs. This cannot be guaranteed when all workloads are run on the same Kubernetes cluster. Additionally, it also provides a mechanism to separate IAM permissions between non-production and production data, since every cluster has its own OIDC issuer. More information on assigning IAM permissions: AWS or Azure details.

Clusters are provisioned by the Conveyor team. Users only need to specify the correct cluster when they create a new environment. A cluster can have multiple environments but an environment can only exist on a single cluster.