Clusters

Conveyor executes batch/streaming jobs in a kubernetes cluster residing in the client's cloud environment (AWS or Azure). In order to further separate the infrastructure used by different workloads, Conveyor support creating multiple kubernetes clusters for a given client on AWS.

This can be used to separate production from non-production workloads and thus eliminate the possibility of a wrongly configured job running on a non-production cluster to impact resources used for production jobs. This cannot be guaranteed when all workloads run on the same kubernetes cluster. Alternatively, it also provides a mechanism to separate IAM permissions between non-production and production data, since every cluster has its own OIDC issuer. More information on assigning IAM permissions: AWS or Azure details.

Clusters are created by the Conveyor team. Users only need to specify the correct cluster when they create a new environment. A cluster can have multiple environments but an environment can only exist on 1 cluster.