Instances
Conveyor supports the following instances types for any jobs:
| Instance type | CPU | Total Memory (AWS) | Total Memory (Azure) |
|---|---|---|---|
| mx.nano | 1* | 0.438 Gb | 0.375 Gb |
| mx.micro | 1* | 0.875 Gb | 0.75 Gb |
| mx.small | 1* | 1.75 Gb | 1.5 Gb |
| mx.medium | 1 | 3.5 Gb | 3 Gb |
| mx.large | 2 | 7 Gb | 6 Gb |
| mx.xlarge | 4 | 14 Gb | 12 Gb |
| mx.2xlarge | 8 | 29 Gb | 26 Gb |
| mx.4xlarge | 16 | 59 Gb | 55 Gb |
| cx.nano | 1* | 0.219 Gb | Not supported |
| cx.micro | 1* | 0.438 Gb | Not supported |
| cx.small | 1* | 0.875 Gb | Not supported |
| cx.medium | 1 | 1.75 Gb | Not supported |
| cx.large | 2 | 3.5 Gb | Not supported |
| cx.xlarge | 4 | 7 Gb | Not supported |
| cx.2xlarge | 8 | 14 Gb | Not supported |
| cx.4xlarge | 16 | 29 Gb | Not supported |
| rx.xlarge | 4 | 28 Gb | Not supported |
| rx.2xlarge | 8 | 59 Gb | Not supported |
| rx.4xlarge | 16 | 120 Gb | Not supported |
(*) These instance types don't get a guaranteed full CPU but only a slice of a full CPU, but they are allowed to burst up to a full CPU if the cluster allows.
The numbers for AWS and Azure differ because nodes on both clouds run different DaemonSets and have different reservation requirements set by the provider. We aim to minimize the node overhead as much as possible while still obeying the minimum requirements of each cloud provider.
Spark resources
When running Spark/PySpark applications, only a part of the total memory for the container is available for Spark itself. The details are described in the following tables:
AWS
| Instance type | CPU | Total Memory (AWS) | Spark memory (AWS) | PySpark memory (AWS) |
|---|---|---|---|---|
| mx.micro | 1* | 0.875 Gb | 0.8 Gb | 0.6 Gb |
| mx.small | 1* | 1.75 Gb | 1.6 Gb | 1.25 Gb |
| mx.medium | 1 | 3.5 Gb | 3.2 Gb | 2.5 Gb |
| mx.large | 2 | 7 Gb | 6.4 Gb | 5 Gb |
| mx.xlarge | 4 | 14 Gb | 12.7 Gb | 10 Gb |
| mx.2xlarge | 8 | 29 Gb | 26.7 Gb | 21 Gb |
| mx.4xlarge | 16 | 59 Gb | 54 Gb | 42.4 Gb |
| cx.medium | 1 | 1.75 Gb | 1.6 Gb | 1.25 Gb |
| cx.large | 2 | 3.5 Gb | 3.2 Gb | 2.5 Gb |
| cx.xlarge | 4 | 7 Gb | 6.4 Gb | 5 Gb |
| cx.2xlarge | 8 | 14 Gb | 12.7 Gb | 10 Gb |
| cx.4xlarge | 16 | 29 Gb | 26.7 Gb | 21 Gb |
| rx.xlarge | 8 | 28 Gb | 26 Gb | 21 Gb |
| rx.2xlarge | 16 | 59 Gb | 54 Gb | 43 Gb |
| rx.4xlarge | 16 | 120 Gb | 112 Gb | 88 Gb |
(*) These instance types don't get a guaranteed full CPU but only a slice of a full CPU, but they are allowed to burst up to a full CPU if the cluster allows.
Azure
| Instance type | CPU | Total Memory (Azure) | Spark memory (Azure) | PySpark memory (Azure) |
|---|---|---|---|---|
| mx.micro | 1* | 0.75 Gb | 0.69 Gb | 0.55 Gb |
| mx.small | 1* | 1.5 Gb | 1.38 Gb | 1.1 Gb |
| mx.medium | 1 | 3 Gb | 2.75 Gb | 2.15 Gb |
| mx.large | 2 | 6 Gb | 5.5 Gb | 4.3 Gb |
| mx.xlarge | 4 | 12 Gb | 11 Gb | 8.6 Gb |
| mx.2xlarge | 8 | 26 Gb | 23.6 Gb | 18.6 Gb |
| mx.4xlarge | 16 | 55 Gb | 50 Gb | 35.7 Gb |
(*) These instance types don't get a guaranteed full CPU but only a slice of a full CPU, but they are allowed to burst up to a full CPU if the cluster allows.
As you can see from the tables, the supported executor memory configs change depending on using regular (Scala) Spark or PySpark.
The explanation for this can be found in the spark.kubernetes.memoryOverheadFactor which can be found in the
Spark settings.
This setting is configured to 0.1 for JVM jobs (Scala and Java Spark), and to 0.4 for non-JVM jobs (PySpark, SparkR).
A portion of the memory is set aside for non-JVM things like: off-heap memory allocations, system-processes, Python, R...
Otherwise, your job would commonly fail with the error "Memory Overhead Exceeded".
Disk space allocation
When an application saves data to disk, it will by default consume disk space from the host that it is running on. It's important to note that this disk space will be shared across all the jobs that are running on the same physical machine. Applications are unable to read each-others files, but a particularly storage-hungry application might consume all available disk-space, potentially causing issues for other jobs running on the same host machine.
Applications requesting a T-shirt size of mx.xlarge or greater will get the "full" instance assigned.
This means that no other applications will be deployed on that instance and will thus not suffer from a "noisy neighbor" problem.
Applications running on smaller instance sizes will receive a slice of a physical machine,
and share the amount of available disk space (about 50GB of allocatable space).
To avoid this issue, you can provision application-specific storage
by specifying the disk_size (and optionally disk_mount_path) when using the ContainerOperator.
Spark applications can make use of the equivalent executor_disk_size when using the SparkSubmitOperator.
This setting will provision additional storage for each executor, which will then be automatically used by Spark.