Instances
Conveyor supports the following instances types for any jobs:
| Instance type | Cpu | Total Memory (AWS) | Total Memory (Azure) |
|---|---|---|---|
| mx.nano | 1* | 0.438Gb | 0.375 Gb |
| mx.micro | 1* | 0.875Gb | 0.75 Gb |
| mx.small | 1* | 1.75Gb | 1.5 Gb |
| mx.medium | 1 | 3.5Gb | 3 Gb |
| mx.large | 2 | 7Gb | 6 Gb |
| mx.xlarge | 4 | 14Gb | 12 Gb |
| mx.2xlarge | 8 | 29Gb | 26 Gb |
| mx.4xlarge | 16 | 59Gb | 55 Gb |
| rx.xlarge | 4 | 28Gb | Not supported |
| rx.2xlarge | 8 | 59Gb | Not supported |
| rx.4xlarge | 16 | 120Gb | Not supported |
(*) These instance types don't get a guaranteed full CPU but only a slice of a full CPU, but they are allowed to burst up to a full CPU if the cluster allows.
The numbers for AWS and Azure differ because nodes on both clouds run different daemonsets as well as have different reservation requirements set by the provider. We aim to minimize the node overhead as much as possible while still obeying the minimum requirements of each cloud provider.
Spark resources
When running Spark/PySpark applications, only a part of the total memory for the container is available for Spark itself. The details are described in the following tables:
AWS
| Instance type | Cpu | Total Memory (AWS) | Spark memory (AWS) | PySpark memory (AWS) |
|---|---|---|---|---|
| mx.micro | 1* | 0.875Gb | 0.8Gb | 0.6Gb |
| mx.small | 1* | 1.75Gb | 1.6Gb | 1.25Gb |
| mx.medium | 1 | 3.5Gb | 3.2Gb | 2.5Gb |
| mx.large | 2 | 7Gb | 6.4Gb | 5Gb |
| mx.xlarge | 4 | 14Gb | 12.7Gb | 10Gb |
| mx.2xlarge | 8 | 29Gb | 26.7Gb | 21Gb |
| mx.4xlarge | 16 | 59Gb | 54Gb | 42.4Gb |
| rx.xlarge | 8 | 28Gb | 26Gb | 21Gb |
| rx.2xlarge | 16 | 59Gb | 54Gb | 43Gb |
| rx.4xlarge | 16 | 120Gb | 112Gb | 88Gb |
(*) These instance types don't get a guaranteed full CPU but only a slice of a full CPU, but they are allowed to burst up to a full CPU if the cluster allows.
Azure
| Instance type | Cpu | Total Memory (Azure) | Spark memory (Azure) | PySpark memory (Azure) |
|---|---|---|---|---|
| mx.micro | 1* | 0.75 Gb | 0.69Gb | 0.55Gb |
| mx.small | 1* | 1.5 Gb | 1.38Gb | 1.1Gb |
| mx.medium | 1 | 3 Gb | 2.75Gb | 2.15Gb |
| mx.large | 2 | 6 Gb | 5.5Gb | 4,3Gb |
| mx.xlarge | 4 | 12 Gb | 11Gb | 8.6Gb |
| mx.2xlarge | 8 | 26 Gb | 23.6Gb | 18.6Gb |
| mx.4xlarge | 16 | 55 Gb | 50Gb | 35.7Gb |
*These instance types don't get a guaranteed full CPU but only a slice of a full CPU, but they are allowed to burst up to a full CPU if the cluster allows.
As you can see from the tables, the supported executor memory configs change depending on using regular (Scala) Spark or PySpark.
The explanation for this can be found in the spark.kubernetes.memoryOverheadFactor which can be found in the
settings here.
This is set to 0.1 for JVM jobs (Scala and Java Spark), and to 0.4 for non-JVM jobs (PySpark, SparkR).
A portion of the memory is set aside for non-JVM things like: off-heap memory allocations, system-processes, Python, R...
Otherwise, your job would commonly fail with the error "Memory Overhead Exceeded".