Skip to main content

Using an external image

Description​

Sometimes, you want to use a third party docker image for one task in your project. This can happen if a third party distributes their own images for any reason, which you need to use in your pipeline. If the third party image is not publicly available, you must make sure that the image can be pulled by Conveyor.

At this point there are multiple ways to accomplish this:

  • Use the Dockerfile of a Conveyor project and specify the third party image in the FROM statement
  • Mirror the third party image in ecr and give Conveyor nodes access to pull these ecr images.

Using a third party image in FROM statement​

This is the most straightforward scenario, as it is similar to what you are used to for any other project. With this approach, you only needs access to the external repository at build time, so you will need to configure credentials when running locally or on CI. The advantage is that Conveyor does not need to manage any credentials.

The downside is that if you use the third party image for a single task and the remaining tasks use a regular Conveyor image, the setup becomes quite complex. The following how-to describes how to use multiple containers in one project.

Mirroring the third party image in ECR to allow pulling​

Alternatively you can make sure that the Conveyor can pull these images when it wants to run them on an EKS cluster. At the moment we do not support user defined image-pull-secrets, but for an ECR repository you can configure permissions.

In order for Conveyor to be able to pull an image from an ECR repository (e.g. conveyor/airflow), do the following:

  • Add the necessary permissions for Conveyor. You can add AWS account IDs to allow cross-account pulling of images.
  • Make sure to add the following actions: ecr:BatchCheckLayerAvailability, ecr:BatchGetImage, ecr:DescribeImages, ecr:DescribeRepositories, ecr:GetAuthorizationToken, ecr:GetDownloadUrlForLayer, ecr:ListImages