Skip to main content

Packages

Packages are a concept in Conveyor that facilitate the sharing of common components across different projects and environments. Example use cases are reusable operators for alerting or configuration-driven jobs for data quality measurement.

Packages are standalone​

A package is not part of a project, and projects do not build or release packages. A package is defined by its own .conveyor/package.yaml, is registered once with conveyor package create, and is released with the conveyor package CLI commands. Usually it lives in its own folder or repository. Projects then import the package in their DAGs.

Unlike projects, the deployment of a package is not bound to a single environment. Every released package version is made available across all environments at the same time.

What a package contains​

Like a project, a package can consist of two parts:

  • Scheduling code: Python code in the /pkgs folder, which is loaded in Airflow. Projects import it in their DAGs with packages.load(). Typically, this is a preconfigured operator, but it can also be a set of helper functions for building DAGs.
  • A container image (optional): built from the Dockerfile of the package. This image contains the application logic, and can be written in any language.

A package without a Dockerfile only ships scheduling code. A package with a Dockerfile usually exposes an operator that sets its image with packages.image(). Without it, the task runs in the image of the project that uses the operator. See the example.

How projects use a package​

A project imports a package in its DAGs with packages.load(). The package's code then runs in one of two places:

  • In Airflow: the scheduling code from /pkgs, such as helper functions and operator definitions. This code can only use the Python libraries already installed in Airflow (see Airflow installation details). Neither projects nor packages can add libraries to Airflow.
  • In a container: when a container task runs, it runs in the image its operator specifies, which is your project's image by default. Operators from a package that ships a container image use the package image instead, so those tasks run in the package image and your other container tasks still run in your project image.

The package image is not a library you import, it is a task you schedule. To share code that you import inside your own containers, see Conveyor packages or Python libraries.

As a rule of thumb, Airflow should only handle scheduling. Anything else, such as communicating with external systems or logic that needs extra dependencies, belongs in a container image, where you are in full control of the environment.

Conveyor packages or Python libraries​

A Conveyor package is not the only way to share code between projects. Choose based on where the shared code needs to run:

  • Use a Conveyor package to share a reusable task, for example a custom operator backed by a common container image, or helpers for writing DAGs. This replaces setups where a dedicated project is deployed to every environment only so that other projects can use its image.
  • Use an internal Python library to share application code that runs inside the containers of your projects, for example a function that uploads the output of a processing job to an external system. Publish the library to a private registry (for example AWS CodeArtifact, GitHub Packages, GitLab Package Registry or JFrog Artifactory), and install it in the Dockerfile of every project that needs it, together with its other dependencies.

Versions​

Packages are versioned artefacts and are intended to follow semantic versioning. Additionally, these versions are immutable, meaning that once a package artefact has been published with a certain version, that same version can never be (re-)assigned to another build of the same package. This ensures stability for your users, as a declared dependency will not be able to change or break.

Projects declare which versions they accept: an exact version, a range, or the latest release when no version is set. When a matching version is released, consuming DAGs pick it up without redeploying the project.

Discovering packages​

Package names are unique within your organisation. conveyor package create returns an error if the name is already taken.

To find out which packages are available, and which versions they have, open the Packages page in the Conveyor UI. The page also shows the usage documentation of each version. From the CLI, conveyor package list lists all packages.

Using packages in your projects​

For more information on how to use packages in your development flow, please refer to our how-to guide.