Dot Conveyor directory

When working with the Conveyor CLI, you might have noticed the YAML files under the .conveyor directory. The two most important files are: packages.yaml and project.yaml. Both reside in the .conveyor directory of their respective project or package.

packages.yaml format

The .conveyor/packages.yaml file contains several configuration options for interacting with a package from the CLI. This file gets written by Conveyor when a package is created from the CLI. If the file does not exist, or there is an issue with it, you can regenerate the file using:

conveyor package generate-config --name <package-name>

The most important options are the id and name property of the package. These should match with the id and the name in the Conveyor UI/API.

The complete structure is as follows:

docker:
    path: .
id: <some-id>
name: <some-name>
workflows:
    path: pkgs

The additional options are:

• docker.path: This identifies the path to the Dockerfile for your package.
• workflows.path: This identifies the path to the workflows files (e.g. containing the Airflow dags) for your package.

project.yaml format

The .conveyor/project.yaml file contains several configuration options for interacting with a project from the CLI. This file gets written by Conveyor when a project is created from the CLI. If the file does not exist, or there is an issue with it, you can regenerate the file using:

conveyor project generate-config --name <project-name>

The most important options are the id and name property of the project. These should match with the id and the name in the Conveyor UI/API.

The complete structure is as follows:

dependencies:
    environment: dev
    projects:
        - name: common
docker:
    path: .
id: <some-id>
ide:
    filename: ide.yaml
name: <some-name>
resources:
    path: resources
streaming:
    filename: streaming.yaml
template: ""
version: "0.3"
workflows:
    path: dags
    version: "2"

The additional options are:

• dependencies: This block identifies the projects this project depends on. More details on this legacy feature is available in the common projects in Airflow.
• docker.path: This identifies the path to the Dockerfile for your project.
• ide: This block identifies where to find the file with 1 or more ide configurations. More details can be found in: IDE configuration.
• resources: This block identifies the path to the resources directory, which can be used to define IAM roles on AWS. This feature should be explicitly allowed by your organisation before you can use it. More details here.
• streaming: This block identifies where to find the file with 1 or more streaming configurations. More details can be found in: Streaming configuration.
• template: This is used to identify the template that was used to generate the project.
• version: This is used by Conveyor to identify the version of the project configuration file.
• workflows.path: This identifies the path to the workflows files (e.g. containing the Airflow dags) for your project.
• workflows.version: This identifies the default Airflow version ("2" or "3") that Conveyor uses to parse your DAGs for this project, for example with conveyor project build, run and validate-dags. It does not have to match the Airflow version of the environment you deploy to. You can override this default for a single invocation with the --airflow-version flag on those commands. New projects default to "3". Projects created while Airflow 2 was still the default may have this explicitly set to "2" in their project.yaml, in which case you need to update it yourself to start validating/building against Airflow 3 by default. See the Airflow 2 to 3 migration guide for more details.