Dot Conveyor directory

When working with the Conveyor CLI, you might have noticed the YAML files under the .conveyor directory. The two most important files are: packages.yaml and project.yaml. Both reside in the .conveyor directory of their respective project or package.

packages.yaml format

The .conveyor/packages.yaml file contains several configuration options for interacting with a package from the CLI. This file gets written by Conveyor when a package is created from the CLI. If the file does not exist, or there is an issue with it, you can regenerate the file using:

conveyor package generate-config --name <package-name>

The most important options are the id and name property of the package. These should match with the id and the name in the Conveyor UI/API.

The complete structure is as follows:

docker:
    path: .
id: <some-id>
name: <some-name>
workflows:
    path: pkgs

The additional options are:

• docker.path: This identifies the path to the Dockerfile for your package.
• workflows.path: This identifies the path to the workflows files (e.g. containing the Airflow dags) for your package.

project.yaml format

The .conveyor/project.yaml file contains several configuration options for interacting with a project from the CLI. This file gets written by Conveyor when a project is created from the CLI. If the file does not exist, or there is an issue with it, you can regenerate the file using:

conveyor project generate-config --name <project-name>

The most important options are the id and name property of the project. These should match with the id and the name in the Conveyor UI/API.

The complete structure is as follows:

dependencies:
    environment: dev
    projects:
        - name: common
docker:
    path: .
id: <some-id>
ide:
    filename: ide.yaml
name: <some-name>
resources:
    path: resources
streaming:
    filename: streaming.yaml
template: ""
version: "0.3"
workflows:
    path: dags
    version: "2"

The additional options are:

• dependencies: This block identifies the projects this project depends on. More details on this legacy feature is available in the common projects in Airflow.
• docker.path: This identifies the path to the Dockerfile for your project.
• ide: This block identifies where to find the file with 1 or more ide configurations. More details can be found in: IDE configuration.
• resources: This block identifies the path to the resources directory, which can be used to define IAM roles on AWS. This feature should be explicitly allowed by your organisation before you can use it. More details here.
• streaming: This block identifies where to find the file with 1 or more streaming configurations. More details can be found in: Streaming configuration.
• template: This is used to identify the template that was used to generate the project.
• version: This is used by Conveyor to identify the version of the project configuration file.
• workflows.path: This identifies the path to the workflows files (e.g. containing the Airflow dags) for your project.