Skip to main content

Projects

Projects are a Conveyor concept to group together all the business logic for a particular usecase. Projects consist of code artifacts and information about how these artifacts need to be run. In the case of batch jobs this will be:

  • Spark or python code that is packaged in a container, and
  • scheduling and run information written in python that create a set of Airflow dags (direct acyclical graph)

Once a project is deployed to an environment, the artifacts will be scheduled according to its configuration.

Every time a project is built, a build ID or version ID is generated that corresponds to a specific container version and version of the Airflow DAGs. To actually run the code, a specific build ID needs to be deployed to an environment. Different environments can, and typically do, run different versions of the same project.

Like a jar, a single project can contain multiple executable entry points.

Creating a project​

In order to create a new project, run the following CLI command:

conveyor project create --name <some-name>

For the full details about how to create projects, please have a look at the CLI documentation.

Defining a default IAM identity​

Conveyor supports setting a default IAM at the project level, which will be used by any Airflow job that does not explicitly override the default.

The default IAM identity can be set and changed using the Conveyor CLI. Note that the identity can be templated with the environment name, which means you can adapt the role that is used based on the environment that your jobs are running in.