Resources

Project resources

Introduction

Project resources allow you to define and create external resources that are associated with your project. Some examples:

• The database used to store the data products belonging to your project
• The AWS IAM role that will be used to access an S3 bucket

External resources are expressed in Terraform and are part of your project source. By default, project resources are stored in the resources folder. You can change this location by overwriting the parameter in your project configuration file.

version: '0.2'
id: 2865729b-2f19-4635-940e-185668d2977a
name: sample-pyspark
template: null
docker:
  path: .
resources:
  path: resources
workflows:
  path: dags

Project resources expect the following variables:

variable "aws_account_id" {}
variable "aws_region" {}
variable "env_name" {}
variable "env_worker_role" {}

The Terraform state will be managed, so you should not configure your own state file. The generated state config will look similar to this:

terraform {
  backend "s3" {}
  required_version = ">= 0.12.8"
  required_providers {
    kubernetes = "v1.11.1"
  }
}

When a project is deployed to an environment, the project resources will be updated before the deployment.

info

You can turn off the creation of project resources at an agent level.

Using a dedicated role to create resources

The agent is allowed to only created a limited set of resources. When your application requires additional rights, use the following steps:

1. Create a new role with a trust policy to the agent task role arn:aws:iam::ACCOUNT_ID:role/conveyor-agent-YYY.

2. Assign the appropriate rights to the role you created.

3. Add provider information to the Terraform resources, to create a provider with an alias. You can then use that special provider in certain resources.

   provider "aws" {
     alias = "your-provider"
     assume_role {
       role_arn     = "arn:aws:iam::ACCOUNT_ID:role/ROLE_THAT_WILL_CREATE_THE_RESOURCES"
     }
   }
   
   resource "aws_sqs_queue" "your_queue" {
     name     = "your-queue-${var.env_name}"
     provider = aws.your-provider
   }