Using the dbt CLI on Conveyor

caution

This functionality is still experimental. Every feedback you can give us is very much appreciated. In case you have suggestions or run into any issues, please contact us via your support channel.

Basic Usage

dbt Core ships with a command-line interface (CLI) for running your dbt project. Using this CLI, you can run dbt commands from your working directory on your computer.

Unfortunately, this functionality might be of little use to you in case you don't have access to your data warehousing backend from your local machine (as is often the case in enterprise environments).

In order to facilitate the local development of dbt projects, while still adhering to your organisations security principles, the Conveyor CLI provides a new subcommand, conveyor dbt, which will behave exactly the same way as the dbt CLI, with the only difference that dbt will actually run inside your Conveyor environment, instead of on your local machine.¹

You can refer to the command docs for the usage instructions, but in short, any command that is a valid dbt CLI command can be prefixed with conveyor to run it remotely. The only addition that is still required is to add an --env flag (as with other conveyor) commands, in order to tell the CLI in which environment you want the command to execute.

Suppose you want to execute the following dbt command to check that your model works as intended:

dbt run --select my_dbt_model

To run this command against your dev environment, you can simply change the command to:

conveyor dbt run --select my_dbt_model --env dev

This will execute the dbt run command remotely, and stream the results back to your terminal, as dbt is running your model.

IAM identity

When executing the dbt command, your task will assume the default IAM role configured on your Conveyor project. If you have not configured a default identity for your project yet, please refer to the documentation for conveyor project update.

Additional configuration

In case you need to pass additional configuration to the dbt adapter you are using, we support setting environment variables through a "fake" dbt adapter.

The example profiles.yml below shows how you can use the conveyor adapter to register values as environment variables, fetching some from the AWS SSM Parameter Store. To use AWS Secrets Manager as the key vault to fetch from, you can simply replace awsSSMParameterStore by awsSecretsManager.

default:
  target: duckdb
  outputs:
    duckdb:
      type: duckdb
      settings:
        s3_region: "{{ env_var('MY_REGION') }}"
        s3_access_key_id: "{{ env_var('S3_ACCESS_KEY_ID') }}"
        s3_secret_access_key: "{{ env_var('S3_SECRET_ACCESS_KEY') }}"

    conveyor:
      type: conveyor
      envVariables:
        MY_REGION:
          value: "eu-west-1"
        S3_ACCESS_KEY_ID:
          awsSSMParameterStore:
            name: /my/secret/parameter
            path: key_id
        S3_SECRET_ACCESS_KEY:
          awsSSMParameterStore:
            name: /my/secret/parameter
            path: access_key

Note that the Conveyor adapter is not an installable package (setting target: conveyor should result in an error), we simply reuse the profiles.yml file as a convenient place to locate these settings.

Footnotes

1. In order to achieve this functionality, we use the same mechanism as used by conveyor run. Your dbt project is packaged and sent to your cloud environment, where it is run as a task on the Conveyor cluster. ↩