Use the dbt task factory with custom dbt adapters

Description

dbt supports many adapters to integrate with all kinds of databases. The dbt image of Conveyor only packages a couple of them. More information on the supported adapters can be found in the technical reference.

When you want to use the ConveyorDbtTaskFactory, you need to generate the manifest file such that the factory knows which models exist in your project as well as the dependencies between them. If you started your project from our dbt template, you will have a Makefile that exposes the make manifest. If you use a database type, specified in the profiles.yml that is not packaged in our default image the command will fail.

How to fix it

Make your own dbt image available containing the necessary adapters. We start from a Debian slim Python image and then install the necessary dependencies using a requirements file. Change the Makefile in your project to use your own image instead of the public.ecr.aws/dataminded/dbt:<version> in the Docker dbt commands.