3. Deploy your project
The project and environment are now in a state where the project can be build and deployed to the environment.
Since we are using Duckdb in this example, which is an in-memory database we do not need to setup one.
3.1 Build the project
The first step is to validate and package your project code in a docker image and upload it. The command is executed in the project folder.
conveyor build
Once this is done you can deploy this project to any environment.
The first time build of a project can take some time depending on your internet connection. Subsequent builds should be significantly faster.
3.3 Deploy the project
Next, deploy the project to the environment:
conveyor deploy --env $ENVIRONMENT_NAME --wait
The --wait flag makes the command wait until the deploy is finished, this is certainly useful in a CI/CD context but also if you want to easily know when a deploy is finished.
3.4 Activate the workflow
In the Conveyor UI navigate to your environment, by clicking environment on the left and clicking on your environment in the list. This will automatically open Airflow.
By default, your workflow will be disabled. Enable it by selecting the toggle on the left next to your DAG. Airflow will now start scheduling your project, one run for each day since the start date you specified when creating the project. For more details on Airflow as well as how Conveyor integrates with Airflow, take a look here
3.5 Explore logs and metrics
Next we can see our runs in the Task executions tab of our environment.
Select the task execution and you will be able to see all the settings of your task.
And underneath you see the logs of your task.
For longer running task CPU and Memory metrics are also available.
3.6 Fix the initial version
The dag fails on the tests of main.my_first_dbt_model. Take a look at the logs in the Task executions tab as well as at the my_first_dbt_model.sql file to see if you can figure out what needs to change.
If you do not see it, look at the comment in the my_first_dbt_model.sql file for a hint.
After fixing the model, run:
conveyor build && conveyor deploy --env $ENVIRONMENT_NAME --wait
Now you can re-trigger the dag by clicking the > and selecting trigger dag in the Airflow UI.
You should see that the task succeeds.
3.7 Run a task from the CLI
To shorten the feedback loop during development you can run a task directly from the cli using the conveyor run command.
This command builds the project and deploys it on the kubernetes cluster, bypassing airflow.
The logs are immediately made available in your console.
conveyor run --env $ENVIRONMENT_NAME --dag $PROJECT_NAME
Great! We just build and deployed our project to an environment for the first time.
This is the main way used by developers to deploy their project to development or production.
And using conveyor run we are able to quickly test out changes without having to deploy our project.