Migrating Datafy to Conveyor
Datafy has been renamed to Conveyor but apart from that little will change for users.
We try hard to be backwards compatible, such that your existing code/scripts keep on working.
In this page we want to give an overview of the changes related to the rename and how these might impact you.
How to install/update to Conveyor
The installation process remains the same, only the name has changed. At least for a while we will support both the old and the new installation instructions, such that nothing will break immediately.
Linux users
wget https://app.conveyordata.com/api/info/cli/location/linux/amd64 -O conveyor_linux_amd64.tar.gz
tar -zxvf conveyor_linux_amd64.tar.gz
chmod +x bin/linux/amd64/conveyor
cp bin/linux/amd64/conveyor /usr/local/bin/conveyor
Mac users
Install conveyor as follows:
brew install datamindedbe/conveyor-formulas/conveyor
Update conveyor through brew:
brew upgrade datamindedbe/conveyor-formulas/conveyor
For the time being brew upgrade datafy will continue to work.
Running Datafy update
When initially updating the CLI through the datafy update command, we will download the latest binary and install it using the name conveyor.
After updating or installing conveyor, you should start your CLI commands with conveyor instead of datafy.
The airflow operators/macros have been renamed
We renamed the airflow classes to use Conveyor instead of Datafy together with their import paths.
For now we will support both the new and the old names.
Class names
| New name | Old name |
|---|---|
| ConveyorContainerOperatorV2 | DatafyContainerOperatorV2 |
| ConveyorSparkSubmitOperatorV2 | DatafySparkSubmitOperatorV2 |
| ConveyorContainerSensor | DatafyContainerSensor |
| ConveyorExternalTaskSensor | DatafyExternalTaskSensor |
| ConveyorDbtTaskFactory | DatafyDbtTaskFactory |
Imports
| New import | Old import |
|---|---|
| conveyor.operators | datafy.operators |
| conveyor.factories | datafy.factories |
| conveyor.sensors | datafy.sensors |
| conveyor.secrets | datafy.secrets |
Finally, the Airflow macros have also been renamed:
macros.conveyor.env()macros.conveyor.image('your_image_name')
For information on how to use them, go to the airflow section of the docs.
When running conveyor build we will show warnings when you are still using the old names and imports.
To simplify migrating, we extended the conveyor project upgrade-dags command to also take care of the renaming.
Running the command replaces all the old references in the dags folder of your current project.
.datafy folder in every project directory
Previously, every project had a .datafy directory where we store some metadata about the project.
Now, new projects will be created with a .conveyor directory. Already existing projects can keep using the .datafy folder or you can rename it yourself if you want.
If both a .datafy folder and a .conveyor folder exist, the .conveyor folder gets precedence.
DNS changes related to Conveyor
The Conveyor UI is hosted on app.conveyordata.com and the docs are available at: docs.conveyordata.com.
For the time being the old DNS domain *.datafy.cloud will redirect to the new domain such that existing bookmarks/links keep on working.
The Airflow API works with both app.conveyordata.com and app.datafy.cloud but we recommend migrating to the new domain.
Finally, the emails that we send for alerting as well as inviting new users will now use the conveyordata.com domain.
.datafy folder in the home directory
New versions of the CLI will move the .datafy folder to .conveyor in your home directory.
Users should not use this folder directly as its primary purpose is caching tokens.