Skip to main content

Migrating Datafy to Conveyor

Datafy has been renamed to Conveyor but apart from that little will change for users. We try hard to be backwards compatible, such that your existing code/scripts keep on working. In this page we want to give an overview of the changes related to the rename and how these might impact you.

How to install/update to Conveyor

The installation process remains the same, only the name has changed. At least for a while we will support both the old and the new installation instructions, such that nothing will break immediately.

Linux users

wget https://app.conveyordata.com/api/info/cli/location/linux/amd64 -O conveyor_linux_amd64.tar.gz
tar -zxvf conveyor_linux_amd64.tar.gz
chmod +x bin/linux/amd64/conveyor
cp bin/linux/amd64/conveyor /usr/local/bin/conveyor

Mac users

Install conveyor as follows:

brew install datamindedbe/conveyor-formulas/conveyor

Update conveyor through brew:

brew upgrade datamindedbe/conveyor-formulas/conveyor

For the time being brew upgrade datafy will continue to work.

Running Datafy update

When initially updating the cli through the datafy update command, we will download the latest binary and install it using the name conveyor.

important

After updating or installing conveyor, you should start your cli commands with conveyor instead of datafy.

The airflow operators/macros have been renamed

We renamed the airflow classes to use Conveyor instead of Datafy together with their import paths. For now we will support both the new and the old names.

Class names

New nameOld name
ConveyorContainerOperatorV2DatafyContainerOperatorV2
ConveyorSparkSubmitOperatorV2DatafySparkSubmitOperatorV2
ConveyorContainerSensorDatafyContainerSensor
ConveyorExternalTaskSensorDatafyExternalTaskSensor
ConveyorDbtTaskFactoryDatafyDbtTaskFactory

Imports

New importOld import
conveyor.operatorsdatafy.operators
conveyor.factoriesdatafy.factories
conveyor.sensorsdatafy.sensors
conveyor.secretsdatafy.secrets

Finally, the Airflow macros have also been renamed:

  • macros.conveyor.env()
  • macros.conveyor.image('X')

For information on how to use them, go to the airflow section of the docs.

When running conveyor build we will show warnings when you are still using the old names and imports.

To simplify migrating, we extended the conveyor project upgrade-dags command to also take care of the renaming. Running the command replaces all the old references in the dags folder of your current project.

warning

If you need to stay on Airflow v1, you should not use the upgrade-dags command as it will also migrate your dags to v2. However remember Airflow v1 is deprecated and you should upgrade as soon as you can.

Dot Datafy folder in every project directory

Previously, every project had a .datafy directory where we store some metadata about the project.

Now, new projects will be created with a .conveyor directory. Already existing projects can keep using the .datafy folder or you can rename it yourself if you want.

important

If both a .datafy folder and a .conveyor folder exist, the .conveyor folder gets precedence.

The Conveyor UI will be hosted on app.conveyordata.com and the docs are available at: docs.conveyordata.com. For the time being the old dns domain *.datafy.cloud will redirect to the new domain such that existing bookmarks/links keep on working.

The Airflow API works with both app.conveyordata.com and app.datafy.cloud but we recommend migrating the new domain.

Finally, the emails that we send for alerting as well as inviting new users will now use the conveyordata.com domain.

Dot Datafy folder in home directory

New versions of the cli will move the .datafy folder to .conveyor in your home directory.

important

Users should not use this folder anymore as its primary purpose is for storing tokens and providing advanced configuration options to the Conveyor team.