Developing with the Conveyor Airflow plugin
We currently offer two approaches to ease developing DAGs with our Airflow plugin. We do not recommend using both methods simultaneously within the same project, as this could result in unforeseen issues.
Installing the Conveyor type stubs
The easiest method is to use our types-conveyor package. This package contains our public interfaces, including their type hints. Installing the package enables your IDE to perform autocompletion for our methods, as well verify type correctness.
As this package is provided via the official PyPI channel, you can simply run pip install types-conveyor in the Python environment you use for linting your DAGs.
In case you use other tools that connect to PyPI (such as Poetry or Pipenv), these will also behave as expected.
It's worth noting that types-conveyor doesn't provide any function implementations, and hence can't be used for testing the behavior of your DAGs. If your goal is to create tests, please refer to the next section.
This package is a recent addition to Conveyor, and we're still looking for feedback. If you're using this feature and notice something is not working as expected, please contact use through your support channel.
Installing the Conveyor Airflow plugin
When your Airflow DAGs become more complex, you might want to write tests to validate your logic before deploying to an Airflow environment. If you want to use Conveyor specific objects (operators, sensors, macros,...) in your tests, you will need to install the actual Conveyor Airflow plugin locally, rather than the type stubs only. This can be done through the following command:
pip install https://static.conveyordata.com/pypi/release/conveyor-airflow-plugins-<CONVEYOR_VERSION>.tar.gz
In this command, replace the CONVEYOR_VERSION with the most recent version, as shown in the release notes.
This version is the same one as is used by the environments of Conveyor.
The interfaces of the classes and functions provided by Conveyor do not change often. In case they do change, we ensure that the changes are backwards compatible. This way you do not need to update the dependency with every new release.