Custom Providers

Airflow
1

Creating packages is something I don’t have alot of experience with but i really want to use the provider format, so bear with me here.

If i wanted to create a provider but have it in the same directory as my airflow instance, what would that need to look like? Would it go in the plugins folder? Same level as the plugins folder? How would i add it to the requirements file, or would it be automatically picked up if its in the correct place in the directory?

Any guidance would be greatly appreciated!

2

Yes, you can place a custom operator in the /plugins directory. If placed here, you do not need to add it to requirements.txt as it will be automatically picked up.

For example if you had a hello_operator.py in /plugins with the following custom operator:

from airflow.models.baseoperator import BaseOperator


class HelloOperator(BaseOperator):
    def __init__(self, name: str, **kwargs) -> None:
        super().__init__(**kwargs)
        self.name = name

    def execute(self, context):
        message = f"Hello {self.name}"
        # Print message to task log
        print(message)
        # Value is also returned as an xcom
        return message

You can then use this operator in your dag_abc.py which is in your /dags folder:

from plugins.hello_operator import HelloOperator
...
...
    start = EmptyOperator(
        task_id='start')

    hello = HelloOperator(
        task_id='hello_task',
        name='adam'
    )

start >> hello
3

I appreciate the response, but this was in regards to building custom Providers not Operators

4

Here’s a provider sample template to reference: