I had a fundamental questions , when we want to pull data from external sources we use Airflow Plugins ,
My question is Why ?
Simmilar thing can be done by executing external python script from the airflow dags instead of creating a hook and operator.
Is there a reason to prefer Airflow Plugin vs use of external script which is triggered from airflow let’s say using a bash operator?
2 Likes
Airflow plugins are helpful minimizing reusable code. You are welcome to write everything with python operators, but if you are constantly doing the same set of python operators, it may be tough to re-use that code.
Furthermore, using hooks/operators also give you a more natural interface to using airflow’s built in connections manager.
You can find more information on this doc:
2 Likes