I am using airflow with GCP’s Cloud Composer.
I am creating data warehouse in BigQuery.
I actually prefer to use the python client for BigQuery and execute my sql statement through it, rather than using the provided BigQuery operator.
I run all of my python scripts with the bash operator.
Everything works fine so far. I can also have more freedom to do what I want in that way.
My question is: is this the best way to execute BigQuery statements ?
Are there possible drawbacks’s ?
On the other hand, I am using also GCP’s Data Fusion with airflow, however the operator provided for starting a pipeline is not working in my environment.
Again, I am using the bash operator and making api calls to Data Fusion instance directly, and all works fine.
Again, my question what are the possible drawback’s?
Or maybe just using bash operator is just fine?
What are your opinions ?
Thanks a lot in advance!