Hi,
I use Airflow to migrate data from MySQL DB tables to GCP BigQuery tables. I just realized that some of our DAG’s have been passing but no data has been moving across (Even though we have new loads on MySQL). I tried checking my queries on Airflow (This is clicking on the " Rendered" button) and I get the view attached
When I look at the deployed code, all the queries are there.
Does anyone have an idea what could be causing this?
Regards.
Hi @Makabata! Thanks for reaching out.
If data isn’t passing through from MySQL to GCP as expected it’s likely something within the DAG code itself or the connection set up with BigQuery or MySQL. It’s hard to say without more context on your use case, code, and environment but feel free to check out our Airflow Guides or Example DAGs/Plugins for some general direction.
Hi @paola. Thank you for your reply.
You were right on the mark. It was indeed a code issue. Found a script which was meant to delete duplicates but was deleting all rows.
Thanks again
