Can I use Airflow on Astronomer to orchestrate Talend jobs?

[paola]

You sure can. As a first step, take a look at this Airflow Guide or this Talend guide on integrating Talend with Airflow.

Generally speaking:

• You’ll need to go through steps 1-2 to get Talend itself set up

• Once you’re running Airflow on Astronomer, you’ll be able to swap out the ECSOperator in step 3 with Airflow’s KubernetesPodOperator.

• Check out these docs for a how-to on running the KubernetesPodOperator on Astronomer:

  • On an Astronomer Deployment: https://www.astronomer.io/docs/kubepodoperator/
  • Locally: https://www.astronomer.io/docs/cli-kubepodoperator/

• Talend will export the jobs as images to their repository of choice, and we’d then point the KubernetesPodOperator at those images.

Here’s the source code from that original guide: tmk-tech-blogs/databricks_workflow.py at master · Talend/tmk-tech-blogs · GitHub

[paola]

Update: We recently published a Guide on how to use Talend and Airflow! https://www.astronomer.io/guides/airflow-talend-integration