You sure can. As a first step, take a look at this guide on integrating Talend with Airflow.
You’ll need to go through steps 1-2 to get Talend itself set up
Once you’re running Airflow on Astronomer, you’ll be able to swap out the
ECSOperatorin step 3 with Airflow’s
Check out these docs for a how-to on running the KubernetesPodOperator on Astronomer:
- On an Astronomer Deployment: https://www.astronomer.io/docs/kubepodoperator/
- Locally: https://www.astronomer.io/docs/cli-kubepodoperator/
Talend will export the jobs as images to their repository of choice, and we’d then point the
KubernetesPodOperatorat those images.
Here’s the source code from that original guide: https://github.com/Talend/tmk-tech-blogs/blob/master/orchestration-containers-airflow/databricks_workflow.py