Deploy DAGs from S3 to Astro Airflow using AWS Lambda

Astronomer Astro
, ,
1

In scenarios, where you are storing your DAGs in S3 bucket and would like to auto-deploy to Astro whenever new changes are pushed, follow the following steps:

  1. Download the latest Astro CLI binary from here, rename to astro_cli.tar.gz and upload to a folder called cli_binary in your S3 bucket

  2. Make sure the AWS Role that Lambda will use has access for GetObject and ListBucket for your bucket along with the basic Lambda role.

  3. Create a lambda python function that uses boto3 and Astro CLI to deploy to your Astro Deployment

  4. Create a trigger on S3 bucket that watches your DAG directory

  5. Upload a DAG to S3 and within minutes you will see the Astro Deployment getting refreshed with the new DAGs

For detailed step-by-step process with examples, please refer to DAG based deploy

1 Like