In scenarios, where you are storing your DAGs in S3 bucket and would like to auto-deploy to Astro whenever new changes are pushed, follow the following steps:
-
Download the latest Astro CLI binary from here, rename to
astro_cli.tar.gz
and upload to a folder calledcli_binary
in your S3 bucket -
Make sure the AWS Role that Lambda will use has access for
GetObject
andListBucket
for your bucket along with the basic Lambda role. -
Create a lambda python function that uses
boto3
and Astro CLI to deploy to your Astro Deployment -
Create a trigger on S3 bucket that watches your DAG directory
-
Upload a DAG to S3 and within minutes you will see the Astro Deployment getting refreshed with the new DAGs
For detailed step-by-step process with examples, please refer to DAG based deploy