How to programmatically create new Airflow Deployments on Astronomer

2

You can run HTTP requests against Houston, our GraphQL API. Interfacing with GraphQL over HTTP is a bit different than a traditional RESTful framework, so I’d recommend reading the GraphQL docs on serving over HTTP as a starting point.

Generating a Service Account (UI)

If the action that you’d like to automate happens at the workspace or deployment level (this is all you’ll be able to do if you’re a Cloud customer, whereas Enterprise customers will have access to system-level mutations and queries), you can generate your service account token directly via the Astro UI. To do this, head into your workspace or deployment (pending the scope you’d like your service token to access), head over to the Service Accounts tab and generate a new key. Copy this key to your clipboard and save it for use in your external script- be sure not to share it with anyone!

Generating a Service Account (API)

If you’re an enterprise customer looking to generate a system-level service token that can access all queries and mutations exposed by our API, you can do so via the Houston API GraphQL playground.

To generate that system-level service token, follow the steps below:

  1. Login to the Astro UI and navigate to the /token route. Copy this auth token to your clipboard.

  2. Head to the Houston GraphQL playground at houston.<BASEDOMAIN>/v1.

  3. In the HTTP Headers section at the bottom of the playground, add the following object:

    { "Authorization": "<TOKEN-FROM-CLIPBOARD>"}

    This will allow Houston to confirm that you are authorized to make requests.

  4. Run the createSystemServiceAccount mutation:

    mutation CreateSystemServiceAccount {
  createSystemServiceAccount(label: "Your Label", role: SYSTEM_ADMIN) {
     apiKey
  }
}

  5. Grab the apiKey returned in the playground’s response window and save it for use in your script. Do not expose this key anywhere- with great power comes great responsibility!

Automating createDeployment Method in Javascript

To find the available arguments to pass to this mutation, inspect the schema in your GraphQL playground:

image
image602×550 36.6 KB

Any field with an exclamation point (!) at the end of the arg type is required, whereas the rest are optional.

Below is an example of creating a deployment running the Local Executor, using the javascript fetch API as a medium for running the request from a browser:

const workspaceUuid = <YOUR-WORKSPACE-ID> // Can be found in your browser URL when you are accessing your workspace
const type = "airflow" // Legacy field- will be deleted or made non-required in a future version of our API, so keep an eye out
const label = "Deployment Label" // Name of your deployment
const config = { executor: "LocalExecutor" } // Which executor you'd like to run this Airflow deployment with


const query = `mutation CreateDeployment($workspaceUuid: Uuid!, $type: String!, $label: String!, $config: JSON) {
  createDeployment(workspaceUuid: $workspaceUuid, type: $type, label: $label, config: $config) { id, label, releaseName }
}`;


fetch('https://houston.BASEDOMAIN/v1', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Accept': 'application/json',
    'Authorization': <SERVICE_ACCOUNT> // The service account key you've generated for your workspace, deployment, or system
}
  },
  body: JSON.stringify({
    query,
    variables: { workspaceUuid, type, label, config },
  })
})
  .then(r => r.json())
  .then(data => console.log('data returned:', data));

Automating createDeployment Method in Python

Alternatively, if you’d like to do run the same example via a python script using the graphql client:

from gql import gql, Client
from gql.transport.requests import RequestsHTTPTransport
astro_sa_token = <SERVICE_ACCOUNT> # The service account key you've generated for your workspace, deployment, or system
astro_workspace_uuid = <YOUR-WORKSPACE-ID> # Can be found in your browser URL when you are accessing your workspace
sample_transport = RequestsHTTPTransport(
    url='https://houston.BASEDOMAIN/v1',
    use_json=True,
    headers={
        "Content-type": "application/json",
        "Authorization": astro_sa_token
    },
    verify=False
)
client = Client(
    retries=3,
    transport=sample_transport,
    fetch_schema_from_transport=True,
)
query = gql(''' 
    mutation CreateDeployment {
      createDeployment(
        workspaceUuid: astro_workspace_uuid,
        type:"airflow",
        label:"Deployment Label",
        config: {executor:"LocalExecutor"}
    )
    {
      releaseName
    }
}
''')
result = client.execute(query)
print(result)

Other Methods

Note that this same logic can apply to any mutation/query you’d like to run against the API, you’ll just need to change up the GraphQL fragment you’re passing to the body of each request. You can also integrate these scripts into a CI/CD process if you’d like to automate certain actions.

2 Likes