How to programmatically create new Airflow Deployments on Astronomer

[madeline]

How can I leverage Astronomer’s Houston API to programmatically create one or more Airflow Deployments as part of an automated workflow?

[pete]

You can run HTTP requests against Houston, our GraphQL API. Interfacing with GraphQL over HTTP is a bit different than a traditional RESTful framework, so I’d recommend reading the GraphQL docs on serving over HTTP as a starting point.

Generating a Service Account (UI)

If the action that you’d like to automate happens at the workspace or deployment level (this is all you’ll be able to do if you’re a Cloud customer, whereas Enterprise customers will have access to system-level mutations and queries), you can generate your service account token directly via the Astro UI. To do this, head into your workspace or deployment (pending the scope you’d like your service token to access), head over to the Service Accounts tab and generate a new key. Copy this key to your clipboard and save it for use in your external script- be sure not to share it with anyone!

Generating a Service Account (API)

If you’re an enterprise customer looking to generate a system-level service token that can access all queries and mutations exposed by our API, you can do so via the Houston API GraphQL playground.

To generate that system-level service token, follow the steps below:

1. Login to the Astro UI and navigate to the /token route. Copy this auth token to your clipboard.

2. Head to the Houston GraphQL playground at houston.<BASEDOMAIN>/v1.

3. In the HTTP Headers section at the bottom of the playground, add the following object:

   { "Authorization": "<TOKEN-FROM-CLIPBOARD>"}
   

   This will allow Houston to confirm that you are authorized to make requests.

4. Run the createSystemServiceAccount mutation:

   mutation CreateSystemServiceAccount {
     createSystemServiceAccount(label: "Your Label", role: SYSTEM_ADMIN) {
        apiKey
     }
   }
   

5. Grab the apiKey returned in the playground’s response window and save it for use in your script. Do not expose this key anywhere- with great power comes great responsibility!

Automating createDeployment Method in Javascript

To find the available arguments to pass to this mutation, inspect the schema in your GraphQL playground:

Any field with an exclamation point (!) at the end of the arg type is required, whereas the rest are optional.

Below is an example of creating a deployment running the Local Executor, using the javascript fetch API as a medium for running the request from a browser:

const workspaceUuid = <YOUR-WORKSPACE-ID> // Can be found in your browser URL when you are accessing your workspace
const type = "airflow" // Legacy field- will be deleted or made non-required in a future version of our API, so keep an eye out
const label = "Deployment Label" // Name of your deployment
const config = { executor: "LocalExecutor" } // Which executor you'd like to run this Airflow deployment with


const query = `mutation CreateDeployment($workspaceUuid: Uuid!, $type: String!, $label: String!, $config: JSON) {
  createDeployment(workspaceUuid: $workspaceUuid, type: $type, label: $label, config: $config) { id, label, releaseName }
}`;


fetch('https://houston.BASEDOMAIN/v1', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Accept': 'application/json',
    'Authorization': <SERVICE_ACCOUNT> // The service account key you've generated for your workspace, deployment, or system
}
  },
  body: JSON.stringify({
    query,
    variables: { workspaceUuid, type, label, config },
  })
})
  .then(r => r.json())
  .then(data => console.log('data returned:', data));

Automating createDeployment Method in Python

Alternatively, if you’d like to do run the same example via a python script using the graphql client:

from gql import gql, Client
from gql.transport.requests import RequestsHTTPTransport
astro_sa_token = <SERVICE_ACCOUNT> # The service account key you've generated for your workspace, deployment, or system
astro_workspace_uuid = <YOUR-WORKSPACE-ID> # Can be found in your browser URL when you are accessing your workspace
sample_transport = RequestsHTTPTransport(
    url='https://houston.BASEDOMAIN/v1',
    use_json=True,
    headers={
        "Content-type": "application/json",
        "Authorization": astro_sa_token
    },
    verify=False
)
client = Client(
    retries=3,
    transport=sample_transport,
    fetch_schema_from_transport=True,
)
query = gql(''' 
    mutation CreateDeployment {
      createDeployment(
        workspaceUuid: astro_workspace_uuid,
        type:"airflow",
        label:"Deployment Label",
        config: {executor:"LocalExecutor"}
    )
    {
      releaseName
    }
}
''')
result = client.execute(query)
print(result)

Other Methods

Note that this same logic can apply to any mutation/query you’d like to run against the API, you’ll just need to change up the GraphQL fragment you’re passing to the body of each request. You can also integrate these scripts into a CI/CD process if you’d like to automate certain actions.

[scheung38]

Added authorization key but:

This is on-prem k8s cluster

[Marek]

Hey I was following this setup and I’ve bumped into strange behavior. I’ve cerated system service account and after I’ve used it to create a workspace I was not able to create any more workspaces. Houston is returning error message: “Insufficient permissions.” Same issue is with deployment, I can only create one deployment and that is all.

I don’t think it is expected behavior.
In addition whenever I’m trying to get some information about service accounts(even with tokens from my admin user account) eg:

query SystemServiceAccount{
      serviceAccounts(serviceAccountUuid:"xxx",entityType:SYSTEM){
        active,
        id,
        label
      }
    }

I’m getting following error: “Cannot read property ‘roleBindings’ of undefined”

Any advice will be appreciated.

[Marek]

This behavior was fixed in latest versions. However there is still a problem with roleBindings being undefined when querying serviceaccounts.

[paola]

Hey @Marek ! Thanks for reporting this. Bit of a late reply here, but the issue you’re getting when querying for Service Accounts is unfortunately a known bug on our end. We recently introduced Deployment-level permissions, which requires corresponding changes to our deploymentServiceAccount and workspaceServiceAccount queries/mutations.

We haven’t made those changes yet, but this issue is documented and on our to-do list. I’ll post back an update in here as soon as it’s resolved! For now, you’ll have to rely on the Astronomer UI for information around your Service Accounts.

For additional assistance, you can always reach out to Astronomer Support as well.

Note: If you try to run $ astro deployment service-account get --deployment-id=<deployment-id> via the Astronomer CLI, you can expect to receive the same Cannot read property 'roleBindings' of undefined error until this is resolved.