Azure Data Factory Trigger not Working

Airflow
1

Hi, I have followed Atronomer’s guides on how to integrate Azure Data Factory (guide1, guide2), including, their actual code. However, regardless of what I do, I always get a task failed with a generic message:

*** Trying to get logs (last 100 lines) from worker pod airflowadfintegrationdemodataqualityfactorypipelinerundqpipelin ***

*** Unable to fetch logs from worker pod airflowadfintegrationdemodataqualityfactorypipelinerundqpipelin ***
(404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': '46763a07-4fd4-4ade-9cac-2cc535cf5d63', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Fri, 25 Feb 2022 14:24:10 GMT', 'Content-Length': '294'})
HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \\"airflowadfintegrationdemodataqualityfactorypipelinerundqpipelin\\" not found","reason":"NotFound","details":{"name":"airflowadfintegrationdemodataqualityfactorypipelinerundqpipelin","kind":"pods"},"code":404}\n'

Part of the message seems to be related with Kubernet’s inability to get the logs, but the rest, seems to show an error regarding the inability to start an ADF run pipeline command.

I have tried multiple configurations on the Connection object (which in the tutorials is an Azure Delta Lake connection type but in reality, I’m using Azure Data Factory) but without success.

The strangest part it, if I do it via Powershell, it works:

$tenantID = "the Azure tenant ID"
$appId = "the Azure App ID"
$clientSecretsValue = "The value associated with the Azure App's Credentials"
$subscriptionId = "the data factory's subscription id"
$resourceGroupName = "the data factory's resource group name"
$factoryName = "the factory name"
$apiVersion = "2018-06-01"
$pipelineName = "the pipeline name"

az login --service-principal -u ${appId} -p ${clientSecretsValue} --tenant ${tenantID}

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/${pipelineName}/createRun?api-version=${apiVersion}"
$response = Invoke-AzRestMethod -Path ${path} -Method POST
$response.content