Is it possible to overwrite the configuration json string that is passed during airflow trigger_dag?
For example, if a DAG was originally triggered by:
airflow trigger_dag my_dag --conf '{"field1": 1, "field2": 2}'
and the pipeline breaks at some task – is it possible to “update” the conf with a new json string such as:
{"field1": 3, "field2": 4}
and clear the broken task to run with the updated conf?
Either way, would be interested in hearing thoughts on other solutions or potentially ways to avoid the situation to begin with.
Thank you!