Hey, @ozgurgul! Thanks for reaching out.
Depending on the operator you are using, there could be a xcom_push parameter associated an operator’s __init__
method. As far as I know, BashOperator is the only operator with that parameter in the past.
On a side note, it looks like even that parameter is on it’s way out in favour for do_xcom_push
, which will be universal for all operators who are a child of BaseOperator.
Pre 1.10.9
and including, BashOperator returned the output if the xcom_push
is set as True.
In 1.10.10
, the conditional is removed since the same logic can be found in TaskInstance since 1.10.5
.
For clarification from the official airflow documentation, this interaction is described under the XCom section and it has been this way as early as 1.8.1
.
In addition, if a task returns a value (either from its Operator’s execute()
method, or from a PythonOperator’s python_callable
function), then an XCom containing that value is automatically pushed.
Back to your example, PostgresOperator has never had any parameters associated with xcom until 1.10.5
when do_xcom_push
was added as a parameters for BaseOperator
. Even then, PostgresOperator
does not have a return statement meaning it is impossible for any values to be pushed to XCom as is.
On the other hand, there are various routes you can take to communicate the result of a select query through XCOM. One easy way is to use a PythonOperator since it is one of the most flexible operators that allows you to do practically anything in python. An example is provided in the XCom section of the documentation and I have provided a more tailored example below.
from airflow.operators.python_operator import PythonOperator
from airflow.hooks.postgres_hook import PostgresHook
def query_and_push(sql):
pg_hook = PostgresHook(postgres_conn_id='pg_conn')
records = pg_hook.get_records(sql=sql)
return records
action = PythonOperator(
task_id='push_result',
python_callable=query_and_push,
provide_context=True,
op_kwargs={
'sql': 'select * from table;'
},
dag=dag
)
Please let me know if you still have questions about XCom! I am happy to explain in further details