Xcom defecting the variable

Airflow
1

Hi,

I have a DAG to query some small set of data from elasticsearch.
I have one task to generate the body to query the elastic with some logic.
I’m pushing the body to xcom and then pulling it in the next task that queries the elastic.
I’m getting an error “Expected [SRART_OBJECT] but found [START_ARRAY]”.
Apparently the problem was with xcom.
It is somehow corrupting the body, maybe the indentation or something else, but when I use the same body without passing it in xcom it works good with no error.
Any ideas?

2

Hi Eranh,

I think there is a size limit to your xcom data. The size is dependent on the database airflow is using. This article talks about the size limits: https://stackoverflow.com/questions/54545384/maximum-memory-size-for-an-xcom-in-airflow

Note that this is the database you set to manage airflow objects, which may be different than the database you’re connecting to.

TLDR;
Size limits by database:
SQLite: 2 GB
Postgres: 1 GB
MySQL: 64 KB

~Joe

3

I’m hardly scratching 10kb, I only pass what should be a “query”.
As I said the content in the UI seems OK.
I guess the problem is indentation or something like that.

4

I don’t know much about ElasticSearch, but, how are you building your query? Are you passing an array? Maybe the query needs to be passed as a string. Your error says it found an array.

~Joe