Consume specific partition/offset values from kafka topic using airflow

Airflow
1

Hi Team,
I am using ‘AwaitMessageTriggerFunctionSensor’ and ‘ConsumeFromTopicOperator’ to consume kakfa topic.
I want to consume specific partition/offset values from kafka topic using airflow but I didn’t find any option in the above two airflow functions.
Please help me in finding the solution to consume specific partition or offset values using airflow.

2

Hey @gvenka31_uhg,

I looked at it briefly and saw that currently following commit_cadence are available for the ConsumeFromTopicOperator to consume Kafka Topics


commit_cadence: When consumers should commit offsets ("never", "end_of_batch","end_of_operator"),
        defaults to "end_of_operator";
        if end_of_operator, the commit() is called based on the max_messages arg. Commits are made after the
        operator has processed the apply_function method for the maximum messages in the operator.
        if end_of_batch, the commit() is called based on the max_batch_size arg. Commits are made after each
        batch has processed by the apply_function method for all messages in the batch.
        if never,  close() is called without calling the commit() method.

Not sure how much it helps you.

Also, Here on Astronomer Registry are different Apache Kafka Airflow Operators/Hooks and Example DAGs, if it helps