I have an API that I need to hit in a Airflow task. It has a Max Calls per Minute of a 100.
Is there anyway inherently with Airflow to do this or would I need to implement custom logic to handle this?
Airflow does not have a built-in feature to handle API rate limiting out of the box. However, you can implement custom logic to handle API rate limiting within your Airflow task.
One approach is to use a Python library such as ratelimit or limits to enforce the rate limit. These libraries allow you to specify the maximum number of calls per minute and will automatically limit the rate of API calls to stay within that limit.
Alternatively, you could also use Airflow’s PythonOperator to execute a Python function that includes the custom rate-limiting logic