I have a use case wherein I need to read some values from a metadata table, and later use these values within tasks, within the context of a DAG.
To elaborate, each of these tasks are nothing but a Snowflake stored procedure written using Snowpark. Coming back to the problem, I am trying to read the initial metadata and load it into few Python lists using Snowpark library. ( For those new to Snowpark, its basically nothing but a simple library that provides Data frame like constructs on top of tables.)
I am unable to get the library installed by specifying it in the requirements.txt file. Now Snowpark as such needs Python 3.8 , however the base image of astronomer-airflow that I am using has Python 3.9 - so that’s an issue. Can someone point out an way in which I could override the version of Python that comes by default in the base image and get my desired version installed?
Secondly, I tried to work around this by trying to use the normal snowflake python connector in requirements.txt file. However even that fails to install.
The strange part is that no kind of error is thrown whatsoever. I do a pip freeze from the scheduler container and and the desired library is not installed.
Could somebody help troubleshoot this issue?