We need to use selenium in a PythonOperator; selenium requires at least one web browser driver (such as a chromedriver) in order to work.
So two questions:
- How can I add that chrome driver to my local astronomer deployment?
- Will it be possible for me to add that chrome driver to the cloud deployment as well?
Answer
Our Airflow containers run on an Alpine Linux based image. You can add system-level packages to your image via the packages.txt
file and python packages via the requirements.txt
file, both of which are automatically generated when you initialize an Airflow project on Astronomer via the CLI (by running astro airflow init
).
In your case, you can:
- Add
chromium-chromedriver
to packages.txt - Add
selenium
to requirements.txt
This applies to both your local deployment as well as your deployment on Astronomer Cloud. Cloud runs the exact same image you build locally, so anything that works locally will work in Cloud.
Note: We’re currently experimenting with a Debian based image compatible with both Astronomer Cloud and Enterprise. If you’d like to try it out, reach out to support@astronomer.io.