We’ve had a few customers with certain security restrictions need to open an SSH tunnel as a part of their workflow. Typically, we tell folks to use the standard SSH hook already built into Airflow.
Here’s an example of accessing a Postgres database using SSH: https://github.com/vparekh94/test_ssh_tunnel/tree/master/
If you’re using the example above locally, make sure you have all necessary dependencies. If you see an error like this, for example -
FileNotFoundError: [Errno 2] No such file or directory: 'ssh': 'ssh'
You’ll need to make sure that you’ve added
openssl-dev to your
packages.txt file and rebuilt the image.
Feel free to also check out a case study on how one of our customers syncs their application database to their data warehouse whilst incorporating SSH tunneling to do so. Descriptions of our product + features are a bit outdated, but it might give you an idea of what your workflows could look like.
If SSH-ing via the hook isn’t an option for you, our Enterprise offering might be worth a look, as it’s a version of our platform deployed directly in your VPC.