Hello,
I am trying to upgrade our AirFlow service from v1.10.15 to v2, but am having a serious headache from the airflow update_check working on some instances but not others.
Both instances are built using the same Dockerfile and airflow.cfg. One (personal sandbox) is using just a postgres container for a meta DB, and that has no issues. But running the update check on a UAT machine which uses an AWS RDS for meta DB fails right after I get a “SUCCESS” on the “Chain between DAG and operator not allowed”. Full error pasted below.
I’ve also verified that I am able to query the meta DB from within the “broken” Airflow container via psycopg2 and pgcli
I am a little perplexed as we don’t use the Connections for anything. I’ve tried rebuilding the container with --no-cache flag in the build command and verifying that everything in the airflow.cfg is the same between the two instances… no dice. StackOverflow and Google have both yielded nothing that seems relevant, any ideas?
Chain between DAG and operator not allowed.................................................................................................SUCCESS
Traceback (most recent call last):
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 2285, in _wrap_pool_connect
return fn()
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 363, in connect
return _ConnectionFairy._checkout(self)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 773, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 492, in checkout
rec = pool._do_get()
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py”, line 139, in _do_get
self.dec_overflow()
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py”, line 69, in exit
exc_value, with_traceback=exc_tb,
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 178, in raise
raise exception
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py”, line 136, in _do_get
return self._create_connection()
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 308, in _create_connection
return _ConnectionRecord(self)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 437, in init
self.__connect(first_connect_check=True)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 657, in _connect
pool.logger.debug(“Error on connect(): %s”, e)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py”, line 69, in exit
exc_value, with_traceback=exc_tb,
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 178, in raise
raise exception
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 652, in __connect
connection = pool._invoke_creator(self)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py”, line 114, in connect
return dialect.connect(*cargs, **cparams)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py”, line 488, in connect
return self.dbapi.connect(*cargs, **cparams)
File “/usr/local/lib/python3.6/site-packages/psycopg2/init.py”, line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not translate host name “postgres” to address: Name or service not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/usr/local/bin/airflow”, line 37, in
args.func(args)
File “/usr/local/lib/python3.6/site-packages/airflow/upgrade/checker.py”, line 118, in run
all_problems = check_upgrade(formatter, rules)
File “/usr/local/lib/python3.6/site-packages/airflow/upgrade/checker.py”, line 38, in check_upgrade
rule_status = RuleStatus.from_rule(rule)
File “/usr/local/lib/python3.6/site-packages/airflow/upgrade/problem.py”, line 44, in from_rule
result = rule.check()
File “/usr/local/lib/python3.6/site-packages/airflow/utils/db.py”, line 74, in wrapper
return func(*args, **kwargs)
File “/usr/local/lib/python3.6/site-packages/airflow/upgrade/rules/conn_type_is_not_nullable.py”, line 45, in check
for conn in invalid_connections.all()
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py”, line 3244, in all
return list(self)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py”, line 3403, in iter
return self._execute_and_instances(context)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py”, line 3425, in _execute_and_instances
querycontext, self._connection_from_session, close_with_result=True
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py”, line 3440, in _get_bind_args
mapper=self._bind_mapper(), clause=querycontext.statement, **kw
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py”, line 3418, in _connection_from_session
conn = self.session.connection(**kw)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 1133, in connection
execution_options=execution_options,
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 1139, in _connection_for_bind
engine, execution_options
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py”, line 432, in _connection_for_bind
conn = bind._contextual_connect()
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 2251, in _contextual_connect
self._wrap_pool_connect(self.pool.connect, None),
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 2289, in wrap_pool_connect
e, dialect, self
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 1555, in handle_dbapi_exception_noconnection
sqlalchemy_exception, with_traceback=exc_info[2], from=e
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 178, in raise
raise exception
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py”, line 2285, in _wrap_pool_connect
return fn()
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 363, in connect
return _ConnectionFairy._checkout(self)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 773, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 492, in checkout
rec = pool._do_get()
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py”, line 139, in _do_get
self.dec_overflow()
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py”, line 69, in exit
exc_value, with_traceback=exc_tb,
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 178, in raise
raise exception
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py”, line 136, in _do_get
return self._create_connection()
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 308, in _create_connection
return _ConnectionRecord(self)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 437, in init
self.__connect(first_connect_check=True)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 657, in _connect
pool.logger.debug(“Error on connect(): %s”, e)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py”, line 69, in exit
exc_value, with_traceback=exc_tb,
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 178, in raise
raise exception
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py”, line 652, in __connect
connection = pool._invoke_creator(self)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py”, line 114, in connect
return dialect.connect(*cargs, **cparams)
File “/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py”, line 488, in connect
return self.dbapi.connect(*cargs, **cparams)
File “/usr/local/lib/python3.6/site-packages/psycopg2/init.py”, line 126, in connect
conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not translate host name “postgres” to address: Name or service not known
(Background on this error at: http://sqlalche.me/e/e3q8)