Log won't show in Airflow UI

We recently upgraded our Astronomer Enterprise platform to v0.23.10.
Since the upgrade, the tasks’ logs do not load into the Airflow UI.

The symptom is the same weather the task was sucessfull or not.
We can see the loading animation but it just hangs forever and the page stays blank.
It’s doing the same thing in both our Airflow 2.0 and Airflow 1.10.5 deployments.

We get this exception in our webserver’s log when we try to access the task log:

[2021-02-09 13:50:21,194] {base.py:150} WARNING - POST http://astronomer-elasticsearch-nginx.bi-astronomer:9200/_count [status:N/A request:30.003s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 242, in perform_request
    method, url, body, retries=Retry(False), headers=request_headers, **kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 386, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1277, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1323, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1272, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1032, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 972, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7fa1b58b0dd0>, 'Connection to astronomer-elasticsearch-nginx.bi-astronomer timed out. (connect timeout=30)')

Do you have any idea of what could be causing this ?

The behaviour is still the same after upgrading to v0.23.11.

Hi @Ubald! Thanks for reaching out, and I’m sorry to hear you’re still having trouble seeing logs in the Airflow UI on Astronomer’s latest.

Are you seeing logs populate in Kibana at all? Do you still see Connection to astronomer-elasticsearch-nginx.bi-astronomer timed out?

Given the potential complexity of this issue, can you reach out to Astronomer Support so our team can dig into the behavior you’re seeing in detail? We’re more than happy to help you there, and I’ll keep an eye on that ticket and post back takeaways here for anyone publicly following.

Hi Paola,
Webserver and scheduler logs seem to be showing normaly in Kibana. I can see the “GET” being done by the webserver.
I see " Connection to astronomer-elasticsearch-nginx.bi-astronomer timed out" in the Kubernetes Dashboard for the webserver pod.

I made a ticket in Astronomer Support and we have a call this afternoon so hopefully we can get this working soon.

Thank you.

1 Like