504 / 8001 MCS Client intermittently timing out connecting to mcs-push-server-eu-central-1.prod.hydra.sophos.com

There seem to be issues on the Sophos Coud Server mcs-push-server-eu-central-1.prod.hydra.sophos.com.

Can sophos please confirm, the server is running as fast as it should and that this issue is not server side? Please do not simply point to the https://centralstatus.sophos.com/ page - this is green.

Over the day we have many 8001 Errors in event log and the Sophos Agent reports Gateway Timeout 504.

The requests go through a SG Webfilter that does no HTTPS interception, and so no AV scanning of HTTPS, it does only logging and category checking. Also you can see that there are all Exceptions set for the Sophos URLs: exceptions="av,sandbox,auth,content,url,ssl,certcheck,certdate,mime,cache,fileextension,size,patience"

The internet connection is fast and available during that times.

Look here, some of the requests take over 2200 seconds for a size of only 53446 bytes!

FAIL 2020:11:10-13:02:36 dnstime="630" aptptime="89" cattime="0" avscantime="0" fullreqtime="2242635532"

FAIL 2020:11:10-13:02:36 dnstime="7271" aptptime="60" cattime="0" avscantime="0" fullreqtime="2237909027"

OK 2020:11:10-13:04:49 dnstime="4382" aptptime="60" cattime="0" avscantime="0" fullreqtime="60029834"

OK: 2020:11:10-13:04:49 dnstime="1" aptptime="76" cattime="0" avscantime="0" fullreqtime="60055172"

FAIL: 2020:11:10-13:27:08 dnstime="1163" aptptime="88" cattime="0" avscantime="0" fullreqtime="1472177632"

A normal web request of the Sophos Client usually only takes about 6 seconds:

(https://tf-edr-message-upload-eu-central-1-prod-bucket.s3.amazonaws.com) dnstime="5184" aptptime="60" cattime="0" avscantime="0" fullreqtime="6079874"

Logs for the 2020:11:10-13:02 Event:

2020:11:10-13:02:36 fw-1 httpproxy[17259]: id="0001" severity="info" sys="SecureWeb" sub="http" name="http access" action="pass" method="CONNECT" 
srcip="1.2.3.4" dstip="52.29.36.14" user="" group="" ad_domain="" statuscode="200" cached="0" profile="REF_HttProContaInterNetwo5 (ServerNet-Proxy-Transparent)" 
filteraction="REF_HttCffServernetp (ServerNet-Proxy-Transparent)" size="53446" request="0xd6b1b100" url="https://mcs-cloudstation-eu-central-1.prod.hydra.sophos.com/" referer="" error="" 
authtime="0" dnstime="630" aptptime="89" cattime="0" avscantime="0" fullreqtime="2242635532" device="0" auth="0" ua="" exceptions="av,sandbox,auth,content,url,ssl,certcheck,certdate,mime,cache,fileextension,size,patience" 

2020-11-10T12:02:43.847Z [ 3920: 4100] [v4.12.686.0] INFO  Establishing push connection
2020-11-10T12:02:43.850Z [ 3920: 4100] [v4.12.686.0] INFO  (async) GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps/push/endpoint/7ba049fd-1cea-24fc-e954-b3c15e142fb0
2020-11-10T12:02:53.883Z [ 3920: 4804] [v4.12.686.0] INFO  (async) 504 Gateway Time-out: conntime=10032ms
2020-11-10T12:02:53.883Z [ 3920: 4100] [v4.12.686.0] WARN  (async) connection timeout
2020-11-10T12:02:53.883Z [ 3920: 4100] [v4.12.686.0] WARN  [push]: error creating async stream: 0
2020-11-10T12:02:53.884Z [ 3920: 4100] [v4.12.686.0] INFO  [push]: Dropping connection after error

I found this posts but they do not provide a solution but the issue is out there for some years now.

https://community.sophos.com/intercept-x-endpoint/f/discussions/111982/sophos-mcs-event-8001-the-sophos-mcs-cliens-service-has-received-an-http-status-504-503-from-the-server/441579#441579

https://community.sophos.com/intercept-x-endpoint/f/discussions/101233/event-id-8001-the-sophos-management-communications-system-client-service-has-received-an-http-status-503-from-the-server-this-might-indicate-that-action-is-necessary