Thread Info
  • State Not Answered
  • Locked Locked
  • Replies 4 replies
  • Subscribers 14 subscribers
  • Views 7289 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Windows Servers MCS 8001 errors: Sophos Management Communications System received HTTP status 4294967295 from the server.

LHerzog

On our Windows Servers I can see that ecent reported frequently after November 4th 2022.

The Sophos Management Communications System client service has received an HTTP status 4294967295 from the server. This might indicate that action is necessary.

That is the corresponding event in MCS Client log belonging to the Windows Event log from 11:10:33.

2022-12-15T10:10:02.527Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/endpoint/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6
2022-12-15T10:10:02.620Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=93ms
2022-12-15T10:10:02.623Z [ 3256: 2676] I [push]: [connect] trying server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps
2022-12-15T10:10:02.623Z [ 3256: 2676] I [push]: [connect] trying direct connection without a proxy
2022-12-15T10:10:02.623Z [ 3256: 2676] I GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps
2022-12-15T10:10:18.027Z [ 3256: 2676] E Request failed: WinHttpSendRequest failed: The connection with the server was terminated abnormally (12030)
2022-12-15T10:10:18.028Z [ 3256: 2676] W [push]: [connect] no configured servers working; trying fallback connection
2022-12-15T10:10:18.028Z [ 3256: 2676] I [push]: [connect] trying server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps
2022-12-15T10:10:18.028Z [ 3256: 2676] I [push]: [connect] trying direct connection without a proxy
2022-12-15T10:10:18.028Z [ 3256: 2676] I GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps
2022-12-15T10:10:18.240Z [ 3256: 2676] I 200 : sent=0 rcvd=0 elapsed=212ms
2022-12-15T10:10:18.241Z [ 3256: 2676] I [push]: [connect] using server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps without a proxy (peer address 52.28.191.22)
2022-12-15T10:10:18.241Z [ 3256: 2676] I (async) GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps/push/endpoint/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6
2022-12-15T10:10:33.647Z [ 3256: 2676] W (async) connection failed
2022-12-15T10:10:33.647Z [ 3256: 2676] I [push]: Dropping connection after error
2022-12-15T10:10:33.649Z [ 3256: 2676] I [backoff] waiting 1354s (960s + 394s skew) after failures: 3

.

Before that date MCS reported occasionally only:

The Sophos Management Communications System client service has received an HTTP status 504 from the server. This might indicate that action is necessary.

The Sophos Management Communications System client service has received an HTTP status 503 from the server. This might indicate that action is necessary.

The Sophos Management Communications System client service has received an HTTP status 502 from the server. This might indicate that action is necessary.

.

There was a big update on the servers at Nov 4th:

	Line 87499: 2022-11-04T15:01:22.964Z [14952:18296] I Installing component 244E68BF-E1BB-4A6B-AC18-A492DE0134C0 (hmpa64) 3.8.3.812
	Line 87724: 2022-11-04T15:15:06.835Z [13084: 1648] I Uninstalling component E17FE03B-0501-4aaa-BC69-0129D965F311 (savxp) 10.8.11.41
	Line 87737: 2022-11-04T15:15:56.775Z [13084: 1648] I Uninstalling component 3D8DC0A9-7F42-4CD5-AA7B-CF29296E7789 (clean64) 3.9.14.1
	Line 88944: 2022-11-04T15:16:22.803Z [13084: 1648] I Installing component 243DECCD-8080-410D-A45F-77F2182715EE (UNINSTALLER64) 1.14.9.9
	Line 88957: 2022-11-04T15:16:23.071Z [13084: 1648] I Installing component 1129226C-32AB-4B72-85E1-A9CC8DFBC859 (SED64) 3.1.1.270
	Line 88968: 2022-11-04T15:16:25.847Z [13084: 1648] I Installing component 3799FB3E-808A-4F7D-AC6A-0C74F931C386 (MCS) 4.17.112
	Line 88980: 2022-11-04T15:16:30.977Z [13084: 1648] I Installing component 0253775E-970D-4876-959C-21B422420E5A (SSE64) 3.85.1.12
	Line 88990: 2022-11-04T15:16:35.769Z [13084: 1648] I Installing component 591706A7-9603-4255-A65F-EA49BB11E8AC (SFS64) 1.9.24.1
	Line 89000: 2022-11-04T15:16:41.779Z [13084: 1648] I Installing component 3CE954A1-0F41-4D9B-B2F0-58AA75334DFD (SHS) 2.9.152
	Line 89010: 2022-11-04T15:16:44.423Z [13084: 1648] I Installing component 5CD1A7B6-812E-47A1-A986-3A6D5D5C19F5 (UI64) 2.6.83.0
	Line 89020: 2022-11-04T15:16:46.503Z [13084: 1648] I Installing component 642A6FD9-A9D6-482D-BD8C-46661F241A0E (AMSI64) 1.9.244
	Line 89030: 2022-11-04T15:16:47.829Z [13084: 1648] I Installing component 70FDD40E-986A-44E5-9620-2B894A06702A (SME64) 1.8.13.2
	Line 89043: 2022-11-04T15:16:50.286Z [13084: 1648] I Installing component 7F682906-6E49-481B-89C5-2DCA36720F4F (ESH64) 3.2.339.0
	Line 89054: 2022-11-04T15:16:55.285Z [13084: 1648] I Installing component BA3387BB-AE88-4403-A36D-F8C0E0B6AEB2 (LIVETERMINAL64) 1.5.245.0
	Line 89064: 2022-11-04T15:16:55.727Z [13084: 1648] I Installing component CD297D6B-58A5-474F-8A0D-0A15803B8B50 (EFW64) 2.1.43
	Line 89074: 2022-11-04T15:16:57.476Z [13084: 1648] I Installing component FileIntegrityMonitoring (FIM) 1.0.1.11.1
	Line 89084: 2022-11-04T15:16:57.871Z [13084: 1648] I Installing component LiveQuery64 (LiveQuery64) 3.5.0.420
	Line 89095: 2022-11-04T15:17:02.290Z [13084: 1648] I Installing component MTR64 (MTR64) 2.4.0.59
	Line 89105: 2022-11-04T15:17:06.377Z [13084: 1648] I Installing component NTP64 (NTP64) 1.16.2923
	Line 89116: 2022-11-04T15:17:14.598Z [13084: 1648] I Installing component SDU64 (SDU64) 6.13.1014
	Line 89126: 2022-11-04T15:17:16.484Z [13084: 1648] I Installing component 244E68BF-E1BB-4A6B-AC18-A492DE0134C0 (HMPA64) 3.8.4.37
	Line 89136: 2022-11-04T15:17:32.058Z [13084: 1648] I Installing component 1FE3E7DF-EFFA-408A-A1B0-89F15BA61F31 (SAUXG) 6.13.1014
	Line 89742: 2022-11-04T20:23:04.156Z [14120: 7888] I Installing component 0253775E-970D-4876-959C-21B422420E5A (SSE64) 3.85.1.12
	Line 90437: 2022-11-05T03:23:05.422Z [15116:17904] I Installing component 0253775E-970D-4876-959C-21B422420E5A (SSE64) 3.85.1.12
	Line 91038: 2022-11-05T09:22:51.793Z [17912:17976] I Installing component 244E68BF-E1BB-4A6B-AC18-A492DE0134C0 (HMPA64) 3.8.4.37

Current version on the servers is:

MCS: Installed V. 4.18.215

MCS Status is green:

I've reported about MCS Event Error 8001 two years ago already:

 504 / 8001 MCS Client intermittently timing out connecting to mcs-push-server-eu-central-1.prod.hydra.sophos.com 



This thread was automatically locked due to age.
  • 0

    The error number in the event log isn't of use, that's just the max size of a unsigned int - UINT_MAX, i.e. 0xFFFFFFFF.

    The error of interest is the WinHTTP error 12030 when the WinHTTP library tried to connect to mcs-push-server-eu-central-1.prod.hydra.sophos.com:

    2022-12-15T10:10:02.623Z [ 3256: 2676] I GET mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps
    2022-12-15T10:10:18.027Z [ 3256: 2676] E Request failed: WinHttpSendRequest failed: The connection with the server was terminated abnormally (12030)

    12030 -  ERROR_WINHTTP_CONNECTION_ERROR = The connection with the server was terminated abnormally

    I assume you connect to https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ ok? 

    A Wireshark trace when restarting MCS Client could be useful if there is an issue setting up the connection, i.e. the TLS handshake. Hopefully you get the message at startup?

    For more information from the WinHTTP library in the context of the connection, I'd probably:

    Stop the MCS Client service

    Run from an admin prompt:

    netsh trace start scenario=InternetClient capture=yes

    Start the MCS Client service

    Check the log for the error, hopefully it happens, then run:

    netsh trace stop

    You can use Microsoft Network Monitor to review the trace. The timestamps in the MCS client log should be helpful to locate the WinHTTP request being created to the URL in question and then step through from there.

    Hope it helps.

  • 0 in reply to Sophos User930

    Thank you   for an other good answer.

    unfortunately I cannot reproduce the error by restarting the service. It just happens from time to time, just like the still appearing status code 504.

    we'd need to dump that all the time and stop when the issue has been noticed again.

    2022-12-15T14:11:06.545Z [ 3256: 2676] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20221215141101486.json
2022-12-15T14:11:15.487Z [ 3256: 7700] I (async) 200 : chunk=8 rcvd=7 conntime=420123ms
2022-12-15T14:11:33.134Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6
2022-12-15T14:11:33.145Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=11ms
2022-12-15T14:11:33.148Z [ 3256: 2676] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/xxxx1508-xxxx-xxxx-xxxx-xxxx422cb6c/feed_id/scheduled_query
2022-12-15T14:11:33.171Z [ 3256: 2676] I 200 : sent=722 rcvd=0 elapsed=22ms
2022-12-15T14:11:33.171Z [ 3256: 2676] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20221215141128084.json
2022-12-15T14:12:00.660Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6
2022-12-15T14:12:00.672Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=11ms
2022-12-15T14:12:00.675Z [ 3256: 2676] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/xxxx1508-xxxx-xxxx-xxxx-xxxx422cb6c/feed_id/scheduled_query
2022-12-15T14:12:00.690Z [ 3256: 2676] I 200 : sent=688 rcvd=0 elapsed=14ms
2022-12-15T14:12:00.690Z [ 3256: 2676] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20221215141155621.json
2022-12-15T14:12:27.378Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6
2022-12-15T14:12:27.390Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=12ms
2022-12-15T14:12:27.393Z [ 3256: 2676] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/xxxx1508-xxxx-xxxx-xxxx-xxxx422cb6c/feed_id/scheduled_query
2022-12-15T14:12:27.410Z [ 3256: 2676] I 200 : sent=815 rcvd=0 elapsed=16ms
2022-12-15T14:12:27.410Z [ 3256: 2676] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20221215141222349.json
2022-12-15T14:12:47.214Z [ 3256: 1968] W (async) Server has disconnected unexpectedly
2022-12-15T14:12:47.214Z [ 3256: 1968] W Push server disconnected unexpectedly
2022-12-15T14:12:47.214Z [ 3256: 1968] I Triggering a command poll
2022-12-15T14:12:47.242Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6
2022-12-15T14:12:47.254Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=12ms
2022-12-15T14:12:47.257Z [ 3256: 2676] I (async) GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps/push/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6
2022-12-15T14:13:02.636Z [ 3256: 2676] W (async) connection failed
2022-12-15T14:13:02.637Z [ 3256: 2676] I [push]: Dropping connection after error
2022-12-15T14:13:02.639Z [ 3256: 2676] I [backoff] waiting 402s (240s + 162s skew) after failures: 1
2022-12-15T14:13:02.651Z [ 3256: 2676] I The telemetry data is: {"mcs":{"agent":{"nonPersistentImage": false,"cloudPlatform":""},"flags":{HUGE-LIST-OF-JUNK}}}
2022-12-15T14:13:02.663Z [ 3256: 2676] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/xxxx1508-xxxx-xxxx-xxxx-xxxx422cb6c/feed_id/scheduled_query
2022-12-15T14:13:02.689Z [ 3256: 2676] I 200 : sent=688 rcvd=0 elapsed=25ms
2022-12-15T14:13:02.689Z [ 3256: 2676] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20221215141249949.json
2022-12-15T14:13:02.693Z [ 3256: 2676] I [push]: [connect] trying server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps
2022-12-15T14:13:02.693Z [ 3256: 2676] I [push]: [connect] trying direct connection without a proxy
2022-12-15T14:13:02.693Z [ 3256: 2676] I GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps
2022-12-15T14:13:02.754Z [ 3256: 2676] I 200 : sent=0 rcvd=0 elapsed=61ms
2022-12-15T14:13:02.755Z [ 3256: 2676] I [push]: [connect] using server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps without a proxy (peer address 18.156.41.215)
2022-12-15T14:13:02.755Z [ 3256: 2676] I (async) GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps/push/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6
2022-12-15T14:13:18.256Z [ 3256: 2676] W (async) connection failed
2022-12-15T14:13:18.256Z [ 3256: 2676] I [push]: Dropping connection after error
2022-12-15T14:13:18.258Z [ 3256: 2676] I [backoff] waiting 729s (480s + 249s skew) after failures: 2
2022-12-15T14:25:28.181Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6
2022-12-15T14:25:28.266Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=85ms
2022-12-15T14:25:28.270Z [ 3256: 2676] I [push]: [connect] trying server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps
2022-12-15T14:25:28.271Z [ 3256: 2676] I [push]: [connect] trying direct connection without a proxy
2022-12-15T14:25:28.271Z [ 3256: 2676] I GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps
2022-12-15T14:25:28.337Z [ 3256: 2676] I 200 : sent=0 rcvd=0 elapsed=66ms

  • 0 in reply to LHerzog

    I believe it is an issue again with Sophos Firewall not working 100% with Sophos EP Software.

    Firewall quit's the connection as can be seen in log and wireshark on endpoint.

    I can see tons of requests going into the wrong firewall rule (5 which is a block rule) from many hosts in our network.

    The DNS records have a TTL of only 4 seconds. I think that is too quick for a Sophos firewall.

  • 0 in reply to LHerzog

    19.5 MR1 has a fix:

    NC-111423 FQDN FQDNs resolving with low TTL (2-5 seconds) are creating issues with wildcard FQDN host.
Unfiltered HTML