3CX DLL-Sideloading attack: What you need to know
On our Windows Servers I can see that ecent reported frequently after November 4th 2022.
The Sophos Management Communications System client service has received an HTTP status 4294967295 from the server. This might indicate that action is necessary.
That is the corresponding event in MCS Client log belonging to the Windows Event log from 11:10:33.
2022-12-15T10:10:02.527Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/endpoint/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6 2022-12-15T10:10:02.620Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=93ms 2022-12-15T10:10:02.623Z [ 3256: 2676] I [push]: [connect] trying server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps 2022-12-15T10:10:02.623Z [ 3256: 2676] I [push]: [connect] trying direct connection without a proxy 2022-12-15T10:10:02.623Z [ 3256: 2676] I GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps 2022-12-15T10:10:18.027Z [ 3256: 2676] E Request failed: WinHttpSendRequest failed: The connection with the server was terminated abnormally (12030) 2022-12-15T10:10:18.028Z [ 3256: 2676] W [push]: [connect] no configured servers working; trying fallback connection 2022-12-15T10:10:18.028Z [ 3256: 2676] I [push]: [connect] trying server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps 2022-12-15T10:10:18.028Z [ 3256: 2676] I [push]: [connect] trying direct connection without a proxy 2022-12-15T10:10:18.028Z [ 3256: 2676] I GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps 2022-12-15T10:10:18.240Z [ 3256: 2676] I 200 : sent=0 rcvd=0 elapsed=212ms 2022-12-15T10:10:18.241Z [ 3256: 2676] I [push]: [connect] using server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps without a proxy (peer address 52.28.191.22) 2022-12-15T10:10:18.241Z [ 3256: 2676] I (async) GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps/push/endpoint/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6 2022-12-15T10:10:33.647Z [ 3256: 2676] W (async) connection failed 2022-12-15T10:10:33.647Z [ 3256: 2676] I [push]: Dropping connection after error 2022-12-15T10:10:33.649Z [ 3256: 2676] I [backoff] waiting 1354s (960s + 394s skew) after failures: 3
.
Before that date MCS reported occasionally only:
The Sophos Management Communications System client service has received an HTTP status 504 from the server. This might indicate that action is necessary.
The Sophos Management Communications System client service has received an HTTP status 503 from the server. This might indicate that action is necessary.
The Sophos Management Communications System client service has received an HTTP status 502 from the server. This might indicate that action is necessary.
There was a big update on the servers at Nov 4th:
Line 87499: 2022-11-04T15:01:22.964Z [14952:18296] I Installing component 244E68BF-E1BB-4A6B-AC18-A492DE0134C0 (hmpa64) 3.8.3.812 Line 87724: 2022-11-04T15:15:06.835Z [13084: 1648] I Uninstalling component E17FE03B-0501-4aaa-BC69-0129D965F311 (savxp) 10.8.11.41 Line 87737: 2022-11-04T15:15:56.775Z [13084: 1648] I Uninstalling component 3D8DC0A9-7F42-4CD5-AA7B-CF29296E7789 (clean64) 3.9.14.1 Line 88944: 2022-11-04T15:16:22.803Z [13084: 1648] I Installing component 243DECCD-8080-410D-A45F-77F2182715EE (UNINSTALLER64) 1.14.9.9 Line 88957: 2022-11-04T15:16:23.071Z [13084: 1648] I Installing component 1129226C-32AB-4B72-85E1-A9CC8DFBC859 (SED64) 3.1.1.270 Line 88968: 2022-11-04T15:16:25.847Z [13084: 1648] I Installing component 3799FB3E-808A-4F7D-AC6A-0C74F931C386 (MCS) 4.17.112 Line 88980: 2022-11-04T15:16:30.977Z [13084: 1648] I Installing component 0253775E-970D-4876-959C-21B422420E5A (SSE64) 3.85.1.12 Line 88990: 2022-11-04T15:16:35.769Z [13084: 1648] I Installing component 591706A7-9603-4255-A65F-EA49BB11E8AC (SFS64) 1.9.24.1 Line 89000: 2022-11-04T15:16:41.779Z [13084: 1648] I Installing component 3CE954A1-0F41-4D9B-B2F0-58AA75334DFD (SHS) 2.9.152 Line 89010: 2022-11-04T15:16:44.423Z [13084: 1648] I Installing component 5CD1A7B6-812E-47A1-A986-3A6D5D5C19F5 (UI64) 2.6.83.0 Line 89020: 2022-11-04T15:16:46.503Z [13084: 1648] I Installing component 642A6FD9-A9D6-482D-BD8C-46661F241A0E (AMSI64) 1.9.244 Line 89030: 2022-11-04T15:16:47.829Z [13084: 1648] I Installing component 70FDD40E-986A-44E5-9620-2B894A06702A (SME64) 1.8.13.2 Line 89043: 2022-11-04T15:16:50.286Z [13084: 1648] I Installing component 7F682906-6E49-481B-89C5-2DCA36720F4F (ESH64) 3.2.339.0 Line 89054: 2022-11-04T15:16:55.285Z [13084: 1648] I Installing component BA3387BB-AE88-4403-A36D-F8C0E0B6AEB2 (LIVETERMINAL64) 1.5.245.0 Line 89064: 2022-11-04T15:16:55.727Z [13084: 1648] I Installing component CD297D6B-58A5-474F-8A0D-0A15803B8B50 (EFW64) 2.1.43 Line 89074: 2022-11-04T15:16:57.476Z [13084: 1648] I Installing component FileIntegrityMonitoring (FIM) 1.0.1.11.1 Line 89084: 2022-11-04T15:16:57.871Z [13084: 1648] I Installing component LiveQuery64 (LiveQuery64) 3.5.0.420 Line 89095: 2022-11-04T15:17:02.290Z [13084: 1648] I Installing component MTR64 (MTR64) 2.4.0.59 Line 89105: 2022-11-04T15:17:06.377Z [13084: 1648] I Installing component NTP64 (NTP64) 1.16.2923 Line 89116: 2022-11-04T15:17:14.598Z [13084: 1648] I Installing component SDU64 (SDU64) 6.13.1014 Line 89126: 2022-11-04T15:17:16.484Z [13084: 1648] I Installing component 244E68BF-E1BB-4A6B-AC18-A492DE0134C0 (HMPA64) 3.8.4.37 Line 89136: 2022-11-04T15:17:32.058Z [13084: 1648] I Installing component 1FE3E7DF-EFFA-408A-A1B0-89F15BA61F31 (SAUXG) 6.13.1014 Line 89742: 2022-11-04T20:23:04.156Z [14120: 7888] I Installing component 0253775E-970D-4876-959C-21B422420E5A (SSE64) 3.85.1.12 Line 90437: 2022-11-05T03:23:05.422Z [15116:17904] I Installing component 0253775E-970D-4876-959C-21B422420E5A (SSE64) 3.85.1.12 Line 91038: 2022-11-05T09:22:51.793Z [17912:17976] I Installing component 244E68BF-E1BB-4A6B-AC18-A492DE0134C0 (HMPA64) 3.8.4.37
Current version on the servers is:
MCS: Installed V. 4.18.215
MCS Status is green:
I've reported about MCS Event Error 8001 two years ago already:
504 / 8001 MCS Client intermittently timing out connecting to mcs-push-server-eu-central-1.prod.hydra.sophos.com
The error number in the event log isn't of use, that's just the max size of a unsigned int - UINT_MAX, i.e. 0xFFFFFFFF.The error of interest is the WinHTTP error 12030 when the WinHTTP library tried to connect to mcs-push-server-eu-central-1.prod.hydra.sophos.com:
2022-12-15T10:10:02.623Z [ 3256: 2676] I GET mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps2022-12-15T10:10:18.027Z [ 3256: 2676] E Request failed: WinHttpSendRequest failed: The connection with the server was terminated abnormally (12030)
12030 - ERROR_WINHTTP_CONNECTION_ERROR = The connection with the server was terminated abnormally
I assume you connect to https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ ok?
A Wireshark trace when restarting MCS Client could be useful if there is an issue setting up the connection, i.e. the TLS handshake. Hopefully you get the message at startup?
For more information from the WinHTTP library in the context of the connection, I'd probably:
Stop the MCS Client service
Run from an admin prompt:
netsh trace start scenario=InternetClient capture=yes
Start the MCS Client serviceCheck the log for the error, hopefully it happens, then run:
netsh trace stop
You can use Microsoft Network Monitor to review the trace. The timestamps in the MCS client log should be helpful to locate the WinHTTP request being created to the URL in question and then step through from there.
Hope it helps.
Thank you Sophos User930 for an other good answer.
unfortunately I cannot reproduce the error by restarting the service. It just happens from time to time, just like the still appearing status code 504.
we'd need to dump that all the time and stop when the issue has been noticed again.
2022-12-15T14:11:06.545Z [ 3256: 2676] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20221215141101486.json 2022-12-15T14:11:15.487Z [ 3256: 7700] I (async) 200 : chunk=8 rcvd=7 conntime=420123ms 2022-12-15T14:11:33.134Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6 2022-12-15T14:11:33.145Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=11ms 2022-12-15T14:11:33.148Z [ 3256: 2676] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/xxxx1508-xxxx-xxxx-xxxx-xxxx422cb6c/feed_id/scheduled_query 2022-12-15T14:11:33.171Z [ 3256: 2676] I 200 : sent=722 rcvd=0 elapsed=22ms 2022-12-15T14:11:33.171Z [ 3256: 2676] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20221215141128084.json 2022-12-15T14:12:00.660Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6 2022-12-15T14:12:00.672Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=11ms 2022-12-15T14:12:00.675Z [ 3256: 2676] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/xxxx1508-xxxx-xxxx-xxxx-xxxx422cb6c/feed_id/scheduled_query 2022-12-15T14:12:00.690Z [ 3256: 2676] I 200 : sent=688 rcvd=0 elapsed=14ms 2022-12-15T14:12:00.690Z [ 3256: 2676] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20221215141155621.json 2022-12-15T14:12:27.378Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6 2022-12-15T14:12:27.390Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=12ms 2022-12-15T14:12:27.393Z [ 3256: 2676] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/xxxx1508-xxxx-xxxx-xxxx-xxxx422cb6c/feed_id/scheduled_query 2022-12-15T14:12:27.410Z [ 3256: 2676] I 200 : sent=815 rcvd=0 elapsed=16ms 2022-12-15T14:12:27.410Z [ 3256: 2676] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20221215141222349.json 2022-12-15T14:12:47.214Z [ 3256: 1968] W (async) Server has disconnected unexpectedly 2022-12-15T14:12:47.214Z [ 3256: 1968] W Push server disconnected unexpectedly 2022-12-15T14:12:47.214Z [ 3256: 1968] I Triggering a command poll 2022-12-15T14:12:47.242Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6 2022-12-15T14:12:47.254Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=12ms 2022-12-15T14:12:47.257Z [ 3256: 2676] I (async) GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps/push/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6 2022-12-15T14:13:02.636Z [ 3256: 2676] W (async) connection failed 2022-12-15T14:13:02.637Z [ 3256: 2676] I [push]: Dropping connection after error 2022-12-15T14:13:02.639Z [ 3256: 2676] I [backoff] waiting 402s (240s + 162s skew) after failures: 1 2022-12-15T14:13:02.651Z [ 3256: 2676] I The telemetry data is: {"mcs":{"agent":{"nonPersistentImage": false,"cloudPlatform":""},"flags":{HUGE-LIST-OF-JUNK}}} 2022-12-15T14:13:02.663Z [ 3256: 2676] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/xxxx1508-xxxx-xxxx-xxxx-xxxx422cb6c/feed_id/scheduled_query 2022-12-15T14:13:02.689Z [ 3256: 2676] I 200 : sent=688 rcvd=0 elapsed=25ms 2022-12-15T14:13:02.689Z [ 3256: 2676] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20221215141249949.json 2022-12-15T14:13:02.693Z [ 3256: 2676] I [push]: [connect] trying server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps 2022-12-15T14:13:02.693Z [ 3256: 2676] I [push]: [connect] trying direct connection without a proxy 2022-12-15T14:13:02.693Z [ 3256: 2676] I GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps 2022-12-15T14:13:02.754Z [ 3256: 2676] I 200 : sent=0 rcvd=0 elapsed=61ms 2022-12-15T14:13:02.755Z [ 3256: 2676] I [push]: [connect] using server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps without a proxy (peer address 18.156.41.215) 2022-12-15T14:13:02.755Z [ 3256: 2676] I (async) GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps/push/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6 2022-12-15T14:13:18.256Z [ 3256: 2676] W (async) connection failed 2022-12-15T14:13:18.256Z [ 3256: 2676] I [push]: Dropping connection after error 2022-12-15T14:13:18.258Z [ 3256: 2676] I [backoff] waiting 729s (480s + 249s skew) after failures: 2 2022-12-15T14:25:28.181Z [ 3256: 2676] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;FIM;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SHS;SWC;UI;APPSPROXY/6483xxxx-xxxx-xxxx-xxxx-xxxxxx22bcc6 2022-12-15T14:25:28.266Z [ 3256: 2676] I 200 : sent=0 rcvd=140 elapsed=85ms 2022-12-15T14:25:28.270Z [ 3256: 2676] I [push]: [connect] trying server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps 2022-12-15T14:25:28.271Z [ 3256: 2676] I [push]: [connect] trying direct connection without a proxy 2022-12-15T14:25:28.271Z [ 3256: 2676] I GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps 2022-12-15T14:25:28.337Z [ 3256: 2676] I 200 : sent=0 rcvd=0 elapsed=66ms
I believe it is an issue again with Sophos Firewall not working 100% with Sophos EP Software.
Firewall quit's the connection as can be seen in log and wireshark on endpoint.
I can see tons of requests going into the wrong firewall rule (5 which is a block rule) from many hosts in our network.
The DNS records have a TTL of only 4 seconds. I think that is too quick for a Sophos firewall.
19.5 MR1 has a fix: