3CX DLL-Sideloading attack: What you need to know
Why is this endpoint reporting Heartbeat Status Red to our XG Firewall this morning? Status "At Risk"!
As result the user cannot access most applications.
XG Showing this:
Central is showing this:
XG Log:
XG430_WP02_SFOS 18.0.5 MR-5-Build586# grep "xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx" /log/heartbeatd.log 2021-06-14 09:28:19 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <3> -> <1> 2021-06-14 09:28:19 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.40) 2021-06-14 09:28:25 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.40) health: 3 2021-06-14 09:32:00 INFO SacProcessor.cpp[17627]:64 discardApp - Sent switchOffConnectionInfo request to endpoint: <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>, Application path :C:\134program files (x86)\134mozilla firefox\134firefox.exe 2021-06-14 09:32:50 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <1> -> <5> 2021-06-14 09:32:53 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <5> -> <1> 2021-06-14 09:32:53 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.40) 2021-06-14 09:32:55 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.40) health: 3 2021-06-14 09:37:01 INFO SacProcessor.cpp[17627]:64 discardApp - Sent switchOffConnectionInfo request to endpoint: <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>, Application path :C:\134program files (x86)\134mozilla firefox\134firefox.exe 2021-06-14 10:02:23 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <1> -> <3> 2021-06-14 10:02:31 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <3> -> <1> 2021-06-14 10:02:31 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.15) 2021-06-14 10:02:34 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.15) health: 3 2021-06-14 10:13:31 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <1> -> <3> 2021-06-14 10:13:34 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <3> -> <1> 2021-06-14 10:13:35 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.10) 2021-06-14 10:13:44 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.10) health: 3
Hi LHerzog,
I have the same behaviour on some Lenovo notebooks. Do you have a solution yet?I just had a 90 minute support session with Sophos support. Result: Client/Firewall/Central communication OK. I was advised to reinstall Central on the client and then report back to support.
Best
M
Hello, Dr Brezner no solution found so far.
I opened the case with XG team, now they moved it over to the Intercept X Team, because they believe the change is from the Endpoint. This is what I think too.
They requested the following:
Could you please send a fresh SDU logs from the Endpoint/Server and please enable remote assistance to Sophos Central.
I'm currently collecting this information.
We noticed it on Dell Notebooks and a MS Surface. All with some native or USB-C Docks. Partially connected to LAN and WiFi at the same time (may be some issue).
we were discussing a similar issue here as well: community.sophos.com/.../sophos-heartbeat---red-in-xg-but-green-in-central
I got an update from support, which is exactly as described above by Sophos User930 :
This has been observed when the Sophos Endpoint event store has become full, to resolve this please follow these steps:
Disable the Tamper Protection (if enabled).
This will purge your local event store and stop the endpoint sending status 3 events to your Firewall.
They described it like: the endpoint has a problem and then reports status 3. XG sees, the Client has a problem and reports "at risk".
Would be cool, to generate an event in Sophos central via MCS "eventlog full, refer to KBxxxxxxxx"
Or just implement self-healing if you know about this issue.
it looks like the workaround does not fix the issue properly or this issue is caused also by other problems on some clients.
we still have clients at "at risk" on XG sometimes where we recreated the db days or hours ago.
and still in central the client is only showing it's nice green status without any issues...
So I created an other case for this:
04399266 as a follow-up for cases 04121743, 04234642
Infos.
in one client's SDU that had the issue on 2021-09-13 I found in health.log
2021-09-12T12:50:05.580Z [ 5384:19892] [v2.7.28.0] INFO Disconnecting client from pipe, as client has exited2021-09-13T06:49:25.244Z [ 5440: 5444] [v2.7.28.0] INFO ----------------------------------------------------------------------------------------------------2021-09-13T06:49:25.247Z [ 5440: 5444] [v2.7.28.0] INFO Starting version 2.7.28.0 of the Sophos Health Service.2021-09-13T06:49:25.247Z [ 5440: 5444] [v2.7.28.0] INFO ----------------------------------------------------------------------------------------------------2021-09-13T06:49:25.300Z [ 5440: 5976] [v2.7.28.0] INFO SAU Policy Features have changed: APPCNTRL AV CLEAN CORE DLP DVCCNTRL EFW HBT LIVEQUERY LIVETERMINAL NTP SAV SDU WEBCNTRL XPD 2021-09-13T06:49:25.334Z [ 5440: 6088] [v2.7.28.0] INFO service tamper protection enabled2021-09-13T06:49:25.348Z [ 5440: 6084] [v2.7.28.0] INFO Ignored service check results: during automatic startup grace period2021-09-13T06:49:40.375Z [ 5440: 6084] [v2.7.28.0] INFO Ignored service check results: during automatic startup grace period2021-09-13T06:49:57.875Z [ 5440: 6020] [v2.7.28.0] INFO Client has connected to pipe2021-09-13T06:49:58.381Z [ 5440: 6020] [v2.7.28.0] INFO Client has connected to pipe2021-09-13T14:46:40.295Z [ 5440: 6020] [v2.7.28.0] INFO Client has connected to pipe2021-09-13T14:46:41.340Z [ 5440: 1124] [v2.7.28.0] INFO Disconnecting client from pipe, as client has exited2021-09-14T06:21:09.688Z [ 5492: 5496] [v2.7.28.0] INFO ----------------------------------------------------------------------------------------------------
So nothing.
If there would have been a risk, it would look like this
2021-09-03T08:44:20.779Z [ 5508: 6212] [v2.7.28.0] INFO Processing event id: {14357A89-ED00-4819-ABEC-475D7A059850}2021-09-03T08:44:20.792Z [ 5508: 6212] [v2.7.28.0] INFO Health state has changed to - Overall: 1, Service: 0, Threat: 1
Then, in the same time, the XG received ~360 Status changes of the client heartbeat between health 1 and 3 and back to 1.
This can be seen in the hearbteat.log on the client.
a 2021-09-13T06:49:26.337Z [5544:7224] - ---------------------------------------------------------------------------------------------------- a 2021-09-13T06:49:26.337Z [5544:7224] - Starting Heartbeat version 1.11.194.0 a 2021-09-13T06:49:26.337Z [5544:7224] - ---------------------------------------------------------------------------------------------------- a 2021-09-13T06:49:26.365Z [5544:7852] - Connection failed. a 2021-09-13T06:49:41.395Z [5544:7852] - Connection succeeded. a 2021-09-13T06:49:41.395Z [5544:7852] - Connected to 'xxxxxxxxx-xxxx-xxxx-xxxx' at IP address 52.5.76.173 on port 8347 a 2021-09-13T06:49:41.416Z [5544:7852] - Sending network status. Active Interfaces: MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx a 2021-09-13T06:49:41.440Z [5544:7852] - Received request to enable enhanced application control a 2021-09-13T06:49:41.440Z [5544:7852] - Sending endpoint state list request a 2021-09-13T06:49:41.440Z [5544:7852] - Sending login status. a 2021-09-13T06:49:41.441Z [5544:7852] - Received response to endpoint state list request, size: 2 a 2021-09-13T06:49:55.389Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T06:50:20.497Z [5544:7852] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edgewebview\application\93.0.961.44\msedgewebview2.exe a 2021-09-13T06:50:26.025Z [5544:7852] - Sending login status. a 2021-09-13T06:51:41.590Z [5544:7852] - Sending network status. Active Interfaces: MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx a 2021-09-13T06:51:41.591Z [5544:7852] - Connection closed (network error). a 2021-09-13T06:51:42.632Z [5544:7852] - Connection succeeded. a 2021-09-13T06:51:42.632Z [5544:7852] - Connected to 'xxxxxxxxx-xxxx-xxxx-xxxx' at IP address 52.5.76.173 on port 8347 a 2021-09-13T06:51:42.651Z [5544:7852] - Sending network status. Active Interfaces: MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx a 2021-09-13T06:51:42.680Z [5544:7852] - Received request to enable enhanced application control a 2021-09-13T06:51:42.680Z [5544:7852] - Sending endpoint state list request a 2021-09-13T06:51:42.680Z [5544:7852] - Sending login status. a 2021-09-13T06:51:42.681Z [5544:7852] - Received response to endpoint state list request, size: 2 a 2021-09-13T06:51:55.557Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T06:52:50.182Z [5544:7852] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edgeupdate\microsoftedgeupdate.exe a 2021-09-13T06:54:29.157Z [5544:7852] - Received request to disable enhanced application control for C:\program files\mozilla firefox\firefox.exe a 2021-09-13T07:50:20.320Z [5544:7852] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edgewebview\application\93.0.961.44\msedgewebview2.exe a 2021-09-13T09:50:52.587Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T09:55:22.235Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T10:18:29.366Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T10:19:02.028Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T10:29:36.148Z [5544:7852] - Connection closed (network error). a 2021-09-13T10:30:06.189Z [5544:7852] - Connection failed. a 2021-09-13T11:30:29.866Z [5544:7852] - Connection succeeded. a 2021-09-13T11:30:29.866Z [5544:7852] - Connected to 'xxxxxxxxx-xxxx-xxxx-xxxx' at IP address 52.5.76.173 on port 8347 a 2021-09-13T11:32:17.277Z [5544:7852] - Sending network status. Active Interfaces: MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx a 2021-09-13T11:32:17.279Z [5544:7852] - Received request to enable enhanced application control a 2021-09-13T11:32:17.279Z [5544:7852] - Sending endpoint state list request a 2021-09-13T11:32:17.279Z [5544:7852] - Sending login status. a 2021-09-13T11:32:17.280Z [5544:7852] - Received response to endpoint state list request, size: 2 a 2021-09-13T11:32:19.288Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T11:32:32.012Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T11:32:32.014Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T11:33:19.299Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T11:33:19.302Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T11:33:32.075Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T11:33:34.305Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T11:33:37.287Z [5544:7852] - Received request to disable enhanced application control for C:\program files\mozilla firefox\firefox.exe a 2021-09-13T11:34:19.317Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T11:34:19.320Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T11:34:32.168Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T11:34:34.323Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T11:35:19.337Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T11:35:19.340Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T11:35:32.258Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T11:35:34.343Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T11:36:19.350Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} *** this keeps going for hours until shutdown of computer *** a 2021-09-13T15:22:11.022Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:22:21.080Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T15:22:26.021Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:23:11.026Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T15:23:11.028Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:23:21.178Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T15:23:26.033Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:24:11.028Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T15:24:11.030Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:24:21.249Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T15:24:26.032Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:25:11.045Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T15:25:11.177Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:25:21.320Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T15:25:26.180Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:26:11.047Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T15:26:11.049Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:26:21.413Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T15:26:26.053Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:27:11.057Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T15:27:11.059Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:27:21.479Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T15:27:26.150Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:28:11.065Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T15:28:11.068Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:28:21.555Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T15:28:26.072Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:29:11.079Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T15:29:11.080Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:29:21.682Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T15:29:26.083Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:30:11.087Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T15:30:11.090Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:30:21.741Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T15:30:26.091Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:31:11.097Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1} a 2021-09-13T15:31:11.099Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-13T15:31:21.841Z [5544:7852] - Sending health status: {"health":3} a 2021-09-13T15:31:26.100Z [5544:7852] - Received notification of endpoint state changes, size: 1 a 2021-09-14T06:21:13.523Z [5500:7260] - ----------------------------------------------------------------------------------------------------
Hello LHerzog,
I sent you a DM to follow up on this thread.
Thanks. Got it.
Info for other users: probably this is some kind of disclosed information but there will be an update of a Intercept-X Component by the end of the Month which should fix the issue.
Sounds good. We have more and more clients with this problem. All the described work arounds did not help or only in the short term.
Hello everyone, we have the same problem with some clients. Particularly with DELL laptops. I hope the SOPHOS team can fix this issue shortly.
check, if they have energy option S0 active. The probably do.
cmd: powercfg /a
If so this may be set to e.g. S3 as workaround until end of Semptember until the new Version of Core Agent 2.19.7 is released.
Thanks LHerzog, we are going to test it while we wait for the new agent version.