Sophos UTM: Decommissioning of obsolete URL categorization services CFFS. Click here for important info.

Thread Info
  • State Verified Answer
  • +1 person also asked this people also asked this
  • Locked Locked
  • Replies 26 replies
  • Answers 1 answer
  • Subscribers 18 subscribers
  • Views 24052 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

XG Firewall marking Endpoint to "at risk" - Endpoint reporting Heartbeat Status 3

LHerzog

Why is this endpoint reporting Heartbeat Status Red to our XG Firewall this morning? Status "At Risk"!

As result the user cannot access most applications.

XG Showing this:

Central is showing this:

XG Log:

XG430_WP02_SFOS 18.0.5 MR-5-Build586# grep "xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx" /log/heartbeatd.log
2021-06-14 09:28:19 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <3> -> <1>
2021-06-14 09:28:19 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.40)
2021-06-14 09:28:25 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.40) health: 3
2021-06-14 09:32:00 INFO SacProcessor.cpp[17627]:64 discardApp - Sent switchOffConnectionInfo request to endpoint: <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>, Application path :C:\134program files (x86)\134mozilla firefox\134firefox.exe
2021-06-14 09:32:50 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <1> -> <5>
2021-06-14 09:32:53 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <5> -> <1>
2021-06-14 09:32:53 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.40)
2021-06-14 09:32:55 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.40) health: 3
2021-06-14 09:37:01 INFO SacProcessor.cpp[17627]:64 discardApp - Sent switchOffConnectionInfo request to endpoint: <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>, Application path :C:\134program files (x86)\134mozilla firefox\134firefox.exe
2021-06-14 10:02:23 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <1> -> <3>
2021-06-14 10:02:31 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <3> -> <1>
2021-06-14 10:02:31 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.15)
2021-06-14 10:02:34 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.15) health: 3
2021-06-14 10:13:31 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <1> -> <3>
2021-06-14 10:13:34 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <3> -> <1>
2021-06-14 10:13:35 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.10)
2021-06-14 10:13:44 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.10) health: 3



This thread was automatically locked due to age.
Parents
  • 0

    Hi LHerzog,

    I have the same behaviour on some Lenovo notebooks. Do you have a solution yet?
    I just had a 90 minute support session with Sophos support. Result: Client/Firewall/Central communication OK. I was advised to reinstall Central on the client and then report back to support.

    Best

    M

  • 0 in reply to Dr Brezner

    Hello,  no solution found so far.

    I opened the case with XG team, now they moved it over to the Intercept X Team, because they believe the change is from the Endpoint. This is what I think too.

    They requested the following:

    1. Can you tell me exactly how the machine is connected to the network, including topology (important)
    2. Is this a laptop/desktop,
    3. If this is a laptop is there a dock in use,
    4. How long has this been going on for,
    5. Has there been any changes to the network such as a replacement modem/cables/changes in network topology etc.

    Could you please send a fresh SDU logs from the Endpoint/Server and please enable remote assistance to Sophos Central.

    I'm currently collecting this information.

    We noticed it on Dell Notebooks and a MS Surface. All with some native or USB-C Docks. Partially connected to LAN and WiFi at the same time (may be some issue).

    we were discussing a similar issue here as well: community.sophos.com/.../sophos-heartbeat---red-in-xg-but-green-in-central

Reply
  • 0 in reply to Dr Brezner

    Hello,  no solution found so far.

    I opened the case with XG team, now they moved it over to the Intercept X Team, because they believe the change is from the Endpoint. This is what I think too.

    They requested the following:

    1. Can you tell me exactly how the machine is connected to the network, including topology (important)
    2. Is this a laptop/desktop,
    3. If this is a laptop is there a dock in use,
    4. How long has this been going on for,
    5. Has there been any changes to the network such as a replacement modem/cables/changes in network topology etc.

    Could you please send a fresh SDU logs from the Endpoint/Server and please enable remote assistance to Sophos Central.

    I'm currently collecting this information.

    We noticed it on Dell Notebooks and a MS Surface. All with some native or USB-C Docks. Partially connected to LAN and WiFi at the same time (may be some issue).

    we were discussing a similar issue here as well: community.sophos.com/.../sophos-heartbeat---red-in-xg-but-green-in-central

Children
  • 0 in reply to LHerzog

    I got an update from support, which is exactly as described above by  :

    This has been observed when the Sophos Endpoint event store has become full, to resolve this please follow these steps:

     Disable the Tamper Protection (if enabled).

    1. Go to services.msc and stop the Sophos Health Service.
    2. Browse to the following folder: <C:\ProgramData\Sophos\Health\Event Store\Database>
    3. Rename events.db to events.orig.
    4. Restart the Sophos Health Service.
    5. Open the Task Manager and kill the Sophos UI.exe process.
    6. Launch a new Sophos UI.exe process from the location below: <C:\Program Files\Sophos\Sophos> UI

     This will purge your local event store and stop the endpoint sending status 3 events to your Firewall.

    They described it like: the endpoint has a problem and then reports status 3. XG sees, the Client has a problem and reports "at risk".

    Would be cool, to generate an event in Sophos central via MCS "eventlog full, refer to KBxxxxxxxx"

    Or just implement self-healing if you know about this issue.

  • 0 in reply to LHerzog

    it looks like the workaround does not fix the issue properly or this issue is caused also by other problems on some clients.

    we still have clients at "at risk" on XG sometimes where we recreated the db days or hours ago.

    and still in central the client is only showing it's nice green status without any issues...

  • 0 in reply to LHerzog

    So I created an other case for this:

    04399266 as a follow-up for cases 04121743, 04234642

  • 0 in reply to LHerzog

    Infos.

    in one client's SDU that had  the issue on 2021-09-13 I found in health.log

    2021-09-12T12:50:05.580Z [ 5384:19892] [v2.7.28.0] INFO  Disconnecting client from pipe, as client has exited
    2021-09-13T06:49:25.244Z [ 5440: 5444] [v2.7.28.0] INFO  ----------------------------------------------------------------------------------------------------
    2021-09-13T06:49:25.247Z [ 5440: 5444] [v2.7.28.0] INFO  Starting version 2.7.28.0 of the Sophos Health Service.
    2021-09-13T06:49:25.247Z [ 5440: 5444] [v2.7.28.0] INFO  ----------------------------------------------------------------------------------------------------
    2021-09-13T06:49:25.300Z [ 5440: 5976] [v2.7.28.0] INFO  SAU Policy Features have changed: APPCNTRL AV CLEAN CORE DLP DVCCNTRL EFW HBT LIVEQUERY LIVETERMINAL NTP SAV SDU WEBCNTRL XPD
    2021-09-13T06:49:25.334Z [ 5440: 6088] [v2.7.28.0] INFO  service tamper protection enabled
    2021-09-13T06:49:25.348Z [ 5440: 6084] [v2.7.28.0] INFO  Ignored service check results: during automatic startup grace period
    2021-09-13T06:49:40.375Z [ 5440: 6084] [v2.7.28.0] INFO  Ignored service check results: during automatic startup grace period
    2021-09-13T06:49:57.875Z [ 5440: 6020] [v2.7.28.0] INFO  Client has connected to pipe
    2021-09-13T06:49:58.381Z [ 5440: 6020] [v2.7.28.0] INFO  Client has connected to pipe
    2021-09-13T14:46:40.295Z [ 5440: 6020] [v2.7.28.0] INFO  Client has connected to pipe
    2021-09-13T14:46:41.340Z [ 5440: 1124] [v2.7.28.0] INFO  Disconnecting client from pipe, as client has exited
    2021-09-14T06:21:09.688Z [ 5492: 5496] [v2.7.28.0] INFO  ----------------------------------------------------------------------------------------------------

    So nothing.

    If there would have been a risk, it would look like this

    2021-09-03T08:44:20.779Z [ 5508: 6212] [v2.7.28.0] INFO  Processing event id: {14357A89-ED00-4819-ABEC-475D7A059850}
    2021-09-03T08:44:20.792Z [ 5508: 6212] [v2.7.28.0] INFO  Health state has changed to - Overall: 1, Service: 0, Threat: 1

    Then, in the same time, the XG received  ~360 Status changes of the client heartbeat between health 1 and 3 and back to 1.

    This can be seen in the hearbteat.log on the client.

    a 2021-09-13T06:49:26.337Z [5544:7224] - ----------------------------------------------------------------------------------------------------
a 2021-09-13T06:49:26.337Z [5544:7224] - Starting Heartbeat version 1.11.194.0
a 2021-09-13T06:49:26.337Z [5544:7224] - ----------------------------------------------------------------------------------------------------
a 2021-09-13T06:49:26.365Z [5544:7852] - Connection failed.
a 2021-09-13T06:49:41.395Z [5544:7852] - Connection succeeded.
a 2021-09-13T06:49:41.395Z [5544:7852] - Connected to 'xxxxxxxxx-xxxx-xxxx-xxxx' at IP address 52.5.76.173 on port 8347
a 2021-09-13T06:49:41.416Z [5544:7852] - Sending network status. Active Interfaces:
MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx
a 2021-09-13T06:49:41.440Z [5544:7852] - Received request to enable enhanced application control
a 2021-09-13T06:49:41.440Z [5544:7852] - Sending endpoint state list request
a 2021-09-13T06:49:41.440Z [5544:7852] - Sending login status.
a 2021-09-13T06:49:41.441Z [5544:7852] - Received response to endpoint state list request, size: 2
a 2021-09-13T06:49:55.389Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T06:50:20.497Z [5544:7852] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edgewebview\application\93.0.961.44\msedgewebview2.exe
a 2021-09-13T06:50:26.025Z [5544:7852] - Sending login status.
a 2021-09-13T06:51:41.590Z [5544:7852] - Sending network status. Active Interfaces:
MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx
a 2021-09-13T06:51:41.591Z [5544:7852] - Connection closed (network error).
a 2021-09-13T06:51:42.632Z [5544:7852] - Connection succeeded.
a 2021-09-13T06:51:42.632Z [5544:7852] - Connected to 'xxxxxxxxx-xxxx-xxxx-xxxx' at IP address 52.5.76.173 on port 8347
a 2021-09-13T06:51:42.651Z [5544:7852] - Sending network status. Active Interfaces:
MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx
a 2021-09-13T06:51:42.680Z [5544:7852] - Received request to enable enhanced application control
a 2021-09-13T06:51:42.680Z [5544:7852] - Sending endpoint state list request
a 2021-09-13T06:51:42.680Z [5544:7852] - Sending login status.
a 2021-09-13T06:51:42.681Z [5544:7852] - Received response to endpoint state list request, size: 2
a 2021-09-13T06:51:55.557Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T06:52:50.182Z [5544:7852] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edgeupdate\microsoftedgeupdate.exe
a 2021-09-13T06:54:29.157Z [5544:7852] - Received request to disable enhanced application control for C:\program files\mozilla firefox\firefox.exe
a 2021-09-13T07:50:20.320Z [5544:7852] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edgewebview\application\93.0.961.44\msedgewebview2.exe
a 2021-09-13T09:50:52.587Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T09:55:22.235Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T10:18:29.366Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T10:19:02.028Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T10:29:36.148Z [5544:7852] - Connection closed (network error).
a 2021-09-13T10:30:06.189Z [5544:7852] - Connection failed.
a 2021-09-13T11:30:29.866Z [5544:7852] - Connection succeeded.
a 2021-09-13T11:30:29.866Z [5544:7852] - Connected to 'xxxxxxxxx-xxxx-xxxx-xxxx' at IP address 52.5.76.173 on port 8347
a 2021-09-13T11:32:17.277Z [5544:7852] - Sending network status. Active Interfaces:
MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx
a 2021-09-13T11:32:17.279Z [5544:7852] - Received request to enable enhanced application control
a 2021-09-13T11:32:17.279Z [5544:7852] - Sending endpoint state list request
a 2021-09-13T11:32:17.279Z [5544:7852] - Sending login status.
a 2021-09-13T11:32:17.280Z [5544:7852] - Received response to endpoint state list request, size: 2
a 2021-09-13T11:32:19.288Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T11:32:32.012Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T11:32:32.014Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T11:33:19.299Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T11:33:19.302Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T11:33:32.075Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T11:33:34.305Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T11:33:37.287Z [5544:7852] - Received request to disable enhanced application control for C:\program files\mozilla firefox\firefox.exe
a 2021-09-13T11:34:19.317Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T11:34:19.320Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T11:34:32.168Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T11:34:34.323Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T11:35:19.337Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T11:35:19.340Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T11:35:32.258Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T11:35:34.343Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T11:36:19.350Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}

***
this keeps going for hours until shutdown of computer
***

a 2021-09-13T15:22:11.022Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:22:21.080Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T15:22:26.021Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:23:11.026Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T15:23:11.028Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:23:21.178Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T15:23:26.033Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:24:11.028Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T15:24:11.030Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:24:21.249Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T15:24:26.032Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:25:11.045Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T15:25:11.177Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:25:21.320Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T15:25:26.180Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:26:11.047Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T15:26:11.049Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:26:21.413Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T15:26:26.053Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:27:11.057Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T15:27:11.059Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:27:21.479Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T15:27:26.150Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:28:11.065Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T15:28:11.068Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:28:21.555Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T15:28:26.072Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:29:11.079Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T15:29:11.080Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:29:21.682Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T15:29:26.083Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:30:11.087Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T15:30:11.090Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:30:21.741Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T15:30:26.091Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:31:11.097Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
a 2021-09-13T15:31:11.099Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-13T15:31:21.841Z [5544:7852] - Sending health status: {"health":3}
a 2021-09-13T15:31:26.100Z [5544:7852] - Received notification of endpoint state changes, size: 1
a 2021-09-14T06:21:13.523Z [5500:7260] - ----------------------------------------------------------------------------------------------------

  • 0 in reply to LHerzog

    Hello LHerzog,

    I sent you a DM to follow up on this thread.

    Kushal Lakhan
    Team Lead, Global Community Support
    Connect with Sophos Support, get alerted, and be informed.
    If a post solves your question, please use the "Verify Answer" button.
    The New Home of Sophos Support Videos!  - Visit Sophos Techvids
  • 0 in reply to Qoosh

    Thanks. Got it.

    Info for other users: probably this is some kind of disclosed information but there will be an update of a Intercept-X Component by the end of the Month which should fix the issue.

  • 0 in reply to LHerzog

    Sounds good. We have more and more clients with this problem. All the described work arounds did not help or only in the short term.

  • 0 in reply to Dr Brezner 
    Hello everyone, we have the same problem with some clients. Particularly with DELL laptops.
I hope the SOPHOS team can fix this issue shortly.
  • 0 in reply to Tomas Casaccia

    check, if they have energy option S0 active. The probably do.

    cmd: powercfg /a

    If so this may be set to e.g. S3 as workaround until end of Semptember until the new Version of Core Agent 2.19.7 is released.

  • 0 in reply to LHerzog 
    Thanks LHerzog, we are going to test it while we wait for the new agent version.
Unfiltered HTML