XG Firewall marking Endpoint to "at risk" - Endpoint reporting Heartbeat Status 3

Why is this endpoint reporting Heartbeat Status Red to our XG Firewall this morning? Status "At Risk"!

As result the user cannot access most applications.

XG Showing this:

Central is showing this:

XG Log:

XG430_WP02_SFOS 18.0.5 MR-5-Build586# grep "xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx" /log/heartbeatd.log
2021-06-14 09:28:19 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <3> -> <1>
2021-06-14 09:28:19 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.40)
2021-06-14 09:28:25 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.40) health: 3
2021-06-14 09:32:00 INFO SacProcessor.cpp[17627]:64 discardApp - Sent switchOffConnectionInfo request to endpoint: <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>, Application path :C:\134program files (x86)\134mozilla firefox\134firefox.exe
2021-06-14 09:32:50 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <1> -> <5>
2021-06-14 09:32:53 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <5> -> <1>
2021-06-14 09:32:53 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.40)
2021-06-14 09:32:55 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.40) health: 3
2021-06-14 09:37:01 INFO SacProcessor.cpp[17627]:64 discardApp - Sent switchOffConnectionInfo request to endpoint: <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>, Application path :C:\134program files (x86)\134mozilla firefox\134firefox.exe
2021-06-14 10:02:23 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <1> -> <3>
2021-06-14 10:02:31 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <3> -> <1>
2021-06-14 10:02:31 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.15)
2021-06-14 10:02:34 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.15) health: 3
2021-06-14 10:13:31 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <1> -> <3>
2021-06-14 10:13:34 INFO EndpointStorage.cpp[17627]:114 endpoint_connectivity_cb - Connectivity changed for <xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx>: <3> -> <1>
2021-06-14 10:13:35 INFO EpStateListBroker.cpp[17627]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx(SSL-VPN-IP.10)
2021-06-14 10:13:44 INFO ModuleStatus.cpp[17627]:138 processMessageStatus - Status request received from endpoint: xxxxxxx-4c58-4236-9dbc-xxxxxxxxxxx (SSL-VPN-IP.10) health: 3



changed topic to better match the problem
[bearbeitet von: LHerzog um 11:10 AM (GMT -7) am 26 Aug 2021]
Parents
  • Hi LHerzog,

    I have the same behaviour on some Lenovo notebooks. Do you have a solution yet?
    I just had a 90 minute support session with Sophos support. Result: Client/Firewall/Central communication OK. I was advised to reinstall Central on the client and then report back to support.

    Best

    M

  • Hello,  no solution found so far.

    I opened the case with XG team, now they moved it over to the Intercept X Team, because they believe the change is from the Endpoint. This is what I think too.

    They requested the following:

    1. Can you tell me exactly how the machine is connected to the network, including topology (important)
    2. Is this a laptop/desktop,
    3. If this is a laptop is there a dock in use,
    4. How long has this been going on for,
    5. Has there been any changes to the network such as a replacement modem/cables/changes in network topology etc.

    Could you please send a fresh SDU logs from the Endpoint/Server and please enable remote assistance to Sophos Central.

    I'm currently collecting this information.

    We noticed it on Dell Notebooks and a MS Surface. All with some native or USB-C Docks. Partially connected to LAN and WiFi at the same time (may be some issue).

    we were discussing a similar issue here as well: community.sophos.com/.../sophos-heartbeat---red-in-xg-but-green-in-central

  • I got an update from support, which is exactly as described above by  :

    This has been observed when the Sophos Endpoint event store has become full, to resolve this please follow these steps:

     Disable the Tamper Protection (if enabled).

    1. Go to services.msc and stop the Sophos Health Service.
    2. Browse to the following folder: <C:\ProgramData\Sophos\Health\Event Store\Database>
    3. Rename events.db to events.orig.
    4. Restart the Sophos Health Service.
    5. Open the Task Manager and kill the Sophos UI.exe process.
    6. Launch a new Sophos UI.exe process from the location below: <C:\Program Files\Sophos\Sophos> UI

     This will purge your local event store and stop the endpoint sending status 3 events to your Firewall.

    They described it like: the endpoint has a problem and then reports status 3. XG sees, the Client has a problem and reports "at risk".

    Would be cool, to generate an event in Sophos central via MCS "eventlog full, refer to KBxxxxxxxx"

    Or just implement self-healing if you know about this issue.

  • it looks like the workaround does not fix the issue properly or this issue is caused also by other problems on some clients.

    we still have clients at "at risk" on XG sometimes where we recreated the db days or hours ago.

    and still in central the client is only showing it's nice green status without any issues...

  • So I created an other case for this:

    04399266 as a follow-up for cases 04121743, 04234642

  • Infos.

    in one client's SDU that had  the issue on 2021-09-13 I found in health.log

    2021-09-12T12:50:05.580Z [ 5384:19892] [v2.7.28.0] INFO  Disconnecting client from pipe, as client has exited
    2021-09-13T06:49:25.244Z [ 5440: 5444] [v2.7.28.0] INFO  ----------------------------------------------------------------------------------------------------
    2021-09-13T06:49:25.247Z [ 5440: 5444] [v2.7.28.0] INFO  Starting version 2.7.28.0 of the Sophos Health Service.
    2021-09-13T06:49:25.247Z [ 5440: 5444] [v2.7.28.0] INFO  ----------------------------------------------------------------------------------------------------
    2021-09-13T06:49:25.300Z [ 5440: 5976] [v2.7.28.0] INFO  SAU Policy Features have changed: APPCNTRL AV CLEAN CORE DLP DVCCNTRL EFW HBT LIVEQUERY LIVETERMINAL NTP SAV SDU WEBCNTRL XPD
    2021-09-13T06:49:25.334Z [ 5440: 6088] [v2.7.28.0] INFO  service tamper protection enabled
    2021-09-13T06:49:25.348Z [ 5440: 6084] [v2.7.28.0] INFO  Ignored service check results: during automatic startup grace period
    2021-09-13T06:49:40.375Z [ 5440: 6084] [v2.7.28.0] INFO  Ignored service check results: during automatic startup grace period
    2021-09-13T06:49:57.875Z [ 5440: 6020] [v2.7.28.0] INFO  Client has connected to pipe
    2021-09-13T06:49:58.381Z [ 5440: 6020] [v2.7.28.0] INFO  Client has connected to pipe
    2021-09-13T14:46:40.295Z [ 5440: 6020] [v2.7.28.0] INFO  Client has connected to pipe
    2021-09-13T14:46:41.340Z [ 5440: 1124] [v2.7.28.0] INFO  Disconnecting client from pipe, as client has exited
    2021-09-14T06:21:09.688Z [ 5492: 5496] [v2.7.28.0] INFO  ----------------------------------------------------------------------------------------------------

    So nothing.

    If there would have been a risk, it would look like this

    2021-09-03T08:44:20.779Z [ 5508: 6212] [v2.7.28.0] INFO  Processing event id: {14357A89-ED00-4819-ABEC-475D7A059850}
    2021-09-03T08:44:20.792Z [ 5508: 6212] [v2.7.28.0] INFO  Health state has changed to - Overall: 1, Service: 0, Threat: 1

    Then, in the same time, the XG received  ~360 Status changes of the client heartbeat between health 1 and 3 and back to 1.

    This can be seen in the hearbteat.log on the client.

    a 2021-09-13T06:49:26.337Z [5544:7224] - ----------------------------------------------------------------------------------------------------
    a 2021-09-13T06:49:26.337Z [5544:7224] - Starting Heartbeat version 1.11.194.0
    a 2021-09-13T06:49:26.337Z [5544:7224] - ----------------------------------------------------------------------------------------------------
    a 2021-09-13T06:49:26.365Z [5544:7852] - Connection failed.
    a 2021-09-13T06:49:41.395Z [5544:7852] - Connection succeeded.
    a 2021-09-13T06:49:41.395Z [5544:7852] - Connected to 'xxxxxxxxx-xxxx-xxxx-xxxx' at IP address 52.5.76.173 on port 8347
    a 2021-09-13T06:49:41.416Z [5544:7852] - Sending network status. Active Interfaces:
    MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx
    a 2021-09-13T06:49:41.440Z [5544:7852] - Received request to enable enhanced application control
    a 2021-09-13T06:49:41.440Z [5544:7852] - Sending endpoint state list request
    a 2021-09-13T06:49:41.440Z [5544:7852] - Sending login status.
    a 2021-09-13T06:49:41.441Z [5544:7852] - Received response to endpoint state list request, size: 2
    a 2021-09-13T06:49:55.389Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T06:50:20.497Z [5544:7852] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edgewebview\application\93.0.961.44\msedgewebview2.exe
    a 2021-09-13T06:50:26.025Z [5544:7852] - Sending login status.
    a 2021-09-13T06:51:41.590Z [5544:7852] - Sending network status. Active Interfaces:
    MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx
    a 2021-09-13T06:51:41.591Z [5544:7852] - Connection closed (network error).
    a 2021-09-13T06:51:42.632Z [5544:7852] - Connection succeeded.
    a 2021-09-13T06:51:42.632Z [5544:7852] - Connected to 'xxxxxxxxx-xxxx-xxxx-xxxx' at IP address 52.5.76.173 on port 8347
    a 2021-09-13T06:51:42.651Z [5544:7852] - Sending network status. Active Interfaces:
    MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx
    a 2021-09-13T06:51:42.680Z [5544:7852] - Received request to enable enhanced application control
    a 2021-09-13T06:51:42.680Z [5544:7852] - Sending endpoint state list request
    a 2021-09-13T06:51:42.680Z [5544:7852] - Sending login status.
    a 2021-09-13T06:51:42.681Z [5544:7852] - Received response to endpoint state list request, size: 2
    a 2021-09-13T06:51:55.557Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T06:52:50.182Z [5544:7852] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edgeupdate\microsoftedgeupdate.exe
    a 2021-09-13T06:54:29.157Z [5544:7852] - Received request to disable enhanced application control for C:\program files\mozilla firefox\firefox.exe
    a 2021-09-13T07:50:20.320Z [5544:7852] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edgewebview\application\93.0.961.44\msedgewebview2.exe
    a 2021-09-13T09:50:52.587Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T09:55:22.235Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T10:18:29.366Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T10:19:02.028Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T10:29:36.148Z [5544:7852] - Connection closed (network error).
    a 2021-09-13T10:30:06.189Z [5544:7852] - Connection failed.
    a 2021-09-13T11:30:29.866Z [5544:7852] - Connection succeeded.
    a 2021-09-13T11:30:29.866Z [5544:7852] - Connected to 'xxxxxxxxx-xxxx-xxxx-xxxx' at IP address 52.5.76.173 on port 8347
    a 2021-09-13T11:32:17.277Z [5544:7852] - Sending network status. Active Interfaces:
    MAC: E4:B9:7A:86:24:83 - INET: 172.xxx.xxx.xxx - INET6: xxxxxxxxxxxxx
    a 2021-09-13T11:32:17.279Z [5544:7852] - Received request to enable enhanced application control
    a 2021-09-13T11:32:17.279Z [5544:7852] - Sending endpoint state list request
    a 2021-09-13T11:32:17.279Z [5544:7852] - Sending login status.
    a 2021-09-13T11:32:17.280Z [5544:7852] - Received response to endpoint state list request, size: 2
    a 2021-09-13T11:32:19.288Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T11:32:32.012Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T11:32:32.014Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T11:33:19.299Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T11:33:19.302Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T11:33:32.075Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T11:33:34.305Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T11:33:37.287Z [5544:7852] - Received request to disable enhanced application control for C:\program files\mozilla firefox\firefox.exe
    a 2021-09-13T11:34:19.317Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T11:34:19.320Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T11:34:32.168Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T11:34:34.323Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T11:35:19.337Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T11:35:19.340Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T11:35:32.258Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T11:35:34.343Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T11:36:19.350Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    
    ***
    this keeps going for hours until shutdown of computer
    ***
    
    a 2021-09-13T15:22:11.022Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:22:21.080Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T15:22:26.021Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:23:11.026Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T15:23:11.028Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:23:21.178Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T15:23:26.033Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:24:11.028Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T15:24:11.030Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:24:21.249Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T15:24:26.032Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:25:11.045Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T15:25:11.177Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:25:21.320Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T15:25:26.180Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:26:11.047Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T15:26:11.049Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:26:21.413Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T15:26:26.053Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:27:11.057Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T15:27:11.059Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:27:21.479Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T15:27:26.150Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:28:11.065Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T15:28:11.068Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:28:21.555Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T15:28:26.072Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:29:11.079Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T15:29:11.080Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:29:21.682Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T15:29:26.083Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:30:11.087Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T15:30:11.090Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:30:21.741Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T15:30:26.091Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:31:11.097Z [5544:7852] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-09-13T15:31:11.099Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-13T15:31:21.841Z [5544:7852] - Sending health status: {"health":3}
    a 2021-09-13T15:31:26.100Z [5544:7852] - Received notification of endpoint state changes, size: 1
    a 2021-09-14T06:21:13.523Z [5500:7260] - ----------------------------------------------------------------------------------------------------

  • Hello LHerzog,

    I sent you a DM to follow up on this thread.

    Kushal Lakhan
    Global Community Support Engineer
    Are you a Sophos Partner? | Product Documentation | @SophosSupport | Sign up for SMS Alerts
    If a post solves your question, please use the "Verify Answer" button.
    The New Home of Sophos Support Videos!  Visit Sophos Techvids
Reply Children