Sophos Heartbeat At risk

Hi,

We’re having frequent issues with a number of Lenovo ThinkPad T14s laptops showing red heartbeat (at risk) status in the XG dashboard. Firewall rules with heartbeat restrictions are also blocking traffic since the status is red. In Sophos Central and on the endpoint the status is green, and all services are running and healthy. Heartbeat status is usually stuck as red and does not change to green or missing. Older laptops such as ThinkPad 490s/T480s are not affected by this.

Heartbeat.log on the endpoints shows it is sending “health:3” as health status which I understand is red health. The log also shows “Connection closed (network error)” after every heartbeat communication to 52.5.76.173 on port 8347.

XG is on the latest firmware 18.0.5-MR5-Build586 and the endpoints on the latest version and rebooted.

Is this a known issue?

Heartbeat.log

a 2021-05-24T07:50:42.217Z [6284:22556] - Received request to enable enhanced application control
a 2021-05-24T07:50:42.217Z [6284:22556] - Sending endpoint state list request
a 2021-05-24T07:50:42.217Z [6284:22556] - Sending login status.
a 2021-05-24T07:50:42.218Z [6284:22556] - Received response to endpoint state list request, size: 1
a 2021-05-24T07:50:52.160Z [6284:22556] - Sending health status: {"health":3}
a 2021-05-24T07:50:52.163Z [6284:22556] - Received notification of endpoint state changes, size: 1
a 2021-05-24T07:50:52.483Z [6284:22556] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edge\application\msedge.exe
a 2021-05-24T07:50:52.486Z [6284:22556] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edge\application\msedge.exe
a 2021-05-24T07:55:19.836Z [6284:22556] - Received request to disable enhanced application control for C:\program files (x86)\internet explorer\iexplore.exe
a 2021-05-24T07:55:20.224Z [6284:22556] - Received request to disable enhanced application control for C:\program files\internet explorer\iexplore.exe
a 2021-05-24T08:05:12.930Z [6284:22556] - Sending network status. Active Interfaces:
MAC: 34:2E:B7:0E:6C:A8 - INET: x.x.x.107 - INET6: fe80::1c3a:a66e:1766:8a9e
MAC: 54:05:DB:25:FF:E5 - INET: x.x.x.109 - INET6: fe80::3c8f:c24:ebad:c943
a 2021-05-24T08:05:12.935Z [6284:22556] - Connection closed (network error).
a 2021-05-24T08:05:13.993Z [6284:22556] - Connection succeeded.
a 2021-05-24T08:05:13.993Z [6284:22556] - Connected to 'ec13ffbf-c542-41b2-8ff6-dc070df936d9' at IP address 52.5.76.173 on port 8347
a 2021-05-24T08:05:14.027Z [6284:22556] - Sending network status. Active Interfaces:
MAC: 34:2E:B7:0E:6C:A8 - INET: x.x.x.107 - INET6: fe80::1c3a:a66e:1766:8a9e
MAC: 54:05:DB:25:FF:E5 - INET: x.x.x.109 - INET6: fe80::3c8f:c24:ebad:c943
a 2021-05-24T08:05:14.038Z [6284:22556] - Received request to enable enhanced application control
a 2021-05-24T08:05:14.038Z [6284:22556] - Sending endpoint state list request
a 2021-05-24T08:05:14.039Z [6284:22556] - Sending login status.
a 2021-05-24T08:05:14.039Z [6284:22556] - Received response to endpoint state list request, size: 0
a 2021-05-24T08:05:15.439Z [6284:22556] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edge\application\msedge.exe
a 2021-05-24T08:05:25.367Z [6284:22556] - Sending health status: {"health":3}
a 2021-05-24T08:05:27.935Z [6284:22556] - Received notification of endpoint state changes, size: 1
a 2021-05-24T08:06:25.227Z [6284:22556] - Connection closed (network error).
a 2021-05-24T08:06:27.312Z [6284:22556] - Connection succeeded.
a 2021-05-24T08:06:27.312Z [6284:22556] - Connected to 'ec13ffbf-c542-41b2-8ff6-dc070df936d9' at IP address 52.5.76.173 on port 8347
a 2021-05-24T08:06:27.314Z [6284:22556] - Connection closed (network error).

Parents Reply Children
  • Hi Yashraj,

    Thank you. Please see screenshots below. I don't have access to this endpoint now but have attached health status from the registry. Earlier this morning the agent showed the green check on the endpoint and seems to check in with Sophos Central correctly. This endpoint showed green health last week and red the week before.

    Regards, Anders

  • if you have access to those devices again check the log here:

    C:\ProgramData\Sophos\Heartbeat\Logs

    Look for "Sending network status. Active Interfaces:" do you have changing interfaces there? Maybe caused by more aggressive energy saving features on the T14s

    I could also imagine different network adaters or virtual adapters on the T14s not working properly with heartbeat.

  • sorry, I did'nt read your initial post properly. This does'nt look like energy saving issues (Connection closed (network error))

  • I've opened a support case today and will be onsite tomorrow for more investigation. It seems to choose the correct network adapter, but Sophos agent is sending health 3 although no problems are visible in the interface. Endpoints are usually stuck as "at risk" for some days and then go back to green, but there's no flapping between red-missing-green. I've tried disabling power saving on the ethernet adapter and also disabled power saving 802.az EEE on the switches but it makes no difference.

    Heartbeat.log

    a 2021-05-24T08:05:25.367Z [6284:22556] - Sending health status: {"health":3}

  • Hi ,

    Could you please provide the case number to update it with the information shared in this thread?

    Thanks,

    Yashraj Singha

    Community Team Lead, Support & Services| Sophos Technical Support
    Support Videos | Product Documentation@SophosSupport | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' button.

  • Hi Yashraj,

    Sophos Support Case 04027645

    Regards, Anders

  • We are experiencing a similar issue. Our devices are Lenovo ThinkPad 14 & Thinkbook 13s, stuck each day for hours "at risk".

    Was there any update to this issue?

    a 2021-05-25T07:01:09.147Z [5168:6608] - Connection closed (network error).
    a 2021-05-25T07:01:10.180Z [5168:6608] - Connection succeeded.
    a 2021-05-25T07:01:10.180Z [5168:6608] - Connected to 'd5340c68-99e5-439b-8429-95cd57919e88' at IP address 52.5.76.173 on port 8347
    a 2021-05-25T07:01:10.192Z [5168:6608] - Sending network status. Active Interfaces:
    MAC: 48:2A:E3:9A:70:44 - INET: 192.168.96.131 - INET6:
    a 2021-05-25T07:01:10.229Z [5168:6608] - Received request to enable enhanced application control
    a 2021-05-25T07:01:10.230Z [5168:6608] - Sending login status.
    a 2021-05-25T07:01:13.690Z [5168:6608] - Sending health status: {"admin":1, "health":1, "service":1, "threat":1}
    a 2021-05-25T07:03:59.780Z [5168:6608] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edge\application\msedge.exe
    a 2021-05-25T07:54:17.028Z [5168:5656] - ----------------------------------------------------------------------------------------------------
    a 2021-05-25T07:54:17.031Z [5168:5656] - Stopped Heartbeat
    a 2021-05-25T07:54:17.031Z [5168:5656] - ----------------------------------------------------------------------------------------------------

    a 2021-05-25T07:54:19.858Z [13220:14304] - Starting Heartbeat version 1.11.194.0
    a 2021-05-25T07:54:19.858Z [13220:14304] - ----------------------------------------------------------------------------------------------------
    a 2021-05-25T07:54:19.882Z [13220:3992] - Connection succeeded.
    a 2021-05-25T07:54:19.882Z [13220:3992] - Connected to 'd5340c68-99e5-439b-8429-95cd57919e88' at IP address 52.5.76.173 on port 8347
    a 2021-05-25T07:54:19.901Z [13220:3992] - Sending network status. Active Interfaces:
    MAC: 48:2A:E3:9A:70:44 - INET: 192.168.96.131 - INET6:
    a 2021-05-25T07:54:19.926Z [13220:3992] - Received request to enable enhanced application control
    a 2021-05-25T07:54:19.927Z [13220:3992] - Sending login status.
    a 2021-05-25T07:54:19.937Z [13220:3992] - Sending health status: {"health":3}
    a 2021-05-25T09:19:55.924Z [13220:3992] - Received request to disable enhanced application control for C:\program files (x86)\microsoft\edge\application\msedge.exe

  • Thank you, I'll keep an eye on this ticket. Please PM me if you need anything else. 

    Thanks,

    Yashraj Singha

    Community Team Lead, Support & Services| Sophos Technical Support
    Support Videos | Product Documentation@SophosSupport | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' button.

  • Hi ,

    Sending health status: {"admin":1, "health":1, "service":1, "threat":1}

    This suggests that there was a threat in your system. Are there are any outstanding alerts on Sophos Central? 

    What is the current status of the endpoint is on the device and Sophos Central? Are you facing a similar issue as mentioned in this thread where the device is reported with Red status on Sophos (XG) Firewall and green on the device and Sophos Central? 

    Thanks,

    Yashraj Singha

    Community Team Lead, Support & Services| Sophos Technical Support
    Support Videos | Product Documentation@SophosSupport | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' button.

  • Hi Chris,

    I’ve done some troubleshooting yesterday and today and on computers where the docking station firmware (ThinkPad Thunderbolt 3 Dock Gen 2)  has been updated to the latest version, the problem has cleared for now at least, but it will probably take a few days to see if this was the cause.