This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Sophos Firewall reported computer not sending heartbeat signals

Since November an increasing number of endpoints is reported from Central with "Sophos Firewall SN reported computer not sending heartbeat signals"

We upgraded our HQ XG from 18.5.4 to 19.0.1 on  Nov 12th but the issue started already before as you can see from the screenshots.

Before that, we only received this alerts occasionally.  Sometimes the message comes multiple times per day for a machine, then a few days no message is created even if the computer is still in use.

What is the issue here?

Central Region is Central Europe

One Computer:



This thread was automatically locked due to age.
Parents
  • Are you able to see any similar errors in the logs located at "C:\ProgramData\Sophos\Heartbeat\Logs"? 

    Could the device be entering a hibernate or sleep state at the times when these events are generated?

    Kushal Lakhan
    Team Lead, Global Community Support
    Connect with Sophos Support, get alerted, and be informed.
    If a post solves your question, please use the "Verify Answer" button.
    The New Home of Sophos Support Videos!  Visit Sophos Techvids
  • I was on the computer and it was in standby.

    I could see the Intel Networkdriver was frequently dumping something all the time during standby.

    Netwtw10
    7026
    7026 - Dump after return from D3 after cmd

    Netwtw10
    7025
    7025 - Dump after return from D3 before cmd

    .

    Probably causing network flapping which triggers Heartbeat Change.

    In the heartbeat log I could see many, many events during standby mode: network has changed - firewall may disconnect

    .

    2022-11-16T09:21:38.596Z [ 5212: 6340] A Sending network status
    2022-11-16T09:21:38.596Z [ 5212: 6340] A The network status has changed, the Firewall may disconnect.
    2022-11-16T09:21:38.598Z [ 5212: 6340] A Connection closed (network error).

    .

    I updated (network) drivers and BIOS at first place and will monitor the situation.

    Can the heartbeat module be tweaked so that it is compatible with Standby?

    Everyone taks about saving energy - would be non-pc to disable standby for heartbeat to work.

  • There is currently a Bug ID under investigation: NC-111152 - Missing Heartbeat behavior for endpoints generating alerts in Central

    __________________________________________________________________________________________________________________

  • it came back. I suspect the Sophos endpoints getting Program updates since monday and that causing the issues. Unfortunately it is impossible to see that easily from Sophos Central. There is only generic "Update succeeded". You need to dig through the log files on the endoints. I don't like that.

  • Those Mails are coming more frequently now and it is annoying. Will Sophos Fix NC-111152?

  • @sophos, you'll never fix it, won't you?

    Those alerts when the computers go to hibernate / sleep are so frequent and useless that everyone is ignoring them. So I'm going to disable them. Good work.

  • Forgot to ask, did you adjust those parameters on the firewall? 

    console> system synchronized-security
    central_registration delay-missing-heartbeat-detection suppress-missing-heartbeat-to-central
    console> system synchronized-security

    __________________________________________________________________________________________________________________

  • yes, I already increased at least some of them. Cannot login to verify currently due to SSH authentication issues - other post in FW forum.

  • firewall is constantly reporting some computers with missing heartbeat. if I check them, they are currently not online on our network because they sleep.

    And it is not a firewall issue from my point of view - the endpoint heartbeat agent should be able to see the upcoming event of windows entering hibernate and then quickly report to the firewall that it will temporarily disconnect from heartbeat.

    Because this is not happening, the firewall just sees a abandoned heartbeat session and correctly reports the issue to central.

    Tweaking delays will not change this when the client (usually) is in hibernate state longer than the delay.

  • The reason of those hits are quite simple: In Hibernation, the client will still send data. But the daemon of Sophos is already closed, as the sub system is already shut down. So the firewall will see the data and will react to the data as interpret them as missing heartbeat.

    If you switch to hibernate, likely the endpoint cannot fetch this information not fast enough until windows will shut down the system. Therefore the missing heartbeat is generated. 

    The ID above is something, where likely the client is not in hibernate, instead the daemon is shutdown for whatever reason. 

    --

    By adding the delay value on the firewall, you likely will decrease those alerts, as the firewall will give some time to the client ip until it will react to it. This will lead to a better reporting, as most windows clients will stop interacting with the gateway after 1-2 minutes going into hibernation. . 

    __________________________________________________________________________________________________________________

  • it is probably something that is also in combination with the NIC Vendor. As it is standard Intel NIC on business Notebooks in our case, this should be quite common. The screenshots from event viewer above show something is happening on the NIC when it is in hibernate. This may revive heartbeat or cause some other issues with that. Anyway my opinion is that this should work unless a customer uses exotic hardware with old drivers. Not a solution is to disable hibernate or disable energy saving feature on the NIC.

  • If you do a tcpdump / packet capture on the IP and do the hibernation, what kind of traffic do you still see? And maybe you will find the reason by researching this traffic further. 

    __________________________________________________________________________________________________________________

Reply
  • If you do a tcpdump / packet capture on the IP and do the hibernation, what kind of traffic do you still see? And maybe you will find the reason by researching this traffic further. 

    __________________________________________________________________________________________________________________

Children
No Data