Sophos UTM: Decommissioning of obsolete URL categorization services CFFS.Click here for important info.

This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Sophos Firewall reported computer not sending heartbeat signals

Since November an increasing number of endpoints is reported from Central with "Sophos Firewall SN reported computer not sending heartbeat signals"

We upgraded our HQ XG from 18.5.4 to 19.0.1 on  Nov 12th but the issue started already before as you can see from the screenshots.

Before that, we only received this alerts occasionally.  Sometimes the message comes multiple times per day for a machine, then a few days no message is created even if the computer is still in use.

What is the issue here?

Central Region is Central Europe

One Computer:



This thread was automatically locked due to age.
Parents
  • Are you able to see any similar errors in the logs located at "C:\ProgramData\Sophos\Heartbeat\Logs"? 

    Could the device be entering a hibernate or sleep state at the times when these events are generated?

    Kushal Lakhan
    Team Lead, Global Community Support
    Connect with Sophos Support, get alerted, and be informed.
    If a post solves your question, please use the "Verify Answer" button.
    The New Home of Sophos Support Videos!  Visit Sophos Techvids
  • I was on the computer and it was in standby.

    I could see the Intel Networkdriver was frequently dumping something all the time during standby.

    Netwtw10
    7026
    7026 - Dump after return from D3 after cmd

    Netwtw10
    7025
    7025 - Dump after return from D3 before cmd

    .

    Probably causing network flapping which triggers Heartbeat Change.

    In the heartbeat log I could see many, many events during standby mode: network has changed - firewall may disconnect

    .

    2022-11-16T09:21:38.596Z [ 5212: 6340] A Sending network status
    2022-11-16T09:21:38.596Z [ 5212: 6340] A The network status has changed, the Firewall may disconnect.
    2022-11-16T09:21:38.598Z [ 5212: 6340] A Connection closed (network error).

    .

    I updated (network) drivers and BIOS at first place and will monitor the situation.

    Can the heartbeat module be tweaked so that it is compatible with Standby?

    Everyone taks about saving energy - would be non-pc to disable standby for heartbeat to work.

  • If you do a tcpdump / packet capture on the IP and do the hibernation, what kind of traffic do you still see? And maybe you will find the reason by researching this traffic further. 

    __________________________________________________________________________________________________________________

  • Hello LHerzog,

    DEV has some binary ready for NC-111152, I would recommend you to open a case with Support, and you can mention about NC-111152, the case would get to GES and they can confirm is your issue matches NC-111152 and install the binary, to see if this resolves your issue.

    IF you do this, share the Case ID.

    Regards,


     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support VideosProduct Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.
  • Thank you   - is that binary for endpoint or firewall?

  • Hello LHerzog,

    This would be for the Sophos Firewall.

    Regards,


     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support VideosProduct Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.
  • thanks! I opended case 06153996 maybe you put a hint there for the tech.

  • that case is like a blind flight I guess.
    that's all done so far.


    console> system synchronized-security delay-missing-heartbeat-detection show
    285
    console> system synchronized-security suppress-missing-heartbeat-to-central show
    120
    console>

  • Hello LHerzog,

    Thank you for the update, I can see your case is now with GES for further investigation.

    Regards,


     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support VideosProduct Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.
  • Support tech found this which is what I already posted above: (modern) standby causes those issues.
    Asking me to disable standby, which is not a desired workaround.

    We've already had a case open in 2021 with SFOS 18.0.5: 04121743  Endpoints reporting "at risk"to XG firewall but Sophos Central showing no issues.

    Support tech wrote there: "I believe from the logs you have provided, I do see that it went from a green connected state to a red missing heartbeat state multiple times back to back. This occurs when heartbeat traffic from the device is no longer sent to the firewall, typically after the machine enters sleep/wakes from sleep or gets disconnected.

    I believe that the endpoints were having issues connecting or enters sleep/wakes from sleep very quickly that the updates were not relayed to central in time therefore it didn't show up there."

    2023-03-28T13:58:53.665Z [ 5156: 5160] I Received Screen Off notification: Endpoint entering Modern Standby


    2023-03-28T13:58:58.229Z [ 4800: 6088] D Failed to connect: system:10065.
    2023-03-28T13:58:58.229Z [ 4800: 6088] D + Connection::OnConnectError()
    2023-03-28T13:58:58.229Z [ 4800: 6088] A Connection failed.

  • After several test we can confirm the statement from Tech Support, that Sophos Heartbeat is not compatible with Modern Standby feature of Windows and / or modern Computers.

    So as customer you have 2 choices to get this worked around:

    1. Disable modern Standby on the OS of all your client computers:

        cmd
        powercfg /a
        should report s0 is enabled. s0 is modern standby
    
        regedit
        [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Power]
        New > DWORD (32-bit) Value.
        PlatformAoAcOverride with the value 0
        reboot
        
        cmd
        powercfg /a
        should report s3 is enabled. s3 is classic standby

    2. disable the mail notification about missing heartbeat in sophos central, and ignore all the alerts that appear on the dashboard through out the day and continue to use the modern standby on the client computers.

    Both workarounds are bad ones.

    Sophos should get Heartbeat be compatible with current computers and operating systems.

  • final statement, I received from support.

    feedback from our Product management team and they have stated- "we do have a roadmap item to improve modern standby behavior, but at this time there is no commitment as to when it would be delivered."

    so it may, hopefully, get better sometimes.

Reply
  • final statement, I received from support.

    feedback from our Product management team and they have stated- "we do have a roadmap item to improve modern standby behavior, but at this time there is no commitment as to when it would be delivered."

    so it may, hopefully, get better sometimes.

Children
  • our logs on firewall are flooded with events when the computers are in modern standby - hard to find anything else.

    livelog on the firewall is flooded with login and logout events

    2023-06-14 11:26:12,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,AD,User user@domain.local of group ADGROUP logged in successfully to Firewall through AD authentication mechanism from 172.16.xxx.xxx ,17701
    2023-06-14 11:26:12,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,N/A,User user@domain.local was logged out of firewall,17703
    2023-06-14 11:20:54,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,AD,User user@domain.local of group ADGROUP logged in successfully to Firewall through AD authentication mechanism from 172.16.xxx.xxx ,17701
    2023-06-14 11:20:53,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,N/A,User user@domain.local was logged out of firewall,17703
    2023-06-14 11:14:54,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,AD,User user@domain.local of group ADGROUP logged in successfully to Firewall through AD authentication mechanism from 172.16.xxx.xxx ,17701
    2023-06-14 11:14:53,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,N/A,User user@domain.local was logged out of firewall,17703
    2023-06-14 11:09:23,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,AD,User user@domain.local of group ADGROUP logged in successfully to Firewall through AD authentication mechanism from 172.16.xxx.xxx ,17701
    2023-06-14 11:09:22,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,N/A,User user@domain.local was logged out of firewall,17703
    2023-06-14 11:03:22,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,AD,User user@domain.local of group ADGROUP logged in successfully to Firewall through AD authentication mechanism from 172.16.xxx.xxx ,17701
    2023-06-14 11:03:21,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,N/A,User user@domain.local was logged out of firewall,17703
    2023-06-14 10:57:21,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,AD,User user@domain.local of group ADGROUP logged in successfully to Firewall through AD authentication mechanism from 172.16.xxx.xxx ,17701
    2023-06-14 10:57:20,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,N/A,User user@domain.local was logged out of firewall,17703
    2023-06-14 10:51:20,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,AD,User user@domain.local of group ADGROUP logged in successfully to Firewall through AD authentication mechanism from 172.16.xxx.xxx ,17701
    2023-06-14 10:51:20,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,N/A,User user@domain.local was logged out of firewall,17703
    2023-06-14 10:44:44,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,AD,User user@domain.local of group ADGROUP logged in successfully to Firewall through AD authentication mechanism from 172.16.xxx.xxx ,17701
    2023-06-14 10:44:01,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,N/A,User user@domain.local was logged out of firewall,17703
    2023-06-14 10:38:41,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,AD,User user@domain.local of group ADGROUP logged in successfully to Firewall through AD authentication mechanism from 172.16.xxx.xxx ,17701
    2023-06-14 10:38:40,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,N/A,User user@domain.local was logged out of firewall,17703
    2023-06-14 10:33:22,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,AD,User user@domain.local of group ADGROUP logged in successfully to Firewall through AD authentication mechanism from 172.16.xxx.xxx ,17701
    2023-06-14 10:33:22,Firewall Authentication,Successful,user@domain.local,172.16.xxx.xxx ,Heartbeat,N/A,User user@domain.local was logged out of firewall,17703
    

    they are generated when the computer is in modern standby.

    and heartbeatd.log is flooded with these events

    [2023-06-14 08:33:41.750Z] WARN HBSessionHandler.cpp[9271]:140 findPinnedEndpointIdentity - session from endpoint rejected, because the endpoint ID already exists: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    [2023-06-14 08:33:41.750Z] WARN HBSession.cpp[9271]:387 bufferConnectedEvent - Pinning for a session failed
    [2023-06-14 08:33:42.900Z] INFO HBSessionHandler.cpp[9271]:125 removeDirtySessions - Number of sessions: 193
    [2023-06-14 08:33:42.993Z] WARN HBSessionHandler.cpp[9271]:140 findPinnedEndpointIdentity - session from endpoint rejected, because the endpoint ID already exists: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    [2023-06-14 08:33:42.993Z] WARN HBSession.cpp[9271]:387 bufferConnectedEvent - Pinning for a session failed
    [2023-06-14 08:33:44.128Z] INFO HBSessionHandler.cpp[9271]:125 removeDirtySessions - Number of sessions: 193
    [2023-06-14 08:33:44.226Z] WARN HBSessionHandler.cpp[9271]:140 findPinnedEndpointIdentity - session from endpoint rejected, because the endpoint ID already exists: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    [2023-06-14 08:33:44.226Z] WARN HBSession.cpp[9271]:387 bufferConnectedEvent - Pinning for a session failed
    [2023-06-14 08:33:45.310Z] INFO HBSessionHandler.cpp[9271]:125 removeDirtySessions - Number of sessions: 193
    [2023-06-14 08:33:45.399Z] WARN HBSessionHandler.cpp[9271]:140 findPinnedEndpointIdentity - session from endpoint rejected, because the endpoint ID already exists: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    [2023-06-14 08:33:45.399Z] WARN HBSession.cpp[9271]:387 bufferConnectedEvent - Pinning for a session failed
    [2023-06-14 08:33:46.536Z] INFO HBSessionHandler.cpp[9271]:125 removeDirtySessions - Number of sessions: 193
    [2023-06-14 08:33:46.650Z] WARN HBSessionHandler.cpp[9271]:140 findPinnedEndpointIdentity - session from endpoint rejected, because the endpoint ID already exists: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    [2023-06-14 08:33:46.650Z] WARN HBSession.cpp[9271]:387 bufferConnectedEvent - Pinning for a session failed
    [2023-06-14 08:33:47.790Z] INFO HBSessionHandler.cpp[9271]:125 removeDirtySessions - Number of sessions: 193
    [2023-06-14 08:33:47.885Z] WARN HBSessionHandler.cpp[9271]:140 findPinnedEndpointIdentity - session from endpoint rejected, because the endpoint ID already exists: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    [2023-06-14 08:33:47.885Z] WARN HBSession.cpp[9271]:387 bufferConnectedEvent - Pinning for a session failed
    [2023-06-14 08:33:49.020Z] INFO HBSessionHandler.cpp[9271]:125 removeDirtySessions - Number of sessions: 193
    [2023-06-14 08:33:49.132Z] WARN HBSessionHandler.cpp[9271]:140 findPinnedEndpointIdentity - session from endpoint rejected, because the endpoint ID already exists: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    [2023-06-14 08:33:49.132Z] WARN HBSession.cpp[9271]:387 bufferConnectedEvent - Pinning for a session failed
    [2023-06-14 08:33:50.284Z] INFO HBSessionHandler.cpp[9271]:125 removeDirtySessions - Number of sessions: 193
    [2023-06-14 08:33:50.352Z] WARN HBSessionHandler.cpp[9271]:140 findPinnedEndpointIdentity - session from endpoint rejected, because the endpoint ID already exists: xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    [2023-06-14 08:33:50.352Z] WARN HBSession.cpp[9271]:387 bufferConnectedEvent - Pinning for a session failed
    

    a fun fact is, that the "computer not sending heartbeat" events generated in central are not generated for that machine.

    more and more hardware arrives, that is on modern standby by default. more logs generated.

  • Checking some of the development tickets we have opened related to this issue, I do see that work is planned to improve this behaviour so that the endpoint will inform the firewall when it is entering a sleep state. 

    If others also encounter this issue, I'd suggest opening a support case to inquire about the following ID: WINEP-30133

    Kushal Lakhan
    Team Lead, Global Community Support
    Connect with Sophos Support, get alerted, and be informed.
    If a post solves your question, please use the "Verify Answer" button.
    The New Home of Sophos Support Videos!  Visit Sophos Techvids