Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Thread Info
  • State Not Answered
  • +1 person also asked this people also asked this
  • Locked Locked
  • Replies 48 replies
  • Subscribers 45 subscribers
  • Views 6899 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Strange issue with Security Heartbeat

Gerhard Seemüller

Hello,

we have noticed a strange issue with Security Heartbeat. Devices often only gain access to the network several minutes after booting. The Heartbeat.log on the endpoint says that the connection initially failed. The heartbeatd.log on the firewall does not contain any recent entries.

Heartbeat.log

2023-02-23T09:36:02.846Z [17344: 8016] A Connection failed.
2023-02-23T09:40:54.412Z [17344: 8016] A Connection succeeded.
2023-02-23T09:40:54.413Z [17344: 8016] A Connected to '81d5633d-0d85-4824-98e4-858c87c7a273' at IP address 52.5.76.173 on port 8347
2023-02-23T09:40:54.413Z [17344: 8016] A Sending network status
2023-02-23T09:40:54.413Z [17344: 8016] A The network status has changed, the Firewall may disconnect.
2023-02-23T09:40:54.415Z [17344: 8016] A Received request to enable enhanced application control
2023-02-23T09:40:54.415Z [17344: 8016] A Sending endpoint state list request
2023-02-23T09:40:54.416Z [17344: 8016] A Sending login status.
2023-02-23T09:40:54.416Z [17344: 8016] A User: USERNAME
2023-02-23T09:40:54.416Z [17344: 8016] A Sending health status: admin=1 health=1 service=1 threat=1 threatService=1
2023-02-23T09:40:54.417Z [17344: 8016] A Received response to endpoint state list request, size: 1
2023-02-23T09:42:00.950Z [17344: 8016] A Received request to disable enhanced application control for C:\program files (x86)\microsoft\edge\application\msedge.exe

heartbeatd.log (there are no newer entries)

[2021-11-30 15:00:20.057] INFO HBSession.cpp[6743]:502 logNewSession - New Session: [172.16.12.74]:8387 connected
[2021-11-30 15:00:20.103] INFO EndpointStorage.cpp[6743]:114 endpoint_connectivity_cb - Connectivity changed for <c25ece7d-a04e-4005-820c-b1a12624518e>: <5> -> <1>
[2021-11-30 15:00:20.103] INFO ModuleSacFirst.cpp[6743]:95 sendEacMessage - send EacSwitchRequest to endpoint (IP=172.16.12.74)
[2021-11-30 15:00:20.106] INFO EpStateListBroker.cpp[6743]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: c25ece7d-a04e-4005-820c-b1a12624518e(172.16.12.74)
[2021-11-30 15:00:23.823] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:00:29.925] INFO ModuleStatus.cpp[6743]:138 processMessageStatus - Status request received from endpoint: c25ece7d-a04e-4005-820c-b1a12624518e (172.16.12.74) health: 1
[2021-11-30 15:01:00.359] INFO SacProcessor.cpp[6743]:64 discardApp - Sent switchOffConnectionInfo request to endpoint: <c25ece7d-a04e-4005-820c-b1a12624518e>, Application path :C:\134program files (x86)\134microsoft\134edge\134application\134msedge.exe
[2021-11-30 15:01:24.061] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:01:27.699] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:07:22.260] INFO GarnerEventReader.cpp[6743]:129 acceptConnectionHandler - Garner plugin connected. Ready to receive garner events.
[2021-11-30 15:09:04.494] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:13:16.599] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:13:44.482] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:15:17.622] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:15:24.041] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:16:27.738] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:21:25.037] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:26:04.897] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:28:16.624] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:30:17.652] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:30:24.252] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:31:27.788] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:33:45.548] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:38:25.333] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:43:16.648] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:45:17.685] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:45:24.498] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:46:06.073] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:46:27.828] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:50:45.751] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:52:23.285] INFO GarnerEventReader.cpp[6743]:129 acceptConnectionHandler - Garner plugin connected. Ready to receive garner events.
[2021-11-30 15:58:16.722] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:58:26.637] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:00:17.719] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:00:24.741] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:01:27.860] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:03:06.144] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:10:22.523] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:10:47.203] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:13:16.701] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:15:17.752] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:15:24.960] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:15:26.535] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:16:27.904] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:18:41.526] INFO SacProcessor.cpp[6743]:64 discardApp - Sent switchOffConnectionInfo request to endpoint: <e97fa787-de12-4693-86dc-6fdbf77e051c>, Application path :C:\134program files (x86)\134microsoft\134edgeupdate\134microsoftedgeupdate.exe
[2021-11-30 16:20:54.552] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:23:07.807] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:25:42.408] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:27:46.955] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:28:16.725] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:29:45.841] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:30:17.778] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:30:25.179] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:31:27.940] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:32:08.488] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:34:20.903] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:35:28.345] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:37:25.183] INFO GarnerEventReader.cpp[6743]:129 acceptConnectionHandler - Garner plugin connected. Ready to receive garner events.
[2021-11-30 16:40:07.373] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:41:06.825] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:43:16.741] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:45:17.808] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:45:25.411] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:46:27.977] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:47:48.809] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:52:27.788] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:58:16.761] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:00:06.391] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:00:09.333] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:00:17.846] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:00:25.617] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:00:44.444] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:01:14.856] INFO EndpointStorage.cpp[6743]:114 endpoint_connectivity_cb - Connectivity changed for <56a453ce-bbef-4fab-b721-d8435c1ef48b>: <1> -> <3>
[2021-11-30 17:01:44.448] INFO EndpointStorage.cpp[6743]:114 endpoint_connectivity_cb - Connectivity changed for <c25ece7d-a04e-4005-820c-b1a12624518e>: <1> -> <3>
[2021-11-30 17:04:48.263] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System[2021-11-30 17:04:48.263] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System

How to fix this problem?

Best regards

Gerhard



This thread was automatically locked due to age.
  • 0 in reply to Gerhard Seemüller

    This is likely a Endpoint issue. This means: The Endpoint log is important.

    TLS Error could be related to time. If you reboot the device, could you check in this timeframe the NTP Time of this device? Is this correct? 

    What we need to check, what is going on: 

    Go to the firewall, Advanced Log. See, what kind of IP you want to reboot/reproduce the issue. Then do: #tcpdump -ni any host IP_of_Device and port 8347 -b -w /tmp/hb.pcap 

    Then reproduce the issue, while this dump is running. If the device is connected and you could reproduce the delay, exit the dump with CTRL+C and download the file via SCP: https://support.sophos.com/support/s/article/KB-000035842?language=en_US 

    __________________________________________________________________________________________________________________

  • 0 in reply to LuCar Toni

    The system time was successfully updated to the current time from the time the computer was shut down.

    I was just about to look at the firewall log at this point. But I can't really scroll back that far and unfortunately you can't specify a specific time range as a filter criterion. I've only managed to reproduce this problem once, because I rarely switch between locations.

  • 0 in reply to Gerhard Seemüller

    Hello  ,

    Were you able to solve the problem somehow in the meantime?
    We have exactly the same problem. When Heartbeats is enabled, it takes a few minutes to establish a network connection after booting or switching from LAN to WLAN. And my log files look pretty much the same as yours.

  • 0 in reply to Schmidmeister Tobias

    Unfortunately, no. We still have the problem with almost every network change. Especially annoying for users of sometimes not very reliable USB-C docks. They are sometimes blocked for several minutes in the middle of work or even during phone calls. Looks like we need to disable the security heartbeat.

  • 0 in reply to Gerhard Seemüller

    changing network status causes Heartbeat to re-establish and it takes 1-3 minutes. We have never experienced heartbeat any different from that. And yes, with users pulling their machines off dockings, travel through the building, lose wifi connection etc, needs some patience sometimes. Also heartbeat has issues with machines that have modern standby enabled. When they are in modern standby (s0) they periodically enable network to pick up new mails and notifications, then go to sleep again. This behaviour causes the firewall to report missing heartbeats of the device.

    Also have in mind, that network threat protection upates on the endpoint happen when they come, unless you have specified update schedule policy in central. ntp updates cause heartbeat to be re-established - during user work = 1-3 minutes without heartbeat.

  • 0 in reply to LHerzog

    Unfortunately, the user cannot see why network access is blocked or the phone call is interrupted just because you disconnect from the dock, unless he opens the browser. That causes a lot of frustration. We simply wish for a much more reliable environment again.

    The standby mode of all devices is deactivated.

    The reference to Network Threat Protection updates is interesting. Without disrupting the user, these updates would probably only be possible immediately after the devices booted up. However, network access is also required at this point, so I guess that's not a solution either.

  • 0 in reply to Gerhard Seemüller

    if you have heartbeat requiring firewall rules, keep in mind that you must (!) allow all sophos central communication through rules that do not require heartbeat. if you have that, all updates work fine without or with red heartbeat.

    Also, if you have MDR or live response licensed, you'll want to do remote forensics on an infected device and it needs to communicate to sophos central (only) for that.

    And yes, it's totally untransparent for the user, what is happening. They only call IT and say, it is not working to access this and that.

  • 0 in reply to LHerzog

    Access is currently allowed to (http and https):

    *.cloudfront.net
    *.sophos.com
    az416426.vo.msecnd.net
    crl.globalsign.com
    crl.globalsign.net
    crl3.digicert.com
    crl4.digicert.com
    dc.services.visualstudio.com
    ocsp.digicert.com
    ocsp.globalsign.com
    ocsp2.globalsign.com

    Maybe an address is missing here?

  • 0 in reply to Gerhard Seemüller

    yes. many. and the heartbeat IP on it's special port?

    there have always been issues with some wildcard FQDN, even from Sophos. Some have been fixed, some came back, some probably still there.

    you have *.sophos.com allowed - that should cover most of it. In fact, over the years, we needed to allow some more specific FQDN.

    This is our Central host set:

      *.fw.prod.hydra.sophos.com
    *.mcs-push-server-eu-central-1.prod.hydra.sophos.com
    *.mcs-push-server-eu-west-1.prod.hydra.sophos.com
    *.mcs-push-server-us-east-2.prod.hydra.sophos.com
    *.mcs-push-server-us-west-2.prod.hydra.sophos.com
    *.mcs-push-server.stn100hnd.ctr.sophos.com
    *.mcs-push-server.stn100syd.ctr.sophos.com
    *.mcs-push-server.stn100yul.ctr.sophos.com
    *.sophos.com
    *.sophosupd.com
    *.sophosupd.net
    *.sophosxl.net
    4.sophosxl.net
    api-cloudstation-eu-central-1.prod.hydra.sophos.com
    central.sophos.com
    cloud.sophos.com
    crl.globalsign.com
    crl.globalsign.net
    crl3.digicert.com
    crl4.digicert.com
    d1.sophosupd.com
    d1.sophosupd.net
    d2.sophosupd.com
    d2.sophosupd.net
    d3.sophosupd.com
    d3.sophosupd.net
    dci.sophosupd.com
    downloads.sophos.com
    dzr-mcs-amzn-eu-west-1-9af7.upe.p.hmr.sophos.com
    dzr-mcs-amzn-us-west-2-fa88.upe.p.hmr.sophos.com
    id.sophos.com
    kinesis.us-west-2.amazonaws.com
    live-terminal-eu-central-1.prod.hydra.sophos.com
    live-terminal-eu-west-1.prod.hydra.sophos.com
    live-terminal-us-east-2.prod.hydra.sophos.com
    live-terminal-us-west-2.prod.hydra.sophos.com
    live-terminal.stn100hnd.ctr.sophos.com
    live-terminal.stn100syd.ctr.sophos.com
    live-terminal.stn100yul.ctr.sophos.com
    mcs-cloudstation-eu-central-1.prod.hydra.sophos.com
    mcs-cloudstation-eu-west-1.prod.hydra.sophos.com
    mcs-cloudstation-us-east-2.prod.hydra.sophos.com
    mcs-cloudstation-us-west-2.prod.hydra.sophos.com
    mcs.stn100hnd.ctr.sophos.com
    mcs.stn100syd.ctr.sophos.com
    mcs.stn100yul.ctr.sophos.com
    mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com
    mcs2-cloudstation-eu-west-1.prod.hydra.sophos.com
    mcs2-cloudstation-us-east-2.prod.hydra.sophos.com
    mcs2-cloudstation-us-west-2.prod.hydra.sophos.com
    mcs2.stn100hnd.ctr.sophos.com
    mcs2.stn100syd.ctr.sophos.com
    mcs2.stn100yul.ctr.sophos.com
    ocsp.digicert.com
    ocsp.globalsign.com
    ocsp2.globalsign.com
    prod.endpointintel.darkbytes.io
    samples.sophosxl.net
    sdds3.sophosupd.com
    sdds3.sophosupd.net
    sdu-feedback.sophos.com
    sus.sophosupd.com
    t1.sophosupd.com
    tf-edr-message-upload-eu-central-1-prod-bucket.s3.amazonaws.
    tf-edr-message-upload-eu-west-1-prod-bucket.s3.amazonaws.com
    tf-edr-message-upload-us-east-2-prod-bucket.s3.amazonaws.com
    tf-edr-message-upload-us-west-2-prod-bucket.s3.amazonaws.com
    utm-cloudstation-eu-central-1.prod.hydra.sophos.com

  • 0 in reply to LHerzog

    You do not need to allow the HB IP + Heartbeat Port. 
    This connection will never leave the firewall on WAN, as the firewall will intercept this connection. 

    In SFOS you can also reduce this list by using Wildcard *.sophos.com (and the other domains).

    This will be integrated in the future for other Sophos products per default. 

    __________________________________________________________________________________________________________________

Unfiltered HTML