Important note about SSL VPN compatibility for 20.0 MR1 with EoL SFOS versions and UTM9 OS. Learn more in the release notes.

Thread Info
  • State Not Answered
  • +1 person also asked this people also asked this
  • Locked Locked
  • Replies 48 replies
  • Subscribers 45 subscribers
  • Views 6899 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Strange issue with Security Heartbeat

Gerhard Seemüller

Hello,

we have noticed a strange issue with Security Heartbeat. Devices often only gain access to the network several minutes after booting. The Heartbeat.log on the endpoint says that the connection initially failed. The heartbeatd.log on the firewall does not contain any recent entries.

Heartbeat.log

2023-02-23T09:36:02.846Z [17344: 8016] A Connection failed.
2023-02-23T09:40:54.412Z [17344: 8016] A Connection succeeded.
2023-02-23T09:40:54.413Z [17344: 8016] A Connected to '81d5633d-0d85-4824-98e4-858c87c7a273' at IP address 52.5.76.173 on port 8347
2023-02-23T09:40:54.413Z [17344: 8016] A Sending network status
2023-02-23T09:40:54.413Z [17344: 8016] A The network status has changed, the Firewall may disconnect.
2023-02-23T09:40:54.415Z [17344: 8016] A Received request to enable enhanced application control
2023-02-23T09:40:54.415Z [17344: 8016] A Sending endpoint state list request
2023-02-23T09:40:54.416Z [17344: 8016] A Sending login status.
2023-02-23T09:40:54.416Z [17344: 8016] A User: USERNAME
2023-02-23T09:40:54.416Z [17344: 8016] A Sending health status: admin=1 health=1 service=1 threat=1 threatService=1
2023-02-23T09:40:54.417Z [17344: 8016] A Received response to endpoint state list request, size: 1
2023-02-23T09:42:00.950Z [17344: 8016] A Received request to disable enhanced application control for C:\program files (x86)\microsoft\edge\application\msedge.exe

heartbeatd.log (there are no newer entries)

[2021-11-30 15:00:20.057] INFO HBSession.cpp[6743]:502 logNewSession - New Session: [172.16.12.74]:8387 connected
[2021-11-30 15:00:20.103] INFO EndpointStorage.cpp[6743]:114 endpoint_connectivity_cb - Connectivity changed for <c25ece7d-a04e-4005-820c-b1a12624518e>: <5> -> <1>
[2021-11-30 15:00:20.103] INFO ModuleSacFirst.cpp[6743]:95 sendEacMessage - send EacSwitchRequest to endpoint (IP=172.16.12.74)
[2021-11-30 15:00:20.106] INFO EpStateListBroker.cpp[6743]:56 markEndpointForUpdates - Endpoint marked for receiving Stonewall updates: c25ece7d-a04e-4005-820c-b1a12624518e(172.16.12.74)
[2021-11-30 15:00:23.823] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:00:29.925] INFO ModuleStatus.cpp[6743]:138 processMessageStatus - Status request received from endpoint: c25ece7d-a04e-4005-820c-b1a12624518e (172.16.12.74) health: 1
[2021-11-30 15:01:00.359] INFO SacProcessor.cpp[6743]:64 discardApp - Sent switchOffConnectionInfo request to endpoint: <c25ece7d-a04e-4005-820c-b1a12624518e>, Application path :C:\134program files (x86)\134microsoft\134edge\134application\134msedge.exe
[2021-11-30 15:01:24.061] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:01:27.699] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:07:22.260] INFO GarnerEventReader.cpp[6743]:129 acceptConnectionHandler - Garner plugin connected. Ready to receive garner events.
[2021-11-30 15:09:04.494] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:13:16.599] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:13:44.482] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:15:17.622] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:15:24.041] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:16:27.738] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:21:25.037] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:26:04.897] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:28:16.624] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:30:17.652] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:30:24.252] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:31:27.788] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:33:45.548] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:38:25.333] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:43:16.648] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:45:17.685] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:45:24.498] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:46:06.073] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:46:27.828] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:50:45.751] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:52:23.285] INFO GarnerEventReader.cpp[6743]:129 acceptConnectionHandler - Garner plugin connected. Ready to receive garner events.
[2021-11-30 15:58:16.722] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 15:58:26.637] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:00:17.719] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:00:24.741] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:01:27.860] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:03:06.144] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:10:22.523] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:10:47.203] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:13:16.701] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:15:17.752] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:15:24.960] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:15:26.535] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:16:27.904] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:18:41.526] INFO SacProcessor.cpp[6743]:64 discardApp - Sent switchOffConnectionInfo request to endpoint: <e97fa787-de12-4693-86dc-6fdbf77e051c>, Application path :C:\134program files (x86)\134microsoft\134edgeupdate\134microsoftedgeupdate.exe
[2021-11-30 16:20:54.552] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:23:07.807] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:25:42.408] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:27:46.955] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:28:16.725] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:29:45.841] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:30:17.778] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:30:25.179] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:31:27.940] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:32:08.488] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:34:20.903] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:35:28.345] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:37:25.183] INFO GarnerEventReader.cpp[6743]:129 acceptConnectionHandler - Garner plugin connected. Ready to receive garner events.
[2021-11-30 16:40:07.373] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:41:06.825] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:43:16.741] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:45:17.808] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:45:25.411] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:46:27.977] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:47:48.809] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:52:27.788] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 16:58:16.761] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:00:06.391] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:00:09.333] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:00:17.846] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:00:25.617] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:00:44.444] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System
[2021-11-30 17:01:14.856] INFO EndpointStorage.cpp[6743]:114 endpoint_connectivity_cb - Connectivity changed for <56a453ce-bbef-4fab-b721-d8435c1ef48b>: <1> -> <3>
[2021-11-30 17:01:44.448] INFO EndpointStorage.cpp[6743]:114 endpoint_connectivity_cb - Connectivity changed for <c25ece7d-a04e-4005-820c-b1a12624518e>: <1> -> <3>
[2021-11-30 17:04:48.263] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System[2021-11-30 17:04:48.263] WARN Path.cpp[6743]:68 getExecutableFilename - Parsing executable filename failed. Falling back to full path: System

How to fix this problem?

Best regards

Gerhard



This thread was automatically locked due to age.
Parents
  • 0

    The log in the GUI is also not very helpful. There it is only shown that the computer had a green security heartbeat after 5 minutes. Of course, the computer itself has a lot of errors in the log, because network access was denied.

  • 0 in reply to Gerhard Seemüller

    Could you check the Endpoint Log? support.sophos.com/.../KB-000038787

    __________________________________________________________________________________________________________________

  • 0 in reply to Gerhard Seemüller

    changing network status causes Heartbeat to re-establish and it takes 1-3 minutes. We have never experienced heartbeat any different from that. And yes, with users pulling their machines off dockings, travel through the building, lose wifi connection etc, needs some patience sometimes. Also heartbeat has issues with machines that have modern standby enabled. When they are in modern standby (s0) they periodically enable network to pick up new mails and notifications, then go to sleep again. This behaviour causes the firewall to report missing heartbeats of the device.

    Also have in mind, that network threat protection upates on the endpoint happen when they come, unless you have specified update schedule policy in central. ntp updates cause heartbeat to be re-established - during user work = 1-3 minutes without heartbeat.

  • 0 in reply to LHerzog

    Unfortunately, the user cannot see why network access is blocked or the phone call is interrupted just because you disconnect from the dock, unless he opens the browser. That causes a lot of frustration. We simply wish for a much more reliable environment again.

    The standby mode of all devices is deactivated.

    The reference to Network Threat Protection updates is interesting. Without disrupting the user, these updates would probably only be possible immediately after the devices booted up. However, network access is also required at this point, so I guess that's not a solution either.

  • 0 in reply to Gerhard Seemüller

    if you have heartbeat requiring firewall rules, keep in mind that you must (!) allow all sophos central communication through rules that do not require heartbeat. if you have that, all updates work fine without or with red heartbeat.

    Also, if you have MDR or live response licensed, you'll want to do remote forensics on an infected device and it needs to communicate to sophos central (only) for that.

    And yes, it's totally untransparent for the user, what is happening. They only call IT and say, it is not working to access this and that.

  • 0 in reply to LHerzog

    Access is currently allowed to (http and https):

    *.cloudfront.net
    *.sophos.com
    az416426.vo.msecnd.net
    crl.globalsign.com
    crl.globalsign.net
    crl3.digicert.com
    crl4.digicert.com
    dc.services.visualstudio.com
    ocsp.digicert.com
    ocsp.globalsign.com
    ocsp2.globalsign.com

    Maybe an address is missing here?

  • 0 in reply to Gerhard Seemüller

    yes. many. and the heartbeat IP on it's special port?

    there have always been issues with some wildcard FQDN, even from Sophos. Some have been fixed, some came back, some probably still there.

    you have *.sophos.com allowed - that should cover most of it. In fact, over the years, we needed to allow some more specific FQDN.

    This is our Central host set:

      *.fw.prod.hydra.sophos.com
    *.mcs-push-server-eu-central-1.prod.hydra.sophos.com
    *.mcs-push-server-eu-west-1.prod.hydra.sophos.com
    *.mcs-push-server-us-east-2.prod.hydra.sophos.com
    *.mcs-push-server-us-west-2.prod.hydra.sophos.com
    *.mcs-push-server.stn100hnd.ctr.sophos.com
    *.mcs-push-server.stn100syd.ctr.sophos.com
    *.mcs-push-server.stn100yul.ctr.sophos.com
    *.sophos.com
    *.sophosupd.com
    *.sophosupd.net
    *.sophosxl.net
    4.sophosxl.net
    api-cloudstation-eu-central-1.prod.hydra.sophos.com
    central.sophos.com
    cloud.sophos.com
    crl.globalsign.com
    crl.globalsign.net
    crl3.digicert.com
    crl4.digicert.com
    d1.sophosupd.com
    d1.sophosupd.net
    d2.sophosupd.com
    d2.sophosupd.net
    d3.sophosupd.com
    d3.sophosupd.net
    dci.sophosupd.com
    downloads.sophos.com
    dzr-mcs-amzn-eu-west-1-9af7.upe.p.hmr.sophos.com
    dzr-mcs-amzn-us-west-2-fa88.upe.p.hmr.sophos.com
    id.sophos.com
    kinesis.us-west-2.amazonaws.com
    live-terminal-eu-central-1.prod.hydra.sophos.com
    live-terminal-eu-west-1.prod.hydra.sophos.com
    live-terminal-us-east-2.prod.hydra.sophos.com
    live-terminal-us-west-2.prod.hydra.sophos.com
    live-terminal.stn100hnd.ctr.sophos.com
    live-terminal.stn100syd.ctr.sophos.com
    live-terminal.stn100yul.ctr.sophos.com
    mcs-cloudstation-eu-central-1.prod.hydra.sophos.com
    mcs-cloudstation-eu-west-1.prod.hydra.sophos.com
    mcs-cloudstation-us-east-2.prod.hydra.sophos.com
    mcs-cloudstation-us-west-2.prod.hydra.sophos.com
    mcs.stn100hnd.ctr.sophos.com
    mcs.stn100syd.ctr.sophos.com
    mcs.stn100yul.ctr.sophos.com
    mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com
    mcs2-cloudstation-eu-west-1.prod.hydra.sophos.com
    mcs2-cloudstation-us-east-2.prod.hydra.sophos.com
    mcs2-cloudstation-us-west-2.prod.hydra.sophos.com
    mcs2.stn100hnd.ctr.sophos.com
    mcs2.stn100syd.ctr.sophos.com
    mcs2.stn100yul.ctr.sophos.com
    ocsp.digicert.com
    ocsp.globalsign.com
    ocsp2.globalsign.com
    prod.endpointintel.darkbytes.io
    samples.sophosxl.net
    sdds3.sophosupd.com
    sdds3.sophosupd.net
    sdu-feedback.sophos.com
    sus.sophosupd.com
    t1.sophosupd.com
    tf-edr-message-upload-eu-central-1-prod-bucket.s3.amazonaws.
    tf-edr-message-upload-eu-west-1-prod-bucket.s3.amazonaws.com
    tf-edr-message-upload-us-east-2-prod-bucket.s3.amazonaws.com
    tf-edr-message-upload-us-west-2-prod-bucket.s3.amazonaws.com
    utm-cloudstation-eu-central-1.prod.hydra.sophos.com

  • 0 in reply to LHerzog

    You do not need to allow the HB IP + Heartbeat Port. 
    This connection will never leave the firewall on WAN, as the firewall will intercept this connection. 

    In SFOS you can also reduce this list by using Wildcard *.sophos.com (and the other domains).

    This will be integrated in the future for other Sophos products per default. 

    __________________________________________________________________________________________________________________

  • 0 in reply to LuCar Toni
    LuCar Toni said:
    In SFOS you can also reduce this list by using Wildcard *.sophos.com (and the other domains).

    at least those never worked for us, without adding them exactly like this. They are the most commonly used hosts.

        *.mcs-push-server-eu-central-1.prod.hydra.sophos.com
        *.mcs-push-server-eu-west-1.prod.hydra.sophos.com
        *.mcs-push-server-us-east-2.prod.hydra.sophos.com
        *.mcs-push-server-us-west-2.prod.hydra.sophos.com

  • 0 in reply to LHerzog

    Shouldn't the heartbeat ip be intercepted by the firewall anyway? I will use the server names directly as far as possible. Unfortunately, as you said, wildcards cause problems from time to time. Thank you for your list. I used the following document as a guide: Domains and Ports to allow - Intercept X Advanced with XDR.

  • 0 in reply to LHerzog

    So essentially you can do *.sophos.com and if the client uses a DNS lookup, it will fill the object. 

    This is what you should see in the firewall in the Network Object. 

    __________________________________________________________________________________________________________________

  • 0 in reply to LuCar Toni

    Do you really have to enable the following very far-reaching Internet access or is it only required for administration via Sophos Central?

    az416426.vo.msecnd.net
    dc.services.visualstudio.com
    *.cloudfront.net

    Domains and ports to allow - Sophos Central Admin domains

Reply Children
No Data
Unfiltered HTML