Sophos Firewall: Heartbeat stops showing any endpoint clients on GUI

Disclaimer: This information is provided as-is for the benefit of the Community. Please contact Sophos Professional Services if you require assistance with your specific environment.

______________________________________________________________________________________________________________________________________

Overview:

Overview:

This article describes the troubleshooting steps & possible solutions when the heartbeat component stops showing any endpoint clients on Sophos Firewall GUI even though the Sophos Central dashboard shows all the information correctly.

Scenario:

The administrator suddenly cannot see any/all endpoint clients connected to the Sophos Firewall GUI under the Heartbeat section. Although, in Sophos Central, all the devices are visible as Green and connected.

What to do:

Confirm if the valid license of heartbeat is associated with central.
Check the status of the heartbeat service using the below command:

service -S | grep heartbeat

Service should be in a running state.

XG450_WP02_SFOS 18.5.1 MR-1-Build326# service -S | grep heart
fwcm-heartbeatd      RUNNING
heartbeat            RUNNING
XG450_WP02_SFOS 18.5.1 MR-1-Build326#

Check the status of registration on central using the below command. It should be registered as below:

XG450_WP02_SFOS 18.5.1 MR-1-Build326# /bin/central-register --status
This SFOS instance is currently registered with Sophos Central

Note: The output will also show you more information like; Access_token, Device_uuid, Pic_uri, and refresh_token.

Check the opcode using the below command:

XG450_WP02_SFOS 18.5.1 MR-1-Build326# opcode -ds nosync hbtrust_synchronize
200 OK
{ "statusmessage": "Synchronization with Sophos Central successful", "status": 200 }
XG450_WP02_SFOS 18.5.1 MR-1-Build326#

Check the heartbeat related logs from /log directory.

XG450_WP02_SFOS 18.5.1 MR-1-Build326# tail -f /log/heartbeat.log
[2021-11-25 11:21:08.908] WARN HBSession.cpp[31780]:341 bufferDisconnectEvent - Incoming connection from 10.44.0.204 failed. SSL error: SSL routines:ssl3_read_bytes sslv3 alert certificate expired
[2021-11-25 11:21:31.444] INFO HBSessionHandler.cpp[31780]:108 removeDirtySessions - Number of sessions: 0
[2021-11-25 11:21:31.450] WARN HBSession.cpp[31780]:341 bufferDisconnectEvent - Incoming connection from 10.44.1.241 failed. SSL error: SSL routines:ssl3_read_bytes sslv3 alert certificate expired

XG450_WP02_SFOS 18.5.1 MR-1-Build326# tail -f /log/hbtrust.log
2021-11-25 11:05:39 INFO Sync.pm[27999]:175 SFOS::HBtrust::Central::Sync::prepare_endpoint_keys - User devices list is empty, nothing to fetch from Central
2021-11-25 11:05:39 INFO Syncinfo.pm[27999]:49 SFOS::HBtrust::Central::Syncinfo::syncinfo - enabled
2021-11-25 11:05:39 INFO Syncmissing.pm[27999]:60 SFOS::HBtrust::Central::Syncmissing::syncmissing - Reporting 4 endpoints as missing to Sophos Central
2021-11-25 11:05:39 INFO Syncmissing.pm[27999]:89 SFOS::HBtrust::Central::Syncmissing::_report_missing_heartbeat - Sending Missing Endpoints to Sophos Central: https://dzr-utm-amzn-us-west-2-fa88.upe.p.hmr.sophos.com/sophos/api/utm/34804ca5-f37d-40f5-a356-bb6a2ab0e3fe/heartbeat/missing
2021-11-25 11:05:40 INFO Syncmissing.pm[27999]:63 SFOS::HBtrust::Central::Syncmissing::syncmissing - Sophos Central requested status for 4 endpoints

In order to search the exact log line use below commands,
- Sophos450_WP02_SFOS 18.5.1 MR-1-Build326# less /log/heartbeat.log | grep 'ssl3_read_bytes sslv3 alert certificate expired'
- Sophos450_WP02_SFOS 18.5.1 MR-1-Build326# less /log/ hbtrust.log | grep 'User devices list is empty, nothing to fetch from Central'

To check the connectivity of the endpoint client and Sophos Firewall, you can check the tcpdump on heartbeat port 8347.
Ensure that the certificate files are updated using the following command:

XG450_WP02_SFOS 18.5.1 MR-1-Build326# ls -l /conf/sysfiles/heartbeatd/

Note: server.crt & server.key should be recently updated.

Possible Solutions

If you get the exact similar logs as mentioned above, the solutions described below may help fix the issue:

To fix the issue, follow KBA - https://support.sophos.com/support/s/article/KB-000037006?language=en_US.
Re-registration of the Sophos Firewall on Sophos Central is required if the aforementioned steps do not resolve the issue.

After the issue is fixed, we can see the heartbeats on the UI of Sophos Firewall, as shown below: