Sophos Heartbeat not working / Errors in heartbeatd.log

After an recent update? The Clients got a new certificate after the upgrade to V18.5 MR2 (or newer). There was a one time change. Check if the client can reach central and check if it can actually refresh the policy. 

Yes, after update from V18.5 MR2. The clients can reach central and receive policy changes. Is there a way to check the certificates?

When did you do the Upgrade? This policy upgrade could take some minutes/hours. 

In the middle of April

You should check this one: https://support.sophos.com/support/s/article/KB-000043489?language=en_US

Otherwise then create a support case. 

I‘ve checked this last week. The checkbox is not enabled in all of our rules.

we've had this when going to MR2 and it was the worst upgrade experience.

as a workaround: re-install intercept X on one of the devices. it pulls the correct certificates during installation, while running machines take hours or days.

but since middle of April is really a long time.

since our upgrade, when the heartbeat certificate of the firewall has been renewed, we notice the following when a known, working client connects:

1st attempt fails with the same error message you posted (this is the connection to the old previously used firewall certificate)

2nd attempt succeeds when connecting to the new firewall heartbeat certificate.

so still our firewall HB log is full with SSL errors even today.

there is some debugging and descriptions in the unorganized responses to this thread: https://community.sophos.com/sophos-xg-firewall/f/discussions/131468/sophos-firewall-v18-5-mr2-feedback-and-experiences

failed: old cert used by client

succeeded: new cert used by client

Sophos argued, this was caused by FW rules with HB requirement, but this was not true - this applies only for firewall rules towards Sophos Central and DNS and there has not been a HB requirement. In the end, they could not identify the root cause after 2 months of debugging.

Check the certificates used by the firewall:

/conf/sysfiles/heartbeatd/

server.crt
server.key

with those used by the client

C:\ProgramData\Sophos\Heartbeat\Config\Heartbeat.xml

you would need to export the individual certificates to crt files or certificate analysis tools and can then identify them.

Debug tips:

• Relevant entries from the HBTrust Log file
• Relevant entries from heartbeat.log in debug mode (service -t json -b '{"debug":"1"}' -ds nosync heartbeat:debug)
• Tcpdump from the UTM for the host 52.5.76.173 on port 8347

Our case No. from January 2022, if case you need it for case-cross-reference.

04793577 / Heartbeat Connection SSL error after update

The blocked endpoint is a new installed workstation.

The firewalls server.crt is equal to the last entry in <utmCerts> section of the endpoints Heartbeat.xml (same fingerprint and serial number) and the certificate is valid until 14th December 2022.

I wonder why the last entry from hbtrust.log is from 22nd April?

2022-04-20 10:36:20Z INFO Sync.pm[21367]:102 SFOS::HBtrust::Central::Sync::get_fingerprints - Getting fingerprints from UTM-PIC: https://utm-cloudstation-eu-central-1.prod.hydra.sophos.com
2022-04-20 10:36:21Z WARN API.pm[21367]:119 SFOS::Common::Central::API::send_request - HTTP/1.1 403 Forbidden
Connection: close
Date: Wed, 20 Apr 2022 10:36:21 GMT
Server: -
Content-Length: 0
Client-Date: Wed, 20 Apr 2022 10:36:21 GMT
Client-Peer: 18.184.204.140:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
Client-SSL-Cert-Subject: /CN=utm-cloudstation-eu-central-1.prod.hydra.sophos.com
Client-SSL-Cipher: ECDHE-RSA-AES128-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL


2022-04-20 10:36:21Z ERROR Tools.pm[21367]:97 SFOS::Common::Central::Tools::report_status - ETOKENEXPIRED: Firewall could not authenticate with current access token

An tcpdump shows regular communication between the blocked endpoint and 52.5.76.173 on port 8347.

Today evening I will turn on the debug mode for heartbeat.log.

The blocked endpoint is a new installed workstation.

The firewalls server.crt is equal to the last entry in <utmCerts> section of the endpoints Heartbeat.xml (same fingerprint and serial number) and the certificate is valid until 14th December 2022.

I wonder why the last entry from hbtrust.log is from 22nd April?

2022-04-20 10:36:20Z INFO Sync.pm[21367]:102 SFOS::HBtrust::Central::Sync::get_fingerprints - Getting fingerprints from UTM-PIC: https://utm-cloudstation-eu-central-1.prod.hydra.sophos.com
2022-04-20 10:36:21Z WARN API.pm[21367]:119 SFOS::Common::Central::API::send_request - HTTP/1.1 403 Forbidden
Connection: close
Date: Wed, 20 Apr 2022 10:36:21 GMT
Server: -
Content-Length: 0
Client-Date: Wed, 20 Apr 2022 10:36:21 GMT
Client-Peer: 18.184.204.140:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
Client-SSL-Cert-Subject: /CN=utm-cloudstation-eu-central-1.prod.hydra.sophos.com
Client-SSL-Cipher: ECDHE-RSA-AES128-SHA256
Client-SSL-Socket-Class: IO::Socket::SSL


2022-04-20 10:36:21Z ERROR Tools.pm[21367]:97 SFOS::Common::Central::Tools::report_status - ETOKENEXPIRED: Firewall could not authenticate with current access token

An tcpdump shows regular communication between the blocked endpoint and 52.5.76.173 on port 8347.

Today evening I will turn on the debug mode for heartbeat.log.

Is this firewall registered to Sophos Central?

This would be a normal log block in that file

2022-05-31 11:52:55Z INFO hbtrust[29709]:73 main:: - Locking HBtrust by setting LOCK_EX on /bin/hbtrust (sync)
2022-05-31 11:52:55Z INFO hbtrust[29709]:112 main:: - Synchronizing Data between Sophos Central and XG Firewall
2022-05-31 11:52:55Z INFO Sync.pm[29709]:102 SFOS::HBtrust::Central::Sync::get_fingerprints - Getting fingerprints from UTM-PIC: https://utm-cloudstation-eu-central-1.prod.hydra.sophos.com
2022-05-31 11:52:56Z INFO Sync.pm[29709]:152 SFOS::HBtrust::Central::Sync::prepare_endpoint_keys - Get fingerprints stored in database
2022-05-31 11:52:56Z INFO Sync.pm[29709]:205 SFOS::HBtrust::Central::Sync::get_endpoint_keys - Getting endpoint certificates from UTM-PIC: https://utm-cloudstation-eu-central-1.prod.hydra.sophos.com
2022-05-31 11:52:56Z INFO Sync.pm[29709]:259 SFOS::HBtrust::Central::Sync::set_fingerprints - Store endpoint information in database
2022-05-31 11:52:56Z INFO Sync.pm[29709]:266 SFOS::HBtrust::Central::Sync::set_fingerprints - Retrieving endpoint information finished successfully
2022-05-31 11:52:56Z INFO Syncinfo.pm[29709]:49 SFOS::HBtrust::Central::Syncinfo::syncinfo - enabled
2022-05-31 11:52:56Z INFO Syncmissing.pm[29709]:60 SFOS::HBtrust::Central::Syncmissing::syncmissing - Reporting 0 endpoints as missing to Sophos Central
2022-05-31 11:52:56Z INFO Syncmissing.pm[29709]:89 SFOS::HBtrust::Central::Syncmissing::_report_missing_heartbeat - Sending Missing Endpoints to Sophos Central: https://utm-cloudstation-eu-central-1.prod.hydra.sophos.com/sophos/api/utm/cfxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc4/heartbeat/missing
2022-05-31 11:52:56Z INFO Syncmissing.pm[29709]:63 SFOS::HBtrust::Central::Syncmissing::syncmissing - Sophos Central requested status for 2 endpoints
2022-05-31 11:52:56Z INFO Syncmissing.pm[29709]:121 SFOS::HBtrust::Central::Syncmissing::_report_endpoint_status - Sending status of requested endpoints to Sophos Central: https://utm-cloudstation-eu-central-1.prod.hydra.sophos.com/sophos/api/utm/cfxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc4/heartbeat/state
2022-05-31 11:52:57Z INFO Syncmeta.pm[29709]:48 SFOS::HBtrust::Central::Syncmeta::syncmeta - Requesting customer information from Sophos Central: https://utm-cloudstation-eu-central-1.prod.hydra.sophos.com/sophos/api/utm/cfxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc4/customer
2022-05-31 11:52:57Z INFO IPSET.pm[29709]:40 SFOS::HBtrust::IPSET::write_ipset_sync_file - Write IPSET synchronization file to: /tmp/hb_magic_ipset
2022-05-31 11:52:57Z INFO Syncmeta.pm[29709]:79 SFOS::HBtrust::Central::Syncmeta::syncmeta - Calling heartbeat_ipset OPCODE to set IPSET and synchronize across HA/Cluster
2022-05-31 11:52:57Z INFO hbtrust[29709]:129 main:: - LOCK_EX on /bin/hbtrust is being removed

Yes and no. The current primary device is registered with Sophos Central. After the update from MR2 to MR3 we had a lot of problems. So we disbled HA, resets the auxilliary device and enabled HA again. Now we are unable to register the second device. Our partner closed the support case with the recommendation to reinstall both devices and restore a backup.

At this time I think, this would be the only way to resolve our issues. 

Last weekend I wasted a lot of time reinstalling the auxiliary with v18.5.3 and v19, strictly following the instructions in this article - without success! I also tried to create the USB stick with Rufus and DD mode. The install process stops after 1% with the error that a partition could not be found.
I feel that Sophos Heartbeat is working again after shutting down both appliances and reinitiating HA. The number of errors in heartbeatd.log is significantly lower.

It is solved! This morning the update to v19 was successful and the heartbeat problem is gone.