Sophos Firewall: v18.5 MR2: Feedback and experiences

Installed and all our workstation Heartbeats are missing (after rebooting workstations).

Quite an issue as heartbeats are required for all workstation connectivity. Had to physically connect to the XG to put a temporary access rule in.

we're having the same issue after upgrade

The clients will not pick a new hb certificate unless we pull the sophos installer from central, disable tamper protection, install over the existing intercept-x installation. when the installer want's to reboot, there is a new certificate in C:\ProgramData\Sophos\Heartbeat\Config\Heartbeat.xml

what a mess, this is so disappointing

the clients can obviously reach central, the firewall has a new certificate but the endpoints sit there like a duck, just throw SSL/TLS errors and you cannot do nothing else than reinstall.

XG:
[2022-01-08 07:20:51.369Z] WARN HBSession.cpp[30041]:344 bufferDisconnectEvent - Incoming connection from 172.16.xxx.xxx failed. SSL error: SSL routines:ssl3_read_bytes tlsv1 alert internal error
[2022-01-08 07:21:51.404Z] WARN HBSession.cpp[30041]:344 bufferDisconnectEvent - Incoming connection from 172.16.xxx.xxx failed. SSL error: SSL routines:ssl3_read_bytes tlsv1 alert internal error

Client:
2022-01-08T06:25:02.216Z [ 6952:10608] A ----------------------------------------------------------------------------------------------------
2022-01-08T06:25:02.218Z [ 6952:10608] A Starting Heartbeat version 1.15.783.0
2022-01-08T06:25:02.219Z [ 6952:10608] A ----------------------------------------------------------------------------------------------------
2022-01-08T06:25:02.279Z [ 6952: 7568] E TLS authentication failed after connecting.
2022-01-08T06:31:53.752Z [ 6952: 7568] A Connection failed.
2022-01-08T06:32:44.793Z [ 6952: 7568] E TLS authentication failed after connecting.
2022-01-08T07:09:19.245Z [ 6952:10608] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:09:19.247Z [ 6952:10608] A Stopped Heartbeat
2022-01-08T07:09:19.249Z [ 6952:10608] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:09:20.519Z [15420: 9900] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:09:20.520Z [15420: 9900] A Starting Heartbeat version 1.15.783.0
2022-01-08T07:09:20.522Z [15420: 9900] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:09:20.573Z [15420:15380] E TLS authentication failed after connecting.
2022-01-08T07:22:19.214Z [15420: 4968] A The connection configuration has changed. Reloading settings.
2022-01-08T07:22:19.221Z [15420: 4968] A The connection configuration has changed. Reloading settings.

Support was no help so far: case ID 04793577 

FWs re-registered to Central already

KB-000043489 is a dead end

KB-000037006 - which is for v17 and earlier has been applied by support. did not change anything

Thanks for your answers JasP and LuCar Toni.

In the meantime we've reinstalled the Intercept X on all Windows Servers, and Admin machines. Fixed already during the installation the heartbeat.xml and the machines had Heartbeat afterwards. As we're did this in the same network, with the same FW rules, this is why I wrote: the clients can reach Central and they can do what they need to pick the new Cert. Also we re-checked DNS communication never had HB requirement.

We could not fix this on Linux Servers with Heartbeat module and Mac computers by reinstalling the agent. They behave differently.

Many machines recovered themselves after many hours. For our linux machines: they were 18h offline from heartbeat and then recovered. Unbelievable...

Yesterday, after wasted 34h with L1 Support this finally got to GES and someone started looking at the firewall.

Now the suggestion is (what JasP said caused the 2nd HB issue) to reregister once again or simply wait one or two days longer.

we have found few of the Sophos of the endpoint got regenerated and that has a starting of date of 7th January.  The certificate on the Firewall has the certificate valid from 17th July 2021 to July 22. Which lead to the mismatch of the certificate. 

deregister the firewall (both) from Sophos Central
Before registering the appliance clear all the contents from
/conf/sysfiles/heartbeatd/ using the command below.
rm -rf /conf/sysfiles/heartbeatd/

He also mentioned the FW files:

/conf/heartbeatd/ep_cert.crt
/conf/heartbeatd/certificate_store.db

Well, I re-registered the firewall straight after we discovered, that we're having a major HB issue. Of course without deleting some certificates on the firewall.

What I've checked on the firewall quite early was the server certificate:

XG430_WP02_SFOS 18.5.2 MR-2-Build380# ls -la /conf/sysfiles/heartbeatd/
drwxrwx---    3 root     heartbea      1024 Jan 10 05:56 .
drwxr-xr-x    7 root     0             1024 Jan  8 09:02 ..
drwxr-xr-x    2 root     heartbea      1024 Jan 10 05:56 ca-certificates
-rw-r-----    1 root     heartbea    699392 Jan 10 05:56 certificate_store.db
-rw-r--r--    1 root     0            57344 Jan 10 03:56 endpoint_store.db
-r--r-----    1 root     0             1667 Jan  8 07:32 server.crt
-r--------    1 root     0             3326 Jan  8 07:32 server.key
-rw-r--r--    1 root     0               54 Jan  8 07:32 sophos-central-customer-info.json
-rw-r--r--    1 root     0             5903 Jan 10 05:56 sophos-central.json

The Server Certificatre has the new timestamp: Jan  8 07:32 server.crt

This is matching the upgrade time to SFOS 18.5.2.

And I have checked that Server Cert externally:

I wonder why support says, the firewall uses an old certificate. I cannot confirm this as seen above.

edit: 2022-01-10 09:00
clients that were offline since the upgrade, including Mac's, are picking the new certificate now after powering on in the network after 2-10 minutes automatically. they create some SSL TLSv1 errors in the heartbeatd log on the FW but then in that time range get green HB after they picked the new cert.

MR2 did renewal the certificate due strict compliance requirements (FIPS etc.). Which means this was a one time change and not a "each and every upgrade".

That's all OK. But there is some knowledge only at Sophos high level supporters and Senior Sales Team about this process to take 1-2 days to re-register all endpoints with heartbeat. I have seen 18 hours for multiple servers in my environment.

This time is insane and this should be written in the known issues to the release notes.

Please do that. Hope that others get warned before they decide to upgrade one evening and have trouble the next days with their production.

I tried to reproduce this on my test lab. I made a upgrade from V18.0 MR5 to V18.5 MR2. The certificate was renewaled. Heartbeat was blocked for some minutes, until MCS was able to fetch the new policy and all clients (multiple clients and servers) could fetch the new certificate. Firewall rule was ANY - ANY - ANY --> Block without heartbeat. Therefore the client could not communicate to any website anymore. But MCS still works due the whitelisting of SFOS. 

You could take a look into your mcsagent.log on the client, if you see a delay or an issue of the communication itself (DNS as explained). MCS is the service to fetch the policies from Central. See: https://support.sophos.com/support/s/article/KB-000034886?language=en_US

Generally speaking, i could not reproduce any kind of issue in this process. 

Thanks for testing it LuCar Toni. Please note we were on 18.0 MR6 before. And probably in a lot of environments, this error doesn't happen. But we're known to pick a lot of the bad apples... Maybe this is again related to HA environment.

I got the MCS logs from a client computer which hadn't been re-installed with endpoint and will forward that SDU to support FTP.

"But MCS still works due the whitelisting of SFOS."

Yes, that is also true here. The Clients are allowed to go to all the Central without Webinterception on the XG, are allowed to DNS, no HB requirement and block without heartbeat on all those basic network FW rules. They were showing with up-2-date time stamp in Central, so management communication was working.

see heartbeatlog issues of that client.

Working 22-1-7
FW Upgrade on 22-1-8
Connection failed until working again 22-1-10

heartbeat.log:

2022-01-07T13:59:32.308Z [ 5760: 7932] A Connection succeeded.
2022-01-07T13:59:32.310Z [ 5760: 7932] A Connected to 'ed98a5bf-ede8-4fbd-99b1-b0b1b0b13f1b' at IP address 52.5.76.173 on port 8347
2022-01-07T13:59:32.323Z [ 5760: 7932] A Connection closed (network error).
2022-01-07T13:59:32.324Z [ 5760: 7936] A Inactive Interfaces changed.
2022-01-07T13:59:33.340Z [ 5760: 7932] A Connection failed.
2022-01-08T06:55:12.246Z [ 5756: 6384] A ----------------------------------------------------------------------------------------------------
2022-01-08T06:55:12.247Z [ 5756: 6384] A Starting Heartbeat version 1.15.783.0
2022-01-08T06:55:12.248Z [ 5756: 6384] A ----------------------------------------------------------------------------------------------------
2022-01-08T06:55:12.300Z [ 5756: 7856] A Connection failed.
2022-01-08T06:55:42.389Z [ 5756: 7856] E TLS authentication failed after connecting.
2022-01-08T06:58:15.178Z [ 5756: 7856] A Connection failed.
2022-01-08T06:58:59.377Z [ 5792: 6496] A ----------------------------------------------------------------------------------------------------
2022-01-08T06:58:59.402Z [ 5792: 6496] A Starting Heartbeat version 1.15.783.0
2022-01-08T06:58:59.402Z [ 5792: 6496] A ----------------------------------------------------------------------------------------------------
2022-01-08T06:58:59.512Z [ 5792: 8076] E TLS authentication failed after connecting.
2022-01-08T06:59:29.600Z [ 5792: 8076] A Connection failed.
2022-01-08T07:03:04.545Z [ 5580: 6224] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:03:04.546Z [ 5580: 6224] A Starting Heartbeat version 1.15.783.0
2022-01-08T07:03:04.547Z [ 5580: 6224] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:03:04.643Z [ 5580: 7712] E TLS authentication failed after connecting.
2022-01-08T07:07:13.311Z [ 5708: 6588] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:07:13.312Z [ 5708: 6588] A Starting Heartbeat version 1.15.783.0
2022-01-08T07:07:13.313Z [ 5708: 6588] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:07:13.422Z [ 5708: 8012] E TLS authentication failed after connecting.
2022-01-08T07:08:34.669Z [ 5708: 8012] A Connection failed.
2022-01-08T07:11:14.053Z [ 5708: 8012] E TLS authentication failed after connecting.
2022-01-08T07:13:12.163Z [ 5708: 6588] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:13:12.164Z [ 5708: 6588] A Stopped Heartbeat
2022-01-08T07:13:12.165Z [ 5708: 6588] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:13:14.529Z [ 7552:17376] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:13:14.529Z [ 7552:17376] A Starting Heartbeat version 1.15.783.0
2022-01-08T07:13:14.530Z [ 7552:17376] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:13:14.676Z [ 7552:14032] E TLS authentication failed after connecting.
2022-01-08T09:46:55.603Z [ 7552: 3476] A The connection configuration has changed. Reloading settings.
2022-01-08T09:46:55.615Z [ 7552:14328] E 2015: The configuration could not be loaded. Default parameters will be used.
2022-01-08T09:46:55.616Z [ 7552: 3476] E 2015: The configuration could not be loaded. Default parameters will be used.
2022-01-08T09:46:55.617Z [ 7552:14328] A The stonewalling configuration has changed. Reloading settings.
2022-01-08T09:47:11.030Z [ 7552: 3476] A The connection configuration has changed. Reloading settings.
2022-01-08T09:47:11.031Z [ 7552:14328] A The stonewalling configuration has changed. Reloading settings.
2022-01-08T09:47:11.033Z [ 7552: 3476] A The connection configuration has changed. Reloading settings.
2022-01-08T09:51:15.060Z [ 7552:17376] A ----------------------------------------------------------------------------------------------------
2022-01-08T09:51:15.061Z [ 7552:17376] A Stopped Heartbeat
2022-01-08T09:51:15.062Z [ 7552:17376] A ----------------------------------------------------------------------------------------------------
2022-01-08T09:51:15.856Z [13524: 4468] A ----------------------------------------------------------------------------------------------------
2022-01-08T09:51:15.857Z [13524: 4468] A Starting Heartbeat version 1.15.783.0
2022-01-08T09:51:15.858Z [13524: 4468] A ----------------------------------------------------------------------------------------------------
2022-01-08T09:51:16.023Z [13524:15816] E TLS authentication failed after connecting.
2022-01-08T09:58:16.239Z [ 5452: 6284] A ----------------------------------------------------------------------------------------------------
2022-01-08T09:58:16.240Z [ 5452: 6284] A Starting Heartbeat version 1.15.783.0
2022-01-08T09:58:16.241Z [ 5452: 6284] A ----------------------------------------------------------------------------------------------------
2022-01-08T09:58:16.277Z [ 5452: 7544] A Connection failed.
2022-01-08T10:01:34.927Z [ 5452: 7544] E TLS authentication failed after connecting.
2022-01-08T10:03:33.687Z [ 5640: 6600] A ----------------------------------------------------------------------------------------------------
2022-01-08T10:03:33.688Z [ 5640: 6600] A Starting Heartbeat version 1.15.783.0
2022-01-08T10:03:33.689Z [ 5640: 6600] A ----------------------------------------------------------------------------------------------------
2022-01-08T10:03:33.798Z [ 5640: 7860] E TLS authentication failed after connecting.
2022-01-08T10:05:10.127Z [ 5640: 7860] A Connection failed.
2022-01-08T10:07:49.448Z [ 5640: 7860] E TLS authentication failed after connecting.
2022-01-08T12:18:18.945Z [ 5640: 7860] A Connection failed.
2022-01-08T12:45:35.451Z [ 5640: 7860] E TLS authentication failed after connecting.
2022-01-08T12:51:08.528Z [ 5640: 7860] A Connection failed.
2022-01-08T12:55:38.696Z [ 5640: 7860] E TLS authentication failed after connecting.
2022-01-08T12:59:10.473Z [ 5640: 7860] A Connection failed.
2022-01-08T13:16:38.627Z [ 5640: 7860] E TLS authentication failed after connecting.
2022-01-08T16:55:35.349Z [ 5640: 6600] A ----------------------------------------------------------------------------------------------------
2022-01-08T16:55:35.350Z [ 5640: 6600] A Stopped Heartbeat
2022-01-08T16:55:35.350Z [ 5640: 6600] A ----------------------------------------------------------------------------------------------------
2022-01-10T07:55:27.201Z [ 5400: 5684] A ----------------------------------------------------------------------------------------------------
2022-01-10T07:55:27.202Z [ 5400: 5684] A Starting Heartbeat version 1.15.783.0
2022-01-10T07:55:27.202Z [ 5400: 5684] A ----------------------------------------------------------------------------------------------------
2022-01-10T07:55:27.243Z [ 5400: 7464] A Connection failed.
2022-01-10T08:02:09.174Z [ 5400: 7432] A The connection configuration has changed. Reloading settings.
2022-01-10T08:02:09.191Z [ 5400: 7432] A The connection configuration has changed. Reloading settings.
2022-01-10T08:02:39.907Z [ 5400: 7464] A Connection succeeded.

note: 2022-01-08T09:46:55 ... E 2015: The configuration could not be loaded. Default parameters will be used.
At that point the heartbeat issue was already happening for more than 2 hours and we tried to work around the issue by deleting the heartbeat.xml file on that client manually. did not solve the issue as you can see. The EP heartbeat was working again on Jan 10th without further action by us.

MCSAgent Log contains no issues.

MCSClient log shows communication to Central was fine:

2022-01-08T07:13:04.143Z [12312: 1324] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:13:04.145Z [12312: 1324] A Starting version 4.15.70.0 of the Sophos MCS Client service.
2022-01-08T07:13:04.145Z [12312: 1324] A ----------------------------------------------------------------------------------------------------
2022-01-08T07:13:04.164Z [12312:16120] I The configuration monitor thread was started.
2022-01-08T07:13:04.164Z [12312:14348] I Config: setting 'statusRegulationDelay' set to 60.
2022-01-08T07:13:04.164Z [12312:14348] I Config: setting 'maximumStatusRegulationDelay' set to 300.
2022-01-08T07:13:04.164Z [12312:14348] I Config: setting 'statusTimeToLive' set to 43200.
2022-01-08T07:13:04.164Z [12312:14348] I Config: setting 'responseRegulationDelay' set to 1.
2022-01-08T07:13:04.164Z [12312:14348] I Config: setting 'maximumResponseRegulationDelay' set to 1.
2022-01-08T07:13:04.164Z [12312:14348] I Config: setting 'errorCountTimeout' set to 300.
2022-01-08T07:13:04.164Z [12312:14348] I Config: setting 'eventRegulationDelay' set to 1.
2022-01-08T07:13:04.164Z [12312:14348] I Config: setting 'maximumEventRegulationDelay' set to 5.
2022-01-08T07:13:04.164Z [12312:14348] I Config: setting 'maximumAggregatedEvents' set to 32.
2022-01-08T07:13:04.164Z [12312:14348] I Config: setting 'commandPollingInterval' set to 55.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'flagsPollingInterval' set to 14400.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'policyPollingInterval' set to 300.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'policyTimeToLive' set to 345600.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'maximumBackoffCount' set to 10.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'maximumBackoffSeconds' set to 7200.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'randomSkewFactor' set to 1.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'httpConnectTimeout' set to 30.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'httpSendTimeout' set to 30.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'httpReceiveTimeout' set to 30.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'statusCacheDuration' set to 604800.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'useSystemProxy' set to 1.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'useAutomaticProxy' set to 1.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'useDirect' set to 1.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'diagnosticTrailLocation' set to C:\ProgramData\Sophos\Management Communications System\Endpoint\Trail.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'registrationToken' set to xxxxxxxxxxxxxxxxxx6f305f2cf397.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'presignedUrlServiceUrl' set to https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com/sophos/management/ep/presignedurls.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'presignedUrlServiceCredentials' set to xxxxxxxxxxxxP0ZQp6Mqg9H4=.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'pushPollRegulationDelayMilliseconds' set to 1.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'pushMaximumPollRegulationDelayMilliseconds' set to 1.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'pushPingTimeout' set to 90.
2022-01-08T07:13:04.165Z [12312:14348] I Config: setting 'pushFallbackPollInterval' set to 55.
2022-01-08T07:13:04.168Z [12312:14348] I Periodic evaluation interval configured for every 86400 seconds
2022-01-08T07:13:04.170Z [12312:12668] I Starting Channel monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\APPWL\Incoming
2022-01-08T07:13:04.171Z [12312: 4480] I Starting Channel monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\EDR\Incoming
2022-01-08T07:13:04.171Z [12312:14104] I Starting Channel monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\ForensicSnapshot\Incoming
2022-01-08T07:13:04.171Z [12312: 1492] I Starting Channel monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\RCA\Incoming
2022-01-08T07:13:04.171Z [12312:17240] I Starting Channel monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\FIM\Incoming
2022-01-08T07:13:04.171Z [12312:14840] I Starting directory change monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming
2022-01-08T07:13:04.172Z [12312:14348] I The Windows event log has been initialized.
2022-01-08T07:13:04.287Z [12312:14348] I Device ID: 4b500259-127a-xxxxxxxx-aab5-xxxxxxxxxx
2022-01-08T07:13:04.287Z [12312:14348] I Tenant ID: f2783ff7-0c37-47e6-9d34-be7dd2a07095
2022-01-08T07:13:04.287Z [12312:14348] I Features: APPCNTRL AV CLEAN CORE DLP DVCCNTRL EFW HBT LIVEQUERY LIVETERMINAL MDR NTP SAV SDU WEBCNTRL XPD
2022-01-08T07:13:04.287Z [12312:14348] I Authentication token expires at 2022-01-08T07:58:34Z
2022-01-08T07:13:04.298Z [12312:12140] I service tamper protection enabled
2022-01-08T07:13:04.456Z [12312:12828] I [connect] trying server https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com/sophos/management/ep
2022-01-08T07:13:04.457Z [12312:12828] I [connect] trying direct connection without a proxy
2022-01-08T07:13:04.457Z [12312:12828] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep
2022-01-08T07:13:04.584Z [12312:12828] I 200 : sent=0 rcvd=168 elapsed=127ms
2022-01-08T07:13:04.584Z [12312:12828] I [connect] using server https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com/sophos/management/ep without a proxy (peer address 3.127.212.169)
2022-01-08T07:13:04.585Z [12312:12828] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/flags/endpoint/b4052095-21a7-xxxxxxx-aa5b-xxxxxxxxx
2022-01-08T07:13:04.619Z [12312:12828] I 200 : sent=0 rcvd=1583 elapsed=34ms
2022-01-08T07:13:04.620Z [12312:12828] I Saved the Central flags
2022-01-08T07:13:04.622Z [12312:12828] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SEA;SHS;SWC;UI;APPSPROXY/endpoint/b4052095-21a7-xxxxxxx-aa5b-xxxxxxxxx
2022-01-08T07:13:04.639Z [12312:12828] I 200 : sent=0 rcvd=140 elapsed=17ms
2022-01-08T07:13:04.787Z [12312:12828] I AGENT status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.793Z [12312:12828] I SAV status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.794Z [12312:12828] I SWC status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.796Z [12312:12828] I ALC status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.797Z [12312:12828] I CORC status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.798Z [12312:12828] I CORE status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.799Z [12312:12828] I HBT status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.800Z [12312:12828] I HMPA status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.801Z [12312:12828] I LiveQuery status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.802Z [12312:12828] I MCS status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.803Z [12312:12828] I MDR status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.804Z [12312:12828] I NTP status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.804Z [12312:12828] I SDU status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.805Z [12312:12828] I SHS status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.806Z [12312:12828] I UI status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:13:04.807Z [12312:12828] I PUT https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/statuses/endpoint/b4052095-21a7-xxxxxxx-aa5b-xxxxxxxxx
2022-01-08T07:13:04.838Z [12312:12828] I 200 : sent=1811 rcvd=0 elapsed=31ms
2022-01-08T07:13:04.841Z [12312:12828] I EFW status processed <- 20220108071303-0005-status-EFW.xml
2022-01-08T07:13:04.842Z [12312:12828] I The agent status has changed to: {domain=domain, is_in_domain=1, computer_name=hostname, operating_system=WIN10, operating_system_friendly_name=Windows 10 Enterprise , os_major_version=10, os_minor_version=0, product_type=4, installation_type=Client, is_server=0, is_domain_controller=0, is_terminal_server=0, build_number=19042, system_language=1031, service_pack_major_version=0, service_pack_minor_version=0, computer_comment=, last_logged_on_user=domain\user, group_on_bootstrap=, user_sessions=((userDomain=domain, userName=user, userPrincipalName=user@domain.de, userSid=S-1-5-21-1803570019-140194396-1541874228-17022, state=0, type=0)), ipv4Addresses=(10.xxx.xxx.11, 192.xxx.xxx.51), ipv6Addresses=(2a02:8071:019c:b900:19ed:45d3:3761:fe1c, 2a02:8071:019c:b900:dd50:c004:9721:f015), macAddresses=(00:FF:BA:EB:40:BC, E4:46:B0:00:2B:2A, 14:85:7F:9D:74:CB, 16:85:7F:9D:74:CA, 14:85:7F:9D:74:CA, 14:85:7F:9D:74:CE), fullyQualifiedDomainName=hostname.domain.de, processorArchitecture=x64, deviceId=4b500259-127a-xxxxxxxx-aab5-xxxxxxxxxx, tenantId=f2783ff7-0c37-47e6-9d34-be7dd2a07095, products=(antivirus, intercept, mdr)}.
2022-01-08T07:13:04.842Z [12312:12828] I Establishing push connection
2022-01-08T07:13:04.844Z [12312:12828] I [push]: [connect] trying server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps
2022-01-08T07:13:04.844Z [12312:12828] I [push]: [connect] trying direct connection without a proxy
2022-01-08T07:13:04.844Z [12312:12828] I GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps
2022-01-08T07:13:04.933Z [12312:12828] I 200 : sent=0 rcvd=0 elapsed=88ms
2022-01-08T07:13:04.933Z [12312:12828] I [push]: [connect] using server https://mcs-push-server-eu-central-1.prod.hydra.sophos.com/ps without a proxy (peer address 18.195.226.46)
2022-01-08T07:13:04.934Z [12312:12828] I (async) GET https://mcs-push-server-eu-central-1.prod.hydra.sophos.com:443/ps/push/endpoint/b4052095-21a7-xxxxxxx-aa5b-xxxxxxxxx
2022-01-08T07:13:05.113Z [12312: 7240] I (async) 200 : connection established
2022-01-08T07:13:05.113Z [12312: 7240] I (async) 200 : chunk=1 rcvd=6 conntime=179ms
2022-01-08T07:13:05.113Z [12312: 7240] I Push connection was disconnected. Not triggering a command poll
2022-01-08T07:13:05.125Z [12312:12828] I The telemetry data is: {"mcs":{"agent":{"cloudPlatform":""},"flags":{"amsi-uac.available":true,"amsi.available":true,"amsi.block-and-clean.enabled":true,"amsi.fastregex.available":true,"behavioral-blocking.available":true,"behavioral.bms.enabled":true,"boot.modernweb.available":true,"boot.modernweb.block_by_scan":true,"boot.modernweb.block_by_url":true,"boot.modernweb.can_decrypt":true,"boot.sed.runtimeiocsjournal.available":true,"boot.ssp-clean.available":true,"boot.sting20.c2c3detections.enabled":true,"boot.sting20.datalossprevention.enabled":true,"boot.sting20.devicecontrol.enabled":true,"boot.sting20.downloadrepscanning.enabled":true,"boot.sting20.ondemandscanning.enabled":true,"boot.sting20.pejitscanning.enabled":true,"boot.sting20.realtimescanning.enabled":true,"boot.sting20.sscm.enabled":true,"boot.sting20.webcontrol.enabled":true,"boot.sting20.webprotection.enabled":true,"health.threat-services.enabled":false,"hmpa.amsiguard.enforce":true,"hmpa.amsiguard.silent":true,"hmpa.apisetguard.enforce":true,"hmpa.apisetguard.silent":true,"hmpa.branchtracing.enforce":true,"hmpa.branchtracing.silent":true,"hmpa.can-terminate-system-process.available":true,"hmpa.cookieguard.enforce":false,"hmpa.cookieguard.silent":true,"hmpa.credguard.v2.enforce":false,"hmpa.credguard.v2.silent":true,"hmpa.credguardsamreg.enforce":true,"hmpa.credguardsamreg.silent":true,"hmpa.cryptoguard.v5.enforce":false,"hmpa.cryptoguardefs.enforce":true,"hmpa.cryptoguardefs.silent":true,"hmpa.ctfguard.enforce":true,"hmpa.ctfguard.silent":true,"hmpa.heapheaphooray.enforce":true,"hmpa.heapheaphooray.silent":true,"hmpa.heapheaphooray.v2.enforce":true,"hmpa.heapheaphooray.v2.silent":true,"hmpa.ignore-attested.available":false,"hmpa.lockdownautorun.v2.enforce":true,"hmpa.lockdownmemory.v2.enforce":true,"hmpa.lockdownmemory.v2.silent":true,"hmpa.stackpivot.enforce":false,"ips.available":true,"ips.available_win7":true,"ips.filter.inbound":true,"ips.filter.outbound":true,"livequery.network-tables.available":true,"mlwindowsdir.available":true,"pinnedglobalreplocal.available":true,"pinnedglobalrepnetwork.available":true,"repair.available":false,"sav.hips.disabled":true,"scheduled_queries.next":false,"sdds3.ready":true,"sed.msthreatintel.enabled":false,"sed.multithreaded-hashing.enabled":true,"sed.pseudohandle-events.enabled":true,"sed.stricter-sophos-event-filtering.enabled":true,"sed.tp2020-denyfilelocks-win10.available":true,"sed.tp2020-denyfilelocks-win7-win8.available":true,"sed.tp2020-forcefilesharing-win10.available":true,"sed.tp2020-forcefilesharing-win7-win8.available":true,"sed.tp2020-oplocks-win10.available":true,"sed.tp2020-oplocks-win7-win8.available":false,"sed.tp2020-process-win10.available":true,"sed.tp2020-process-win7.available":true,"sed.tp2020-process-win8.available":true,"sed.tp2021-log-win10.available":true,"sed.tp2021-log-win7-win8.available":true,"sed.tpsafeboot.available":true,"ssp-clean.enabled":true,"ssp.appc.reporting.available":true,"ssp.clear-historian-db-file.enabled":true,"ssp.instant-core-clean-items.available":true,"ssp.multiplefilesubmission.available":true,"ssp.static.postanalysis.available":true,"ssp.submitfilemetadata.available":true,"sting20-pe.enabled":true,"su-setup.available":true,"vdldetections.available":true,"ztna.available":true},"preferredServer":{"server":"mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com","viaProxy":false,"viaMessageRelay":false,"authScheme":0},"pushServer":{"server":"mcs-push-server-eu-central-1.prod.hydra.sophos.com","isConnected":true},"remapper":{}}}
2022-01-08T07:13:38.080Z [12312:12828] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/4b500259-127a-xxxxxxxx-aab5-xxxxxxxxxx/feed_id/scheduled_query
2022-01-08T07:13:38.113Z [12312:12828] I 200 : sent=9538 rcvd=0 elapsed=33ms
2022-01-08T07:13:38.113Z [12312:12828] I Feed channel scheduled_query: uploading file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071333070.json result 0 purge false
2022-01-08T07:13:38.113Z [12312:12828] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071333070.json
2022-01-08T07:13:59.895Z [12312:12828] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SEA;SHS;SWC;UI;APPSPROXY/endpoint/b4052095-21a7-xxxxxxx-aa5b-xxxxxxxxx
2022-01-08T07:13:59.914Z [12312:12828] I 200 : sent=0 rcvd=140 elapsed=18ms
2022-01-08T07:14:04.859Z [12312: 7240] I (async) 200 : chunk=2 rcvd=7 conntime=60180ms
2022-01-08T07:14:04.860Z [12312: 7240] I The configuration has changed. Reloading settings.
2022-01-08T07:14:08.229Z [12312:12828] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/4b500259-127a-xxxxxxxx-aab5-xxxxxxxxxx/feed_id/scheduled_query
2022-01-08T07:14:08.266Z [12312:12828] I 200 : sent=1298 rcvd=0 elapsed=37ms
2022-01-08T07:14:08.266Z [12312:12828] I Feed channel scheduled_query: uploading file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071403221.json result 0 purge false
2022-01-08T07:14:08.266Z [12312:12828] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071403221.json
2022-01-08T07:14:08.279Z [12312:12828] I The telemetry data is: {"mcs":{"agent":{"cloudPlatform":""},"flags":{"amsi-uac.available":true,"amsi.available":true,"amsi.block-and-clean.enabled":true,"amsi.fastregex.available":true,"behavioral-blocking.available":true,"behavioral.bms.enabled":true,"boot.modernweb.available":true,"boot.modernweb.block_by_scan":true,"boot.modernweb.block_by_url":true,"boot.modernweb.can_decrypt":true,"boot.sed.runtimeiocsjournal.available":true,"boot.ssp-clean.available":true,"boot.sting20.c2c3detections.enabled":true,"boot.sting20.datalossprevention.enabled":true,"boot.sting20.devicecontrol.enabled":true,"boot.sting20.downloadrepscanning.enabled":true,"boot.sting20.ondemandscanning.enabled":true,"boot.sting20.pejitscanning.enabled":true,"boot.sting20.realtimescanning.enabled":true,"boot.sting20.sscm.enabled":true,"boot.sting20.webcontrol.enabled":true,"boot.sting20.webprotection.enabled":true,"health.threat-services.enabled":false,"hmpa.amsiguard.enforce":true,"hmpa.amsiguard.silent":true,"hmpa.apisetguard.enforce":true,"hmpa.apisetguard.silent":true,"hmpa.branchtracing.enforce":true,"hmpa.branchtracing.silent":true,"hmpa.can-terminate-system-process.available":true,"hmpa.cookieguard.enforce":false,"hmpa.cookieguard.silent":true,"hmpa.credguard.v2.enforce":false,"hmpa.credguard.v2.silent":true,"hmpa.credguardsamreg.enforce":true,"hmpa.credguardsamreg.silent":true,"hmpa.cryptoguard.v5.enforce":true,"hmpa.cryptoguardefs.enforce":true,"hmpa.cryptoguardefs.silent":true,"hmpa.ctfguard.enforce":true,"hmpa.ctfguard.silent":true,"hmpa.heapheaphooray.enforce":true,"hmpa.heapheaphooray.silent":true,"hmpa.heapheaphooray.v2.enforce":true,"hmpa.heapheaphooray.v2.silent":true,"hmpa.ignore-attested.available":false,"hmpa.lockdownautorun.v2.enforce":true,"hmpa.lockdownmemory.v2.enforce":true,"hmpa.lockdownmemory.v2.silent":true,"hmpa.stackpivot.enforce":false,"ips.available":true,"ips.available_win7":true,"ips.filter.inbound":true,"ips.filter.outbound":true,"livequery.network-tables.available":true,"mlwindowsdir.available":true,"pinnedglobalreplocal.available":true,"pinnedglobalrepnetwork.available":true,"repair.available":false,"sav.hips.disabled":true,"scheduled_queries.next":false,"sdds3.ready":true,"sed.msthreatintel.enabled":false,"sed.multithreaded-hashing.enabled":true,"sed.pseudohandle-events.enabled":true,"sed.stricter-sophos-event-filtering.enabled":true,"sed.tp2020-denyfilelocks-win10.available":true,"sed.tp2020-denyfilelocks-win7-win8.available":true,"sed.tp2020-forcefilesharing-win10.available":true,"sed.tp2020-forcefilesharing-win7-win8.available":true,"sed.tp2020-oplocks-win10.available":true,"sed.tp2020-oplocks-win7-win8.available":false,"sed.tp2020-process-win10.available":true,"sed.tp2020-process-win7.available":true,"sed.tp2020-process-win8.available":true,"sed.tp2021-log-win10.available":true,"sed.tp2021-log-win7-win8.available":true,"sed.tpsafeboot.available":true,"ssp-clean.enabled":true,"ssp.appc.reporting.available":true,"ssp.clear-historian-db-file.enabled":true,"ssp.instant-core-clean-items.available":true,"ssp.multiplefilesubmission.available":true,"ssp.static.postanalysis.available":true,"ssp.submitfilemetadata.available":true,"sting20-pe.enabled":true,"su-setup.available":true,"vdldetections.available":true,"ztna.available":true},"preferredServer":{"server":"mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com","viaProxy":false,"viaMessageRelay":false,"authScheme":0},"pushServer":{"server":"mcs-push-server-eu-central-1.prod.hydra.sophos.com","isConnected":true},"remapper":{}}}
2022-01-08T07:14:38.308Z [12312:12828] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/4b500259-127a-xxxxxxxx-aab5-xxxxxxxxxx/feed_id/scheduled_query
2022-01-08T07:14:38.337Z [12312:12828] I 200 : sent=985 rcvd=0 elapsed=28ms
2022-01-08T07:14:38.337Z [12312:12828] I Feed channel scheduled_query: uploading file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071433313.json result 0 purge false
2022-01-08T07:14:38.337Z [12312:12828] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071433313.json
2022-01-08T07:14:54.712Z [12312:12828] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SEA;SHS;SWC;UI;APPSPROXY/endpoint/b4052095-21a7-xxxxxxx-aa5b-xxxxxxxxx
2022-01-08T07:14:54.738Z [12312:12828] I 200 : sent=0 rcvd=140 elapsed=26ms
2022-01-08T07:15:04.483Z [12312:12828] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/4b500259-127a-xxxxxxxx-aab5-xxxxxxxxxx/feed_id/scheduled_query
2022-01-08T07:15:04.516Z [12312:12828] I 200 : sent=1084 rcvd=0 elapsed=33ms
2022-01-08T07:15:04.517Z [12312:12828] I Feed channel scheduled_query: uploading file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071459481.json result 0 purge false
2022-01-08T07:15:04.517Z [12312:12828] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071459481.json
2022-01-08T07:15:04.603Z [12312:10384] I (async) 200 : chunk=3 rcvd=7 conntime=120177ms
2022-01-08T07:15:12.734Z [12312:12828] I HMPA status is identical to last cached status, ignoring timestamp: NOT sending to server
2022-01-08T07:15:12.736Z [12312:12828] I PUT https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/statuses/endpoint/b4052095-21a7-xxxxxxx-aa5b-xxxxxxxxx
2022-01-08T07:15:12.805Z [12312:12828] I 200 : sent=13056 rcvd=0 elapsed=69ms
2022-01-08T07:15:12.815Z [12312:12828] I ALC status processed <- 20220108071315-0010-status-ALC.xml
2022-01-08T07:15:12.817Z [12312:12828] I APPSPROXY status processed <- 20220108071402-0011-status-APPSPROXY.xml
2022-01-08T07:15:38.485Z [12312:12828] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/4b500259-127a-xxxxxxxx-aab5-xxxxxxxxxx/feed_id/scheduled_query
2022-01-08T07:15:38.532Z [12312:12828] I 200 : sent=1027 rcvd=0 elapsed=47ms
2022-01-08T07:15:38.532Z [12312:12828] I Feed channel scheduled_query: uploading file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071533475.json result 0 purge false
2022-01-08T07:15:38.532Z [12312:12828] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071533475.json
2022-01-08T07:15:49.527Z [12312:12828] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SEA;SHS;SWC;UI;APPSPROXY/endpoint/b4052095-21a7-xxxxxxx-aa5b-xxxxxxxxx
2022-01-08T07:15:49.550Z [12312:12828] I 200 : sent=0 rcvd=140 elapsed=23ms
2022-01-08T07:16:04.352Z [12312:10384] I (async) 200 : chunk=4 rcvd=7 conntime=180180ms
2022-01-08T07:16:38.666Z [12312:12828] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/v2/data_feed/device/4b500259-127a-xxxxxxxx-aab5-xxxxxxxxxx/feed_id/scheduled_query
2022-01-08T07:16:38.701Z [12312:12828] I 200 : sent=813 rcvd=0 elapsed=35ms
2022-01-08T07:16:38.702Z [12312:12828] I Feed channel scheduled_query: uploading file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071633672.json result 0 purge false
2022-01-08T07:16:38.702Z [12312:12828] I Feed channel scheduled_query: uploaded file C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming\scheduled-20220108071633672.json
2022-01-08T07:16:44.346Z [12312:12828] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SEA;SHS;SWC;UI;APPSPROXY/endpoint/b4052095-21a7-xxxxxxx-aa5b-xxxxxxxxx
2022-01-08T07:16:44.369Z [12312:12828] I 200 : sent=0 rcvd=140 elapsed=23ms
2022-01-08T07:17:04.098Z [12312:10384] I (async) 200 : chunk=5 rcvd=7 conntime=240179ms
2022-01-08T07:17:08.700Z [12312:12828] I POST https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443

An other log snip of MSClient where you can see exact timestamps that had failures in the heatbeat log shown above:

2022-01-08T10:03:32.932Z [ 5836: 6456] I Config: setting 'diagnosticTrailLocation' set to C:\ProgramData\Sophos\Management Communications System\Endpoint\Trail.
2022-01-08T10:03:32.932Z [ 5836: 6456] I Config: setting 'registrationToken' set to xxxxxxxxxxxxxxxxx.
2022-01-08T10:03:32.932Z [ 5836: 6456] I Config: setting 'presignedUrlServiceUrl' set to https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com/sophos/management/ep/presignedurls.
2022-01-08T10:03:32.932Z [ 5836: 6456] I Config: setting 'presignedUrlServiceCredentials' set to xxxxxxxxxxxxxxJuH50qiP0ZQp6Mqg9H4=.
2022-01-08T10:03:32.932Z [ 5836: 6456] I Config: setting 'pushPollRegulationDelayMilliseconds' set to 1.
2022-01-08T10:03:32.932Z [ 5836: 6456] I Config: setting 'pushMaximumPollRegulationDelayMilliseconds' set to 1.
2022-01-08T10:03:32.932Z [ 5836: 6456] I Config: setting 'pushPingTimeout' set to 90.
2022-01-08T10:03:32.932Z [ 5836: 6456] I Config: setting 'pushFallbackPollInterval' set to 55.
2022-01-08T10:03:33.075Z [ 5836: 6456] I Periodic evaluation interval configured for every 86400 seconds
2022-01-08T10:03:33.095Z [ 5836: 7052] I Starting Channel monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\FIM\Incoming
2022-01-08T10:03:33.096Z [ 5836: 7040] I Starting Channel monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\APPWL\Incoming
2022-01-08T10:03:33.096Z [ 5836: 7044] I Starting Channel monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\EDR\Incoming
2022-01-08T10:03:33.097Z [ 5836: 7060] I Starting Channel monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\ForensicSnapshot\Incoming
2022-01-08T10:03:33.097Z [ 5836: 7064] I Starting Channel monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\RCA\Incoming
2022-01-08T10:03:33.098Z [ 5836: 7068] I Starting directory change monitor for: C:\ProgramData\Sophos\Management Communications System\Endpoint\Channels\LiveQueryScheduled\Incoming
2022-01-08T10:03:33.100Z [ 5836: 6456] I The Windows event log has been initialized.
2022-01-08T10:03:33.575Z [ 5836: 6456] I Device ID: 4b500259-127a-xxxxxxxxx-aab5-xxxxxxxxxxx
2022-01-08T10:03:33.575Z [ 5836: 6456] I Tenant ID: f2783ff7-0c37-xxxxxxxxx-9d34-xxxxxxxxxxxxx
2022-01-08T10:03:33.575Z [ 5836: 6456] I Features: APPCNTRL AV CLEAN CORE DLP DVCCNTRL EFW HBT LIVEQUERY LIVETERMINAL MDR NTP SAV SDU WEBCNTRL XPD
2022-01-08T10:03:33.575Z [ 5836: 6456] I Authentication token expires at 2022-01-09T07:57:52Z
2022-01-08T10:03:33.580Z [ 5836: 6456] I The configuration has changed. Reloading settings.
2022-01-08T10:03:33.606Z [ 5836: 7776] I service tamper protection enabled
2022-01-08T10:03:34.188Z [ 5836: 7772] I [connect] trying server https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com/sophos/management/ep
2022-01-08T10:03:34.188Z [ 5836: 7772] I [connect] trying direct connection without a proxy
2022-01-08T10:03:34.188Z [ 5836: 7772] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep
2022-01-08T10:03:34.647Z [ 5836: 7772] I 200 : sent=0 rcvd=168 elapsed=459ms
2022-01-08T10:03:34.648Z [ 5836: 7772] I [connect] using server https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com/sophos/management/ep without a proxy (peer address 52.28.79.68)
2022-01-08T10:03:34.649Z [ 5836: 7772] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/flags/endpoint/b4052095-21a7-xxxx-aa5b-xxxxxxxxxx
2022-01-08T10:03:34.944Z [ 5836: 7772] I 200 : sent=0 rcvd=1583 elapsed=295ms
2022-01-08T10:03:34.945Z [ 5836: 7772] I Saved the Central flags
2022-01-08T10:03:34.948Z [ 5836: 7772] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SEA;SHS;SWC;UI;APPSPROXY/endpoint/b4052095-21a7-xxxx-aa5b-xxxxxxxxxx
2022-01-08T10:03:34.981Z [ 5836: 7772] I 200 : sent=0 rcvd=140 elapsed=32ms

Do you have a Support case open for this? As far as i can see: Central did not push / render the HBT policy. Therefore the client did not get the policy with the applied certificate. If this worked after some time, the rendering was repaired by Central itself. 

Sophos Support will have access to the Logs of Central and the rendering of the policy. 

thanks for that additional information. may I ask you to add that reference to the central logs to case 04793577  so the tech working on it can check the central logs before they expire?

ist that the point where MCS Client receives the new Heartbeat Policy on Jan 10th? It looks like to me:

	Line 4982: 2022-01-10T08:02:08.449Z [ 5476: 7324] I Features: APPCNTRL AV CLEAN CORE DLP DVCCNTRL EFW HBT LIVEQUERY LIVETERMINAL MDR NTP SAV SDU WEBCNTRL XPD
	Line 4987: 2022-01-10T08:02:08.558Z [ 5476: 7324] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/commands/applications/ALC;AMSI;CORC;CORE;EFW;HBT;HMPA;LiveQuery;LiveTerminal;MCS;MDR;NTP;SAV;SDU;SEA;SHS;SWC;UI;APPSPROXY/endpoint/b4052095-21a7-xxxx-aa5b-xxxxxxx3be1
	Line 4995: 2022-01-10T08:02:08.801Z [ 5476: 7324] I Received policy fragment for adapter HBT, with type 27
	Line 4996: 2022-01-10T08:02:08.817Z [ 5476: 7324] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/policy/fragment/application/HBT/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxb4cf7b26d12b14d5664bf
	Line 4999: 2022-01-10T08:02:08.956Z [ 5476: 7324] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/policy/fragment/application/HBT/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx69c1b367e294080e7c3cb5a
	Line 5001: 2022-01-10T08:02:09.065Z [ 5476: 7324] I HBT27 policy queued -> 20220110080209-0001-policy-HBT27.xml
	Line 5001: 2022-01-10T08:02:09.065Z [ 5476: 7324] I HBT27 policy queued -> 20220110080209-0001-policy-HBT27.xml
	Line 5004: 2022-01-10T08:02:09.132Z [ 5476: 7324] I Received policy fragment for adapter HBT, with type 27
	Line 5005: 2022-01-10T08:02:09.145Z [ 5476: 7324] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/policy/fragment/application/HBT/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx98de246ec1f392c0c639ff4
	Line 5007: 2022-01-10T08:02:09.291Z [ 5476: 7324] I HBT27 policy queued -> 20220110080209-0002-policy-HBT27.xml
	Line 5007: 2022-01-10T08:02:09.291Z [ 5476: 7324] I HBT27 policy queued -> 20220110080209-0002-policy-HBT27.xml
	Line 5010: 2022-01-10T08:02:09.374Z [ 5476: 7324] I Received policy fragment for adapter HBT, with type 27
	Line 5011: 2022-01-10T08:02:09.389Z [ 5476: 7324] I GET https://mcs2-cloudstation-eu-central-1.prod.hydra.sophos.com:443/sophos/management/ep/policy/fragment/application/HBT/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx50aa56b26f446edc1ef760
	Line 5013: 2022-01-10T08:02:09.507Z [ 5476: 7324] I HBT27 policy queued -> 20220110080209-0003-policy-HBT27.xml
	Line 5013: 2022-01-10T08:02:09.507Z [ 5476: 7324] I HBT27 policy queued -> 20220110080209-0003-policy-HBT27.xml
	Line 5051: 2022-01-10T08:02:10.533Z [ 5476: 7324] I HBT status processed <- 20220110080209-0019-status-HBT.xml
	Line 5051: 2022-01-10T08:02:10.533Z [ 5476: 7324] I HBT status processed <- 20220110080209-0019-status-HBT.xml

The HBT Config Status file is:

<?xml version="1.0"?>
<statusCache>
    <cacheTime>2022-01-10T08:02:10.427328Z</cacheTime>
    <status><?xml version="1.1" encoding="UTF-8"?><StatusAndConfig><status><?xml version='1.0' encoding='UTF-8'?><status version="1.15.783.0"><CompRes Res='Same' RevID='3658192e804f6a6xxxxxxxxxxxxxxe4808e2bb6f5bb073' policyType='27'/></status></status><config></config></StatusAndConfig></status>
</statusCache>

And the HBT 27 Policy  ProgramData\Sophos\Management Communications System\Endpoint\Cache\HBT27.policy contains all the certificates and has date: 2022.01.10  09:02  (CET)

<?xml version="1.0"?>
<policy RevID="3658192e804f6a6xxxxxxxxxxxxxxe4808e2bb6f5bb073" policyType="27">
  <destination address="52.5.76.173" port="8347"/>
  <addresslist/>
  <enabled>true</enabled>
  <ztnaEnabled>false</ztnaEnabled>
  <renewalparams triggerDaysBefore="90" switchDaysAfter="4"/>
  <epcert fingerprint="M9n3Tv/xxxxxxxxxxx=">-----BEGIN CERTIFICATE-----
  
  certs following below...

Yes - Its the Policy (with the certificate). 

who may face this issue after upgrade and find this information in this terribly long thread here, refer to the case# mentioned earlier and this NC-83739 when contacting Sophos Support. From what I know, by today (Feb. 3rd) the issue is not 100% identified but has something to do more with Central than the XG. Central should inform/push new certificates to the endpoints but this process may take hours or days which is - of course - a little bit unacceptable.

who may face this issue after upgrade and find this information in this terribly long thread here, refer to the case# mentioned earlier and this NC-83739 when contacting Sophos Support. From what I know, by today (Feb. 3rd) the issue is not 100% identified but has something to do more with Central than the XG. Central should inform/push new certificates to the endpoints but this process may take hours or days which is - of course - a little bit unacceptable.