Installation Problem Windows 10 Enterprise

This issue is currently occurring in my environment, seemingly out of nowhere.

Ive had a ticket open for the last 5 days, but it hasnt gone anywhere yet. Meanwhile, we are way out of compliance.

Below is the error being seen via the Enterprise Console.

These are all existing devices (Windows 10 enterprise) with existing (working installs) and machines began popping up in the "errors" list

last week with the above syntax, after normal updates/pushes from Enterprise console.

I was able to manually troubleshoot one device that was displaying in the list (server actually) And the only way I 

was able to get past this was to completely remove all components, then use Microsoft's MSIZAP app, which wipes 

the app from the installer db, removes reg entries, etc.. then reinstalling from scratch. Obviously not a fix that we are looking for since there are dozens of devices failing with this same problem.

My ticket with support is still pending.

Hello Eric Breen,

this is actually a different issue though it might have a common cause. Furthermore yours is not a Central environment and AFAIK Patch is not available with Central.

You say that upgrading of Patch failed but an install from scratch succeeded? Is the issue with Patch on all endpoints (all the same level of Windows 10) or have some upgraded? Patch was 10.0.311 since SESC 10.7.x, with 10.8.2 it should be 10.0.312. Is it indeed the update to 10.8.2 (there's, BTW, also a potential issue with Detoured DLL).

Christian

QC said:

Hello Eric Breen,

this is actually a different issue though it might have a common cause. Furthermore yours is not a Central environment and AFAIK Patch is not available with Central.

You say that upgrading of Patch failed but an install from scratch succeeded? Is the issue with Patch on all endpoints (all the same level of Windows 10) or have some upgraded? Patch was 10.0.311 since SESC 10.7.x, with 10.8.2 it should be 10.0.312. Is it indeed the update to 10.8.2 (there's, BTW, also a potential issue with Detoured DLL).

Christian

 

When you say "Central" do you mean the cloud offering?

If so, then you are correct. Our Sophos environment/platform is currently on-prem.

I just reviewed the documentation for the .DLL issue, but nowhere does it suggest that my problem is related.

The documentation doesnt provide any behavioral characteristics to help identify if one is experiencing the problem or not.

I do have a ticket opened with support, but I havent heard back. (opened Monday) I have submitted documentation detailing behavior being experienced, a few screenshots

and 4 SDU archives from 3 affected devices, and 1 server.

Hello Eric Breen,

yes, Central is the cloud product.

The console (management server) does not push any updates. Even Protect Computers isn't a push - it just creates a scheduled task that runs setup.exe. Endpoints always download and install what they find in the CID - normally according to the schedule. 

I just referred to the DLL article because that is an issue with the 10.8.2 version that is AFAIK being rolled out (i.e. some customers already have received it) and with 10.8.2 the Patch version also changes. Without a version change AutoUpdate should not attempt to install/update Patch and therefore you shouldn't see this error. Are your endpoints on 10.8.1 or already 10.8.2?

Christian

Hello Eric,

Can I please get the ticket number so that we can follow up?

Thank you!

Sorry for the delay in my response. My ticket # for this issue is: 8317465

QC, 

I apologize for the delay in my response.

I took this screenshot last Friday as a direct result of your question regarding endpoint version, and I did not upload it at the time.

I hope the attached screenshot helps to answer your question. This was taken from server in the environment I happened to be working on at the time I 

saw your question.

Hello Eric Breen,

the delay no problem.
Hm, indeed 10.8.2.311 that has a minor update for Patch - thus after a long time AutoUpdate attempts to update it. AFAIK the error is issued by ALUpdate.exe based on whatever information. The corresponding line in the ALUpdate log contains ALUpdate(Install.Failure): Installation of product .... Just curious - there's a line (actually in every cycle) that says UpdateCoordinator::UpdateNow: current platform is platformtoken. What is the value value of platformtoken?

Wonder why there are no other reports of this error. What's the Patch Agent version on your endpoints - all 1.0.311.1, only those with the error (has their number increased?) with 1.0.311 and other 1.0.312 or higher, or?

Christian

Hello Eric Breen,

the delay no problem.
Hm, indeed 10.8.2.311 that has a minor update for Patch - thus after a long time AutoUpdate attempts to update it. AFAIK the error is issued by ALUpdate.exe based on whatever information. The corresponding line in the ALUpdate log contains ALUpdate(Install.Failure): Installation of product .... Just curious - there's a line (actually in every cycle) that says UpdateCoordinator::UpdateNow: current platform is platformtoken. What is the value value of platformtoken?

Wonder why there are no other reports of this error. What's the Patch Agent version on your endpoints - all 1.0.311.1, only those with the error (has their number increased?) with 1.0.311 and other 1.0.312 or higher, or?

Christian

Christian, 

Output from ALUpdate trace logs from 3 examples (That I submitted to Sophos support) 

2x Win 10 laptops, and 1x windows server

Trace(2018-Aug-30 12:41:59): UpdateCoordinator::UpdateNow: current platform is WIN_10_X64 reelase: 1803

and another..

Trace(2018-Aug-30 08:55:13): UpdateCoordinator::UpdateNow: current platform is WIN_10_X64 reelase: 1703

and the server..

Trace(2018-Aug-30 08:28:09): UpdateCoordinator::UpdateNow: current platform is WIN_2008_R2_SVR_X64 reelase: 

The number of endpoints experiencing this has NOT increased since this began.

Via the EC, I opened 14 random endpoints experiencing the (software cannot be installed on the OS) error, and each endpoint is showing Patch Agent 1.0.311.1

Hello Eric Breen,

thanks. The others have Patch 1.0.312+ I assume. Platform is one property that ALUpdate determines and considers. Dunno if reelease is deliberate or a type but apparently it's not the Redstone . I was asking because perhaps the platform was incorrect or unavailable. Just an idea.
Another idea was the AutoUpdate version, 5.11 on 10.8.1, 5.12 on 10.8.2 Preview, and 5.14 on 18.0.2 Recommended. But as AutoUpdate updates itself as last component it can't be 5.14. So there's not yet a pattern and it's seemingly random but hard.

BTW: I opened 14 random endpoints - you get the patch agent version information for a list of or all endpoints using the Patch tab in the console. No need to view their details.

Christian

QC said:

So there's not yet a pattern and it's seemingly random but hard.

BTW: I opened 14 random endpoints - you get the patch agent version information for a list of or all endpoints using the Patch tab in the console. No need to view their details.

Christian

I dont see a pattern either, nor does my supervisor. Its not minimizing our concern though, because we are way out of compliance, for almost 2 weeks now.

Thanks for the heads up. I was doing it the hard way. 

Hello Eric Breen,

important things first:
way out of compliance
because? Because Patch fails to update? Or because endpoints show updating errors (even though in the Status tab they show as Up to date: Yes)? Compliance is a ***, sometimes, or: more often than desired. In spirit it's a good thing, in practice it can be irrational. Depends on your internal or external auditors.

IMO it's not yet a serious non-compliance for the following reasons:

• the update to Patch is a minor one (1.0.311 to 1.0.312.7), unlikely that it changes anything of major importance
• Patch is "just" an assessment tool. An endpoint's correct results generated with up to date data by an up to date Patch Agent give you its patch status w.r.t. the data feed. If compliance is based on the results but the quality of the feed has not been assessed - how significant is it?
• the slightly backlevel Patch does not diminish protection on the endpoint
• last but not least: As new versions are rolled out to customers over a longer period (10.8.2 Recommended is not yet available for me, nor is Preview yet on 10.8.2.2) the next to last version should be as good as the very latest one in terms of compliance

If it's only the error message that your auditors don't like you could downgrade the affected endpoints to the Previous Recommended 10.8.1. This is not to say that the issue should be investigated (and resolved).

Some (hopefully minor) errors on my side:
* should have read more carefully - it's definitely a typo as it says reelase, e and l switched (the additional output of release/reelase has been introduced in AutoUpdate 5.12)
* AutoUpdate is not the last component updated by AutoUpdate, Patch and SSP come later in the cycle
* nevertheless whether AU is updated before Patch or not shouldn't have an effect as ALUpdate.exe isn't restarted in the cycle,

The Patch install should have been attempted by ALUpdate 5.11 and it seems whatever bug or glitch is causing this already existed  before 10.8.2/5.14 - or it's the Setup.dll that comes with Patch.

Thinking about it - I was carried away by the screenshot with the OS errors you provided and concentrated on Patch. You said the AV agent in services was missing. So the Patch update error was just one of the errors on these endpoints and other components (particularly SAV) also failed (on all the 40+ machines)?

Christian

QC said:

If it's only the error message that your auditors don't like you could downgrade the affected endpoints to the Previous Recommended 10.8.1. This is not to say that the issue should be investigated (and resolved).

• Assuming a downgrade would work.

Some (hopefully minor) errors on my side:
* should have read more carefully - it's definitely a typo as it says reelase, e and l switched (the additional output of release/reelase has been introduced in AutoUpdate 5.12)

• I noticed this typo as well, but this was the syntax right from the log. It was not edited.

The Patch install should have been attempted by ALUpdate 5.11 and it seems whatever bug or glitch is causing this already existed  before 10.8.2/5.14 - or it's the Setup.dll that comes with Patch.

Thinking about it - I was carried away by the screenshot with the OS errors you provided and concentrated on Patch. You said the AV agent in services was missing. So the Patch update error was just one of the errors on these endpoints and other components (particularly SAV) also failed (on all the 40+ machines)?

• The AV service was missing from the server that I had easy access too. I have no evidence at this time that other (currently affected) endpoints would/are displaying the same problem. I felt it was strange enough to mention it, as it was throwing the same error as all the others. I just had easy access to it, so I was able to spend time investigating the problem on this box. Its easier than taking a laptop from an end user, despite the difference in OS architecture. I dont think this is OS specific, althought I still havent ruled out a Windows update (or several causing this)

Furthermore, I am unaware if you are an employee of the company or not. It really doesnt matter, as you have been assisting none the less. I just wanted to reiterate that I have an open ticket and the latest reply from the vendor is that they are still investigating (what sounds like) a "known" problem. I am anticipating escalation, though nothing has been confirmed yet.

As far as your comments regarding compliance, They are company policies that I didnt author, I just follow them.

That storm is being weathered by me.

Hello Eric Breen,

I'm not Sophos and not a Partner, just a customer's representative.

still investigating
thought as much. Normally you won't hear how far up it already is or some estimate how long it will take (if it's already possible to tell). There have been changes to the updating architecture and as far as I can see there are more to come.

Haven't been of much help so far unless at least one of my suggestions works [:)]. 

Christian