[Sophos Notification] Sophos Central Endpoint and SEC: Computers fail/hang on boot after the Microsoft Windows April 9, 2019 update

Hello David,

There is no mention of anything to do with PET
correct, but apparently this is how it is/was/should have been done. There's no need to describe the mechanism in detail if you expect it to work. Seems it doesn't though.
PET was only for reporting
It isn't - please see the /remediate command line parameter.

Christian

So when can we expect this to be resolved? Not updating isn't a solution.

Hopefully this isn't like the last major windows 7 issue we ran into that took over a month for sophos to "resolve" https://community.sophos.com/kb/en-us/131685

Hello Sammore,

it looks - from the rather prompt reaction of Microsoft - like this is not an issue for which Sophos has to bear the undivided blame (or even the major part of it). Microsoft wouldn't block their patches if some other vendor has "self-inflicted" problems.

Christian

To be a little fairer to Microsoft, I feel that both Microsoft and Sophos should be working closer together when OS changes such as the new patches are implemented. From what I have seen so far, if you are not running Sophos, Avast or Bit Defender then you were unaffected by problems with the updates that were released.

The files that were patched, more specifically WININIT.dll and PRINTUI.dll, appear to have a new file signature. Had Sophos been made aware of these changes (If they weren't, I am speculating here) then I am sure their Dev team could have rolled an update out to account for these changes and prevent the problems being experienced.

There is actually nothing wrong with the updates released by Microsoft. It's just simply a case that in order for them to work without issue, 3rd party vendors need to be aware of the changes and have patches in place for their products prior to release.

I'm not interested in who bares the blame, I'm interested in a solution that resolves the issue at hand and helps us avoid these issues in the future.

Right now we're at the mercy of Sophos to release a fix, whose track record in fixing these issues is slow to say the least.

Hi Everyone,

We appreciate your patience. Our team has been working non-stop to resolve the issue and we can now say confidently that we have identified the permanent fix and testing is underway. We plan to start automatically rolling out the fix to customers very soon, and if there are any cases where the update has to be manually applied, we will contact those customers directly.

The KBA will be updated as soon as possible once we have more information on the ETA and the Fix.

UPDATE: Sophos + Microsoft Windows April 9 update

I've just had it confirmed, by Sophos support (under case no: 8794284) that the automated update for the SEC AV/HIPS policies, referenced in the Sophos KB article 133945, isn't actually working for anybody, even the support team's own SEC. So, we currently have three, conflicting states from Sophos: 
1. SEC is being "automagically" updated and everything is cool. 
2. We need to run PET /remediate to try and force the policy update, but then some policies aren't being updated, "where the GUID in the CorrelationID column is NOT enclosed in curly brackets {}." 
Admittedly this one is from one of your forum moderators, @QC. 
3. The automatic update isn't working and Sophos don't yet know why and haven't actually updated the KB article, to reflect that this isn't working.

Can you please get someone to clarify what is going on?

Cheers,

David.

Hi @deejinoz,

I can confirm that the automatic exclusions are working (I just confirmed it on my lab environment) and I could also see that the endpoints receiving the updates. However, let me check with the case that you mentioned and see what could be causing the issue in your scenario.

Hello Richie Knight,

[this is not finger-pointing]
it's not that simple. Show me a vendor or a community that has never issued a patch that, although carefully tested, had unexpected and unwanted side-effects. Or at least none that affected only third-parties but not their (one of their) own product(s). working closer together is a noble concept but it's already hard to accomplish it "in-house".

need to be aware of the changes
How? In case of source-code patches you can theoretically asses the changes and their consequences. Or at least find the cause for an issue. In practice you need very good knowledge of the patched component. With binary, closed-source patches you have to rely on the information you get - unlikely that a vendor will disclose full details. Having access to patches in advance helps a little but not much. You'd detect issues that have sufficient prevalence - provided there isn't an additional factor that is present only in certain installations. Furthermore, the third-parties also need some time to have [their] patches in place.
And we're talking about documented standard interfaces. Of course it's possible that changes "inside the black box" cause something to break - but normally this should not be the case. And if a race-condition is involved it might be hard to debug. The vendor is not aware of significant changes, the third-party not aware of any "hacks", and as soon as you attach a debugger or subscribe to events the problem goes away.

if you are not running Sophos, Avast or Bit Defender
There's more than one way to skin a cat. Meaning: AFAIK no OS, filesystem or filesystem implementation provides an AV interface with corresponding semantics (and consequently there aren't AV-aware applications). I've never investigated why. Naturally an AV should do its work as early as feasible and with minimum impact on startup. Ideally it should also perform a self-check.  There's no pre-defined or recommended strategy though and thus no either-all-or-none.

a new file signature
as the sole cause is unlikely. It's not only user-space applications that are patched, system files change regularly.

Not even a closed ecosystem can guarantee that you'll be safe from such issues.

Christian

Hi Christian.

QC said:

[this is not finger-pointing]
it's not that simple. Show me a vendor or a community that has never issued a patch that, although carefully tested, had unexpected and unwanted side-effects. Or at least none that affected only third-parties but not their (one of their) own product(s). working closer together is a noble concept but it's already hard to accomplish it "in-house".

need to be aware of the changes
How? In case of source-code patches you can theoretically asses the changes and their consequences. Or at least find the cause for an issue. In practice you need very good knowledge of the patched component. With binary, closed-source patches you have to rely on the information you get - unlikely that a vendor will disclose full details. Having access to patches in advance helps a little but not much. You'd detect issues that have sufficient prevalence - provided there isn't an additional factor that is present only in certain installations. Furthermore, the third-parties also need some time to have [their] patches in place.
And we're talking about documented standard interfaces. Of course it's possible that changes "inside the black box" cause something to break - but normally this should not be the case. And if a race-condition is involved it might be hard to debug. The vendor is not aware of significant changes, the third-party not aware of any "hacks", and as soon as you attach a debugger or subscribe to events the problem goes away.

if you are not running Sophos, Avast or Bit Defender
There's more than one way to skin a cat. Meaning: AFAIK no OS, filesystem or filesystem implementation provides an AV interface with corresponding semantics (and consequently there aren't AV-aware applications). I've never investigated why. Naturally an AV should do its work as early as feasible and with minimum impact on startup. Ideally it should also perform a self-check.  There's no pre-defined or recommended strategy though and thus no either-all-or-none.

a new file signature
as the sole cause is unlikely. It's not only user-space applications that are patched, system files change regularly.

Not even a closed ecosystem can guarantee that you'll be safe from such issues.

Christian