[Sophos Notification] Sophos Central Endpoint and SEC: Computers fail/hang on boot after the Microsoft Windows April 9, 2019 update

Hello Richie,

just to make sure, I'm not Sophos, not a partner, and I have no inside knowledge (just a long time in IT).

Systems are too complex. Most of the time issues are confined to certain configurations and combinations of HW and SW so the general public never really hears of them. There's not one what if, there are very many. We are more or less used to "minor" problems. 200 computers, same model, identical software environment - three refuse a certain patch. Either you can live with it or it's reinstall time. If you can live with it - then you have different configurations and over time no longer a homogeneous environment. Which computers would you select as test base?

Christian

Thanks for the clarity Christian.

Much like yourself, I too have been working in I.T. for a long time. (19 years in fact. Now I feel old haha). I work for a leading MSP, (no names mentioned) who support many thousands of systems. We are a Sophos platinum partner (one of the biggest) and will need to be seeking further reassurances regarding these problems in moving forward.

As I am sure you can imagine, when problems such as these occur, we have a major headache and few ways to cover the cost and time in cleanup. (Which can be extremely high). The longer these issues continue, the more costs are incurred and the bigger the strain on our client relations which ultimately have an affect on Sophos in terms of our buying power.

Beta testing is a must in my opinion to try and help minimize these types of incidents. I fully understand problems as a result of patching but the severity of the issues caused in this instance is simply not acceptable.

Richie.

Hello Richie,

19 years
almost twice this ... I'm not feeling I am ;).
platinum partner
thought you'd get more information than us mere customers.

problems such as these
are indeed more than a pain.
As said, I'm just inferring from a few bits of information (I didn't even "consult the internet" or "asked Mr. Google"). Initially I became aware of the problem because a server admin reported an issue after applying the latest Microsoft patches. Didn't even mention Sophos but the (SEC) console showed an updating error, I checked the Sophos knowledgebase and found 133945. Not sure if the Microsoft articles already mentioned Sophos under Known issues, I think not. Anyway, I'm not aware that Sophos has a close relation with Microsoft. That Microsoft quickly blocked the patches was (IMO) a sign of, err, consciousness of potential guilt. Therefore I'm quite sure that Sophos didn't have a chance to proactively work on it. As you say, it might have been something that was deemed inconsequential - whether bona fide or due to negligence I can't say.

Beta testing
these were monthly patches (BTW meanwhile McAfee has also been added to the known issues list and they mention a fix for CSRSS as culprit. McAfee also uses the term might occur thus Beta testing might or might not have revealed the issue in time). The software industry is not like the automotive industry. When you buy a car you expect that it contains all the essential elements you need and that the components work together. The software industry promises something similar with SaaS. It doesn't look like there will be more "cooperation" - unless, perhaps, there's pressure from the outside. But I digress.

Putting the lack of communication from Sophos aside, what could be next?

1. I'd expect that Microsoft won't unblock the patches before the involved vendors have a permanent solution (not a workaround like the exclusions) in place. Depending on the actual cause and the importance of the outstanding fixes the patches should deliver they'd either put some pressure on the vendors setting a date when they'll stop blocking the original patches or issue a new set
2. Systems that have the patches installed and Sophos running and updating (because either they are not affected or the exclusions work as intended) should be "safe"
3. Systems where the patches are blocked should also be safe for the moment (see 1.). It's probably not a major security risk that the patches are delayed
4. Systems that are still not working - 133945 has been updated today saying: We plan to start automatically rolling out the fix to customers starting 25th April 2019 and this will take place over a two to three week period. If there are any cases where the update has to be manually applied, we will contact those customers directly. It would be more assuring if there were more details - how to determine that the update has been applied, whether the fix is for all available SAV versions, and what will happen with the Microsoft patches (and when).
   Waiting another two or three weeks is likely unacceptable. The exclusions should have worked (and I had some affected systems and they fortunately recovered with the exclusions in place) but if they don't there should be a way to be put ahead in the roll-out.

Christian

Hi Everyone,

Our team will be rolling out the fix to customers starting 25th April 2019 and this will take place over a two to three week period. If there are any cases where the update has to be manually applied, we will contact those customers directly. The KBA (Overview section) is updated with the Fix release information. Please follow the KBA for more updates.

Hello all,

hm ... starting 25th April 2019 and this will take place over a two to three week period and Note: Microsoft plans to remove this temporary block week commencing 06th May 2019.

Christian

QC said:

Hello all,

hm ... starting 25th April 2019 and this will take place over a two to three week period and Note: Microsoft plans to remove this temporary block week commencing 06th May 2019.

Christian

 

I guess it goes to show that the Microsoft updates are fine as I have said and that the Sophos patch will stop it becoming a problem again. As previously mentioned, if only these companies would work closer together. (Assuming it gets rolled out in time).

Hi Gowtham,

I escalated the support case two days ago, to level two and have heard absolutely nothing since. So, I just called to chase them up and have been told I'll be called back in an hour.

My experience of Sophos's CMS and support processes is a very long way away from what should be expected from a global security company.

Regards,

David.

Hi David, 

I am trying to reach the assigned engineer for your case for further updates. Please allow us some time to get back to you with an update. 

I have also been trying to get a response from them, since last Friday. My emails are have all been ignored, completely. So, I have called them today only to be told that, yesterday, they were told that the only way to get the exclusions into SEC is to run PET with /remediate. Your KB article mentions nothing about this. In fact it clearly states that this is an automatic process. No mention of having to run PET at all.

Quite frankly, I am wholly unimpressed with both your support services as well as the way in which Sophos have handled this whole situation.

It is highly noticeable, in the Spiceworks articles regarding this bug, that Avast! appear to have been not only quicker to respond but have had a more speedy resolution.

I am still waiting to hear back from the level two support engineering duty manager...

Regards,

David.

I can confirm that yesterday's 1809 build update... 17763.439 doesn't change the outcome. Still locks at windows login screen after update restart. (granted its not a windows issue, they just closed a kernel loophole)

Re-imaged my test machine, named it, domained it, made sure sophos was latest one available to our enterprise... 10.8.3.322 (10.8.3.441 will not drop to us yet, no clue why this is taking so long) then installed 17763.439 from downloaded .msu file. Restarted, grabbed sophos SDU log files, restarted again. Locked at windows login screen.

Can't access safe mode without keyboard/mouse control as we are domained here at work.