This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

[Sophos Notification] Sophos Central Endpoint and SEC: Computers fail/hang on boot after the Microsoft Windows April 9, 2019 update

Hi Everyone,

After installing the following Microsoft Windows updates Sophos has received reports of computers failing to boot:

The issue is currently being investigated. For more updates and workaround, please follow the below KBA.

Following the Microsoft Windows 09th April update computers fail/hang on boot



This thread was automatically locked due to age.
Parents Reply Children
  • Hello Sammore,

    it looks - from the rather prompt reaction of Microsoft - like this is not an issue for which Sophos has to bear the undivided blame (or even the major part of it). Microsoft wouldn't block their patches if some other vendor has "self-inflicted" problems.

    Christian

  • To be a little fairer to Microsoft, I feel that both Microsoft and Sophos should be working closer together when OS changes such as the new patches are implemented. From what I have seen so far, if you are not running Sophos, Avast or Bit Defender then you were unaffected by problems with the updates that were released.

    The files that were patched, more specifically WININIT.dll and PRINTUI.dll, appear to have a new file signature. Had Sophos been made aware of these changes (If they weren't, I am speculating here) then I am sure their Dev team could have rolled an update out to account for these changes and prevent the problems being experienced.

    There is actually nothing wrong with the updates released by Microsoft. It's just simply a case that in order for them to work without issue, 3rd party vendors need to be aware of the changes and have patches in place for their products prior to release.

  • I'm not interested in who bares the blame, I'm interested in a solution that resolves the issue at hand and helps us avoid these issues in the future.

     

    Right now we're at the mercy of Sophos to release a fix, whose track record in fixing these issues is slow to say the least.

  • Hello Richie Knight,

    [this is not finger-pointing]
    it's not that simple. Show me a vendor or a community that has never issued a patch that, although carefully tested, had unexpected and unwanted side-effects. Or at least none that affected only third-parties but not their (one of their) own product(s). working closer together is a noble concept but it's already hard to accomplish it "in-house".

    need to be aware of the changes
    How? In case of source-code patches you can theoretically asses the changes and their consequences. Or at least find the cause for an issue. In practice you need very good knowledge of the patched component. With binary, closed-source patches you have to rely on the information you get - unlikely that a vendor will disclose full details. Having access to patches in advance helps a little but not much. You'd detect issues that have sufficient prevalence - provided there isn't an additional factor that is present only in certain installations. Furthermore, the third-parties also need some time to have [their] patches in place.
    And we're talking about documented standard interfaces. Of course it's possible that changes "inside the black box" cause something to break - but normally this should not be the case. And if a race-condition is involved it might be hard to debug. The vendor is not aware of significant changes, the third-party not aware of any "hacks", and as soon as you attach a debugger or subscribe to events the problem goes away.

    if you are not running Sophos, Avast or Bit Defender
    There's more than one way to skin a cat. Meaning: AFAIK no OS, filesystem or filesystem implementation provides an AV interface with corresponding semantics (and consequently there aren't AV-aware applications). I've never investigated why. Naturally an AV should do its work as early as feasible and with minimum impact on startup. Ideally it should also perform a self-check.  There's no pre-defined or recommended strategy though and thus no either-all-or-none.

    a new file signature
    as the sole cause is unlikely. It's not only user-space applications that are patched, system files change regularly.

    Not even a closed ecosystem can guarantee that you'll be safe from such issues.

    Christian

  • Hi Christian.

    QC said:

    [this is not finger-pointing]
    it's not that simple. Show me a vendor or a community that has never issued a patch that, although carefully tested, had unexpected and unwanted side-effects. Or at least none that affected only third-parties but not their (one of their) own product(s). working closer together is a noble concept but it's already hard to accomplish it "in-house".

    need to be aware of the changes
    How? In case of source-code patches you can theoretically asses the changes and their consequences. Or at least find the cause for an issue. In practice you need very good knowledge of the patched component. With binary, closed-source patches you have to rely on the information you get - unlikely that a vendor will disclose full details. Having access to patches in advance helps a little but not much. You'd detect issues that have sufficient prevalence - provided there isn't an additional factor that is present only in certain installations. Furthermore, the third-parties also need some time to have [their] patches in place.
    And we're talking about documented standard interfaces. Of course it's possible that changes "inside the black box" cause something to break - but normally this should not be the case. And if a race-condition is involved it might be hard to debug. The vendor is not aware of significant changes, the third-party not aware of any "hacks", and as soon as you attach a debugger or subscribe to events the problem goes away.

    if you are not running Sophos, Avast or Bit Defender
    There's more than one way to skin a cat. Meaning: AFAIK no OS, filesystem or filesystem implementation provides an AV interface with corresponding semantics (and consequently there aren't AV-aware applications). I've never investigated why. Naturally an AV should do its work as early as feasible and with minimum impact on startup. Ideally it should also perform a self-check.  There's no pre-defined or recommended strategy though and thus no either-all-or-none.

    a new file signature
    as the sole cause is unlikely. It's not only user-space applications that are patched, system files change regularly.

    Not even a closed ecosystem can guarantee that you'll be safe from such issues.

    Christian

     
    Thanks for your reply. I am merely commenting on what I have found so far.
     
    Working closer together is really something that should be high on large vendors minds such as Sophos and Microsoft in my opinion. (I honestly don't know to what extent this happens at the moment but your post sheds a little more light on the relationship). If this is even a problem in-house then internal policies need to be looked at. Quality testing is something I am keen on seeing implemented more especially given the environment we live and work in today, the large importance given to security and our reliance upon our systems needing to be functioning. I know this is something that has been somewhat lacking by Microsoft in recent months/years although they did release a press statement days prior to these latest patches saying that they fully intend on implementing better fast ring testing of new patches in the coming months. (Woohooo)!
     
    The files that were patched by Microsoft and listed in the change logs in this instance are vital to the system in which they are installed on. (WININET.dll being one of them). A breakdown of what this dll is used for can be found here... https://docs.microsoft.com/en-us/windows/desktop/wininet/wininet-reference 
     
    What surprises me is how such an important component in Windows used for networking and internet access can be patched without any consideration as to the ramifications this is going to have for security vendors such as Sophos. I'm assuming that in this instance, Microsoft deemed the patching of these files as inconsequential to security vendors as the method in which security vendors should be monitoring such system files would be non-destructive. (Otherwise they would notify you surely?) Do Microsoft provide vendors such as Sophos any baselines in terms of how they should be interfacing with such files and do Sophos adhere to such recommendations should they exist?
     
    Looking at the errors generated by many of our SEC's and cross referencing them against what was patched by Microsoft and logs found on affected systems, am I not right in thinking that if Sophos was not protecting Windows against a changing WININET.dll these problems would never have happened with these new patches? So far my findings would suggest that this would have been the case as many non-Sophos systems work well with the patches Microsoft pulled. (I am not trying to lay blame here, I just feel that in moving forward a better approach needs to be implemented by Sophos security products for what happens should these files get patched).
     
    Hopefully the fast ring testing Microsoft has promised for future CU's will help prevent these types of problems on a large scale.
     
    Richie.
  • Hello Richie,

    just to make sure, I'm not Sophos, not a partner, and I have no inside knowledge (just a long time in IT).

    Systems are too complex. Most of the time issues are confined to certain configurations and combinations of HW and SW so the general public never really hears of them. There's not one what if, there are very many. We are more or less used to "minor" problems. 200 computers, same model, identical software environment - three refuse a certain patch. Either you can live with it or it's reinstall time. If you can live with it - then you have different configurations and over time no longer a homogeneous environment. Which computers would you select as test base?

    Christian

  • Thanks for the clarity Christian.

    Much like yourself, I too have been working in I.T. for a long time. (19 years in fact. Now I feel old haha). I work for a leading MSP, (no names mentioned) who support many thousands of systems. We are a Sophos platinum partner (one of the biggest) and will need to be seeking further reassurances regarding these problems in moving forward.

    As I am sure you can imagine, when problems such as these occur, we have a major headache and few ways to cover the cost and time in cleanup. (Which can be extremely high). The longer these issues continue, the more costs are incurred and the bigger the strain on our client relations which ultimately have an affect on Sophos in terms of our buying power.

    Beta testing is a must in my opinion to try and help minimize these types of incidents. I fully understand problems as a result of patching but the severity of the issues caused in this instance is simply not acceptable.

    Richie.

  • Hello Richie,

    19 years
    almost twice this ... I'm not feeling I am ;).
    platinum partner
    thought you'd get more information than us mere customers.

    problems such as these
    are indeed more than a pain.
    As said, I'm just inferring from a few bits of information (I didn't even "consult the internet" or "asked Mr. Google"). Initially I became aware of the problem because a server admin reported an issue after applying the latest Microsoft patches. Didn't even mention Sophos but the (SEC) console showed an updating error, I checked the Sophos knowledgebase and found 133945. Not sure if the Microsoft articles already mentioned Sophos under Known issues, I think not. Anyway, I'm not aware that Sophos has a close relation with Microsoft. That Microsoft quickly blocked the patches was (IMO) a sign of, err, consciousness of potential guilt. Therefore I'm quite sure that Sophos didn't have a chance to proactively work on it. As you say, it might have been something that was deemed inconsequential - whether bona fide or due to negligence I can't say.

    Beta testing
    these were monthly patches (BTW meanwhile McAfee has also been added to the known issues list and they mention a fix for CSRSS as culprit. McAfee also uses the term might occur thus Beta testing might or might not have revealed the issue in time). The software industry is not like the automotive industry. When you buy a car you expect that it contains all the essential elements you need and that the components work together. The software industry promises something similar with SaaS. It doesn't look like there will be more "cooperation" - unless, perhaps, there's pressure from the outside. But I digress.

    Putting the lack of communication from Sophos aside, what could be next?

    1. I'd expect that Microsoft won't unblock the patches before the involved vendors have a permanent solution (not a workaround like the exclusions) in place. Depending on the actual cause and the importance of the outstanding fixes the patches should deliver they'd either put some pressure on the vendors setting a date when they'll stop blocking the original patches or issue a new set
    2. Systems that have the patches installed and Sophos running and updating (because either they are not affected or the exclusions work as intended) should be "safe"
    3. Systems where the patches are blocked should also be safe for the moment (see 1.). It's probably not a major security risk that the patches are delayed
    4. Systems that are still not working - 133945 has been updated today saying: We plan to start automatically rolling out the fix to customers starting 25th April 2019 and this will take place over a two to three week period. If there are any cases where the update has to be manually applied, we will contact those customers directly. It would be more assuring if there were more details - how to determine that the update has been applied, whether the fix is for all available SAV versions, and what will happen with the Microsoft patches (and when).
      Waiting another two or three weeks is likely unacceptable. The exclusions should have worked (and I had some affected systems and they fortunately recovered with the exclusions in place) but if they don't there should be a way to be put ahead in the roll-out.

    Christian

  • Hello all,

    hm ... starting 25th April 2019 and this will take place over a two to three week period and Note: Microsoft plans to remove this temporary block week commencing 06th May 2019.

    Christian

  • QC said:

    Hello all,

    hm ... starting 25th April 2019 and this will take place over a two to three week period and Note: Microsoft plans to remove this temporary block week commencing 06th May 2019.

    Christian

     

    I guess it goes to show that the Microsoft updates are fine as I have said and that the Sophos patch will stop it becoming a problem again. As previously mentioned, if only these companies would work closer together. (Assuming it gets rolled out in time).