Multiple computers getting netio.sys BSOD after 2022.2.1.9 update

Now (at 6 weeks since first reported), we are still affected by at least daily BSODs, although spread through different users on different hardware...

Now heavy-heartedly disabling web interception for the most heavily affected users, as I don't see any real progress here...

Not amused!!

Support provided updated sntp.sys from Development-Team. Any experiences on trying this?

Yes, i'm facing this problem on Remotedesktop-Sessionhosts, 64GB RAM each. I got possible fix without providing complete memory dump. Just refer to support.sophos.com/.../KB-000044389

Active dump type would be fine. Being only Active pages it should be quite a bit smaller than 32GB.  It should also zip pretty well.

I just know that in the case I opened with support, they would NOT proceed without a COMPLETE memory dump. 

FFin I now filed our own case and inquired about getting my hands on the "potential fix".
Will keep you guys posted in case I get it and on the outcome...

Thanks for chiming in so far!

Any update on this?

Well, sort of...

We did file our own case on Monday and are now working through the process with a dedicated support representative.

At least for our case I can confirm that support won't hand out the patched driver without FIRST getting at least one full memory dump and the SDU logs. "Active memory dump" will be sufficient, though.

So in the meantime we have set up two of our more heavily affected users with the required settings in order to collect the required type of memory dumps.

Basically we are "waiting for BSODs" now (the same users totalled 5 BSODs together yesterday... but you know the deal with Murphy...).

Once I manage to collect and upload them, they will have a look into that - probably to verify if the case is the same one the patched driver is supposed to address.

Depending on that we'll see how this proceeds...

Thanks for the update. Then we'll have to open a case.

In the meantime we did provide two full memory dumps from two different users and machines this monday.

I did not hear back since, and we still don't have access to the test driver. I assume they are checking if our cases have the same or a similar reason, and if the test driver even addresses that same issue or not.

Also after providing the dumps I enabled the full workaround solution according to the KB for the "known-to-be-affected" users.
Only to learn yesterday, that the workaround - although making the issue less frequent - does not reliably work. One of the users just had another BSOD. Thankfully he was set up to provide memory dumps, so Sophos engineers now got a 3rd full dump with SDU logs for a case that is NOT mitigated by the "turn off security" workaround.

Received the test driver, next step is rolling it out for our most heavily affected users. Keeping fingers crossed!

Hello Samuel, where did you receive the test driver?

Hello Samuel, where did you receive the test driver?

from Support after "paying the bill" - providing a full memory dump

Yeah, we ended up providing three full dumps from two different machines. The last dump was after enabling the full "workaround" for all affected machines...

Anyway today the test driver was rolled out to the first machine and the workaround was disabled for it. Now we will have to wait and see...

Quick update: The results from our testing look promising. The two machines that we are testing the patch on each experienced 4 BSODs in a 3 day period two weeks ago, before enabling the workaround for affected users.

In the last 3 days those two machines produced zero regular BSODs (with the workaround disabled for them, of course).

Unfortunately one machine still had a single BSOD, but after the reboot a pending windows update was installed, and current assumption with Sophos support agent is that background installation of the windows update might have removed the "manually deployed" patched driver - that is, the patched driver might not have been active at that point.

Anyway even if the driver was active, 1 instead of 8 BSODs in a given time frame sounds like a substantial improvement.

Hello

Any update with this testing this patch? 

Not a single BSOD on the two test machines ever since. Other than on unpatched machines... Had to move more users into the BSOD-Mitigation group today.

Our case was seemingly escalated to the "GES" team yesterday morning. Now we're just waiting for feedback.

Hey Samuel,

How have things been going for you with testing? I havent had a chance to do anything other than company-wide implement the "official" workaround and we are still getting occasional BSODs. Not really happy with this whole experience.

We kept our two test users on the test driver and we have not seen any adverse effects.

We got one more netio.sys BSOD for one of those users - but the manually installed test driver seems to get disabled by system protection particularly when Windows updates are being installed, which was the case then -  so that is not necessarily significant. In total we got 2 netio.sys BSODs for two machines since we started testing. The driver is now in daily production use for 37 days for those users, and they used to get anything from 5-10 BSODs per week without the fix.

Last update I got from a "Global Escalation Engineer" (after asking back myself again, of course...):

Thank you for the email. The October release has now been rescheduled for a planned release, on November the 10th 2022.

But of course also some expectation management:

... this is the planned date and can be pulled at the very last minute.

So I keep my fingers crossed that Sophos manages to get this shipped "soon".