Sophos Firewall: v19.0 GA: Feedback and experiences

https://support.sophos.com/support/s/article/KB-000044122?language=en_US

Dead firewalls everywhere after scheduling FW updates after the first 20 appliances upgraded without issue. Why can't the upgrade process detect that the certificate is not valid and advise instead of proceeding to kill the unit completely?

The problem was discovered recently and only affects Cyberoam Appliances or old installations coming from Cyberoam.

Sophos always recommend to read the Release notes carefully: https://docs.sophos.com/releasenotes/index.html?productGroupID=nsg&productID=xg&versionID=19.0

The Upgrade and known issue section is explaining the issue. 

This will be addressed in the next version.

One option is to put it in the release notes and rely of the user reading them before the upgrade. Fair for most people on this forum, we're a bit geeky, not sure about the standard business user who just wants to upgrade their firmware, and receives no prompt on the GUI interface to read the release notes first. Also bear in mind Sophos often push out emails saying "Make sure you're running the latest firmware or you could be at risk / unsupported"

Another option may be a re-release of the existing v19 with a built in check - (if coming from Cyberoam, do not apply, and prompt with message) or equivalent.

I guess what I'm implying is, lets not try to move too much of the emphasis to the end user when trying to offer a user friendly product.

It is not that easy to re release a product / firmware from such perspective. There are many implication on doing this. Not only technical, legal etc. 

ahhh I see, okay cool

Ryan summaries my concerns better than I have.

For us, we manage over 100 Sophos units. We upgraded 1 without issue, waited a few weeks, then did another 5 and again no issues, so push out the update to the rest of the units centrally. Then all of a sudden we had 12 clients dead in the water with no internet access. And because upgrades to critical infrastructure has to happen out of hours, the failures occurred at night and we have to wait until the morning when suddenly 12 different units are down and staff can't work. It's fine for an MSP to perform the steps to rectify, but you can't get a client to log in to advanced shell and run those commands.

I get the issues with reissuing firmware to resolve a bug, but what I don't understand is that if the Sophos knows it's doing a firmware update and then it ends up in Failsafe mode, it clearly knows the process has failed. Why can't it mark that FW image as faulty so that next the time appliance boots it goes back to the known good FW image? That way we could just tell the client to power cycle the unit and they'd be back online and log in remotely to fix. This was one of my biggest issues with Cyberoam previously as well. If a FW fails to boot, next time boot to the other one. That way the client is not dead in the water.

Ryan summaries my concerns better than I have.

For us, we manage over 100 Sophos units. We upgraded 1 without issue, waited a few weeks, then did another 5 and again no issues, so push out the update to the rest of the units centrally. Then all of a sudden we had 12 clients dead in the water with no internet access. And because upgrades to critical infrastructure has to happen out of hours, the failures occurred at night and we have to wait until the morning when suddenly 12 different units are down and staff can't work. It's fine for an MSP to perform the steps to rectify, but you can't get a client to log in to advanced shell and run those commands.

I get the issues with reissuing firmware to resolve a bug, but what I don't understand is that if the Sophos knows it's doing a firmware update and then it ends up in Failsafe mode, it clearly knows the process has failed. Why can't it mark that FW image as faulty so that next the time appliance boots it goes back to the known good FW image? That way we could just tell the client to power cycle the unit and they'd be back online and log in remotely to fix. This was one of my biggest issues with Cyberoam previously as well. If a FW fails to boot, next time boot to the other one. That way the client is not dead in the water.

First of all, a SIG/ISO build is a finish state. You are using a build XYZ version. In this version, the issue was not discovered. So to "flag this as a failed image" the code is not there to flag in, as the issue was not known to the firewall at this point.

Sophos could revoke the entire build. This means pull back build XYZ for all customers. Re release a new version (build ABC with the fix). This takes a long time to go through all cycles (QA checks, legal, release infrastructure etc.). 

Failsafe is some issue, which is actually quite rare in the latest releases. I guess the last time i saw failsafe was 2 years ago. So this is not a common issue. If the migration does not work, it can pull back to the old version. But fail safe is some sort of kernel panic mode. So the entire firewall stopped at this point. 

There is a lot of background work currently to get a state of this cannot happen again. 

And again, we do not talk about "every customer is affected". It seems like only a customer base with a old configuration coming from cyberoam. 

I fully understand you can't revoke the entire build, and I understand that the issue wasn't known when the firmware was released, but what I'm saying is that if the Sophos does an upgrade and the device enters "failsafe" then when entering "failsafe mode" it should automatically mark the alternative image as default for the next boot of the device. That way, if a firmware update fails and breaks the device - for whatever reason - we can just tell the customer to power cycle the device and it will boot back in to the previous firmware.