248 firewalls - SSD firmware update - This will not be fun

I understand the chance of having to power cycle.  But we already have to power cycle sometimes when doing standard, scheduled firmware updates.  

Good to know that something's in the works so we don't have to do the one-by-one check and likely update.

So far I have had 7 XGS requiring the SSD Firmware - 3 went fine 2 Locked up and needed to be unplugged and reset and I have my main two in HA to do this Sunday in a DC. This will take down our State sites .....

Sadly I am unable to be on site as one in WA, Qld and others in Vic but not close by - all remote sites.

I had some onsite staff power cycle one and it didn't come back - (he waited 2 minutes before plugging it back in) and it needed a second power cycle to come alive.

Not much fun to be had here :-(

I've done 3 and all of them required a power cycle. I've decided that with 200+ firewalls to manage, I'm just not going to both and skip the SSD Firmware update. After all, the devices have been running fine, in some cases for many years, so why go through all the heartache?

I had a 4 work fine without a Hard Reboot and 3 that needed it.

5 / 7 were stand alone and 2 are in HA - when I did the SSD update on HA Primary the Aux went into error and needed a hard reboot. Then I needed to do the SSD update on the Aux.

After all that I did the v20 update on both during the planned outage window :-)

I guess just ensure someone is onsite to give it the reset if needed.

In my situation, I've got more than 80 customers with 200+ firewalls. The logistics involved to manually update, and have the customer on-site after hours to nurture the device in case it needs a reboot is astronomical. Again, the devices have been running fine, in some cases for many years, so why go through all the heartache.

Probably not so much an issue if you've got < 10 devices and 1-2 customers.

So very true - I don't envy 200 :-)

It is worth pointing out, because I don't think Sophos have done a very good job of communicating this - the patch is really only for those firewalls have have been randomly hanging. If you've had a firewall running for months/years without that happening then this patch is of little benefit. Across our 200+ devices it's only been an issue with 15-20 and most had been manually patched previous by GES.

It is not for the problematic firewalls but instead a future improvement to resolve issues with the SSD in the future. 

But again: I assume there is not "rush" in doing it NOW for all firewalls. You can do it for problematic firewalls now but should think about doing it whenever you have a window. 

Plenty of partners already "outsourced" this to the Customer themself. They clearly communicating this to the affected customers and what this could imply so the customer can do it themself or with the help of the Partner. The good part is, it is actually done within minutes and if you know, you have to hard reset it, it means the customer has to be in his server room for 15 mins to resolve this by a simply unplugging. 

As a partner, you also know, there are problematic customers, who do not want to deal with that, so you are not communicating this to them and doing it yourself. 

But that is what i could see from Partners right now (talking to the bigger MSPs and bigger partners). 

3 didn't come up out of 10 updated last night.  Power cycled (pulled power for 2 minutes) one of them twice, took 30 minutes to come to life.  

4 out of 10 failed to boot up.  2 out of them failed to boot on the first attempt to remove power, but came up after the 2nd try.