248 firewalls - SSD firmware update - This will not be fun

What's the best way for partners to handle this?  Seems like a logistical nightmare.  Most of the firewalls I've connected to in the last two days have the SSD firmware upgrade banner, so I'm going to guess it's a large portion of the installed base.

Performing the upgrade is manual and requires someone SSH to each firewall and run system ssd update.  Then wait while the firewall restarts and hopefully doesn't need to call one of our customers to tell them they need to go into their office after hours to give it a power cycle.

Sure, we can do several at a time, but this is not good.  We have our documentation in ITGlue, so basically the process will be go lookup IP and credentials for customer A firewall, login, update, wait and verify it came back.  Go to customer B, do the same, Go to customer C, do the same, all the way through. 

Takes a lot of time. Going to have to have some place to document who has this done and who doesn't. Nothing in Partner Central until you login to the customer, then to each firewall so no easy way as a partner to look. Thought Central was a place where we can schedule updates but not for this. Why not have it be a hot-fix or firmware update that can be scheduled without all the extra work? Going backwards.

For your end users, not a big deal.  For partners, this is gonna stink. Seems like we get the short end too often.

Edited TAGs
[edited by: emmosophos at 5:31 PM (GMT -8) on 26 Jan 2024]
  • I understand the chance of having to power cycle.  But we already have to power cycle sometimes when doing standard, scheduled firmware updates.  

    Good to know that something's in the works so we don't have to do the one-by-one check and likely update.

    Sophos Firewall Engineer 16.0, 16.5, 17.0, 17.1, 17.5, 18.0, 18.5, 19.0, 19.5, 20.0
    Sophos Firewall Architect 18.0, 18.5, 19.0, 19.5, 20.0
    Sophos Firewall Technician 18.0, 18.5, 19.0, 19.5, 20.0
    Sophos Central & Endpoint Architect 3.0, 4.0
    Sophos Central Email v2.0
    Sophos Mobile v9.6
    Sophos ZTNA 1.0, 2.0
    Synchronized Security Accredited
    Sophos Gold Partner

  • So far I have had 7 XGS requiring the SSD Firmware - 3 went fine 2 Locked up and needed to be unplugged and reset and I have my main two in HA to do this Sunday in a DC. This will take down our State sites .....

    Sadly I am unable to be on site as one in WA, Qld and others in Vic but not close by - all remote sites.

    I had some onsite staff power cycle one and it didn't come back - (he waited 2 minutes before plugging it back in) and it needed a second power cycle to come alive.

    Not much fun to be had here :-(

  • I've done 3 and all of them required a power cycle. I've decided that with 200+ firewalls to manage, I'm just not going to both and skip the SSD Firmware update. After all, the devices have been running fine, in some cases for many years, so why go through all the heartache?

  • I had a 4 work fine without a Hard Reboot and 3 that needed it.

    5 / 7 were stand alone and 2 are in HA - when I did the SSD update on HA Primary the Aux went into error and needed a hard reboot. Then I needed to do the SSD update on the Aux.

    After all that I did the v20 update on both during the planned outage window :-)

    I guess just ensure someone is onsite to give it the reset if needed.

  • In my situation, I've got more than 80 customers with 200+ firewalls. The logistics involved to manually update, and have the customer on-site after hours to nurture the device in case it needs a reboot is astronomical. Again, the devices have been running fine, in some cases for many years, so why go through all the heartache.

    Probably not so much an issue if you've got < 10 devices and 1-2 customers.

  • It is worth pointing out, because I don't think Sophos have done a very good job of communicating this - the patch is really only for those firewalls have have been randomly hanging. If you've had a firewall running for months/years without that happening then this patch is of little benefit. Across our 200+ devices it's only been an issue with 15-20 and most had been manually patched previous by GES.

  • It is not for the problematic firewalls but instead a future improvement to resolve issues with the SSD in the future. 

    But again: I assume there is not "rush" in doing it NOW for all firewalls. You can do it for problematic firewalls now but should think about doing it whenever you have a window. 

    Plenty of partners already "outsourced" this to the Customer themself. They clearly communicating this to the affected customers and what this could imply so the customer can do it themself or with the help of the Partner. The good part is, it is actually done within minutes and if you know, you have to hard reset it, it means the customer has to be in his server room for 15 mins to resolve this by a simply unplugging. 

    As a partner, you also know, there are problematic customers, who do not want to deal with that, so you are not communicating this to them and doing it yourself. 

    But that is what i could see from Partners right now (talking to the bigger MSPs and bigger partners).