Sophos Firewall: v19.5 MR2: Feedback and experiences

The day before this update came out we updated a customer from 19.0.x (19.0.2 i think) to 19.5.1, and it failed spectacularly. It seems to start up just fine but then as soon as the HA Auxiliary device joins it all comes crashing down. The Primary can ping maybe 1 in 3 devices on the network. We can connect remotely sometimes. Forcing a HA failover doesn't fix anything.

In the release notes of 19.5.2 I see that NC-115019 refers to "Primary device in HA becomes unresponsive.", which vaguely matches our issue, but could also refer to a completely different issue (eg that the primary freezes after some number of days). How can I find out when that bug started? That NC number is not mentioned in the known issues list, maybe because it is now resolved? If the bug was created in 19.5.[01] then that could be the cause. If not I guess i'll raise a case with Sophos support, but doing that is such a pain

I had that happen too during one of our HA upgrades. We had to remove and re-create the HA connection from scratch to get it to work. The secondary upgraded fine and took over but the primary was unresponsive, and had to be physically restarted to even get into it's GUI or SSH. Once we did that, we were able to go into the GUI, remove the HA completely from both, upgrade the primary, then re-create the HA.

Frustrating endeavor because it was an active-backup HA, and the secondary didn't take over the license in the re-creation process. Had to remotely add an evaluation license for the necessary firewall features for it to function properly while we did the work. Hopefully it doesn't happen again since that serial number has now burned it's evaluations, but we do also have the option of adding a 1-month MSP license to it through Sophos Central Partner in that case.

I think that the primary firewall should have another management IP address in case of these kinds of issues, separate from the regular address that they both take together when in HA, much like the secondary has. But it wouldn't have mattered in this particular issue as the only access available was console until it was rebooted. I'll need to set up OOB console access for these kinds of deployments going forward, or at least OOB power outlet access to force reboots. But I do hate shutting down non-gracefully, as that occasionally corrupts the PostgreSQL database.

I wonder if the hardware could include a small battery (similar to RAID controller battery size) that does a graceful shutdown when no AC power is found? But that's a huge ask, as it's not an easy thing to implement without possible false-alarm shutdown events, and would have to monitor at the hardware level to be reliable, not the software level. Maybe the higher end versions have this? I was using an XGS 2100 if I remember correctly, so I wouldn't be surprised if bigger models have it, but I haven't looked.

I had that happen too during one of our HA upgrades. We had to remove and re-create the HA connection from scratch to get it to work. The secondary upgraded fine and took over but the primary was unresponsive, and had to be physically restarted to even get into it's GUI or SSH. Once we did that, we were able to go into the GUI, remove the HA completely from both, upgrade the primary, then re-create the HA.

Frustrating endeavor because it was an active-backup HA, and the secondary didn't take over the license in the re-creation process. Had to remotely add an evaluation license for the necessary firewall features for it to function properly while we did the work. Hopefully it doesn't happen again since that serial number has now burned it's evaluations, but we do also have the option of adding a 1-month MSP license to it through Sophos Central Partner in that case.

I think that the primary firewall should have another management IP address in case of these kinds of issues, separate from the regular address that they both take together when in HA, much like the secondary has. But it wouldn't have mattered in this particular issue as the only access available was console until it was rebooted. I'll need to set up OOB console access for these kinds of deployments going forward, or at least OOB power outlet access to force reboots. But I do hate shutting down non-gracefully, as that occasionally corrupts the PostgreSQL database.

I wonder if the hardware could include a small battery (similar to RAID controller battery size) that does a graceful shutdown when no AC power is found? But that's a huge ask, as it's not an easy thing to implement without possible false-alarm shutdown events, and would have to monitor at the hardware level to be reliable, not the software level. Maybe the higher end versions have this? I was using an XGS 2100 if I remember correctly, so I wouldn't be surprised if bigger models have it, but I haven't looked.