XGS Stopped Working (18.5.3)

RE: XGS Stopped Working (18.5.3)

Andreas Hahn1 — Wed, 27 Jul 2022 08:26:53 GMT

We were able to solve our problem, it was the IPv6 DHCP server. The problem has occurred in all versions from 18.5.2

RE: XGS Stopped Working (18.5.3)

JasP — Fri, 06 May 2022 13:42:22 GMT

This site had an XG for a couple of years without any issue. It sat behind a Cisco router. The Cisco router terminated the WAN connections and provided client and site to site VPNs.

We replaced that solution with two XGS in HA and terminated the WAN connections and client and site to site VPNs on the XGS (completely removing the Cisco router). The rest of the config remained the same. Problems started within a week of making the change.

I did mention this thread to our support engineer. He said that the problems were too generic to know if they are the same issue we are experiencing. It is worth bearing this in mind. We may not necessarily all be suffering from the same route cause.

RE: XGS Stopped Working (18.5.3)

LHerzog — Fri, 06 May 2022 12:24:20 GMT

OK, so it is mixed hardware. Too bad.

I remember we had an issue where a specific config change caused XG HA to become unresponsive and at some point did not pass traffic until the HA Aux (yes, the slave node!) node was rebooted.

Is it possible that this issue begins with any config change? Check Admin audit log for changes and then HA logs if they match together.

in our case we could see this starting when we did the config change:

==> /log/ha_tunnel.log <==
Mar 02 18:16:40 ssh: connect to host hapeer port 22: Connection refused

Mar 02 18:16:41 ssh: connect to host hapeer port 22: Connection refused

Mar 02 18:16:42 ssh: connect to host hapeer port 22: Connection refused

Mar 02 18:16:47 ssh: connect to host hapeer port 22: Connection timed out

Mar 02 18:16:52 ssh: connect to host hapeer port 22: Connection timed out

Mar 02 18:16:57 ssh: connect to host hapeer port 22: Connection timed out

RE: XGS Stopped Working (18.5.3)

MCSCARRASCO — Fri, 06 May 2022 12:12:46 GMT

In our case XG, but I read that there are problems with XGS too.

RE: XGS Stopped Working (18.5.3)

Andreas Hahn — Fri, 06 May 2022 12:03:18 GMT

we have XGS

RE: XGS Stopped Working (18.5.3)

LHerzog — Fri, 06 May 2022 11:59:42 GMT

has everyone here XGS machines or also XG machines? I wonder it is an issue with the co-processor.

RE: XGS Stopped Working (18.5.3)

Andreas Hahn — Fri, 06 May 2022 11:49:04 GMT

Thank you for your feedback, since it affects both firewalls from the HA, we will set up a new firewall and test whether it also occurs there

RE: XGS Stopped Working (18.5.3)

JasP — Fri, 06 May 2022 11:45:37 GMT

We are logging a permanent serial console connection to both XGS to see if they can capture a kernel dump when the XGS fails. Waiting for a failure now.

RE: XGS Stopped Working (18.5.3)

MCSCARRASCO — Fri, 06 May 2022 11:42:02 GMT

With us there was the service, analysis and no assertive diagnosis. We are currently operating on secondary equipment that does not have the problem. when possible, we will leave the problematic equipment operating and put a monitoring system to collect logs in an attempt to find the reason for the traffic stop.

RE: XGS Stopped Working (18.5.3)

Andreas Hahn — Fri, 06 May 2022 09:43:55 GMT

Hello everyone, does anyone already have any feedback, with us the support still analyzes the problem...

RE: XGS Stopped Working (18.5.3)

JasP — Wed, 04 May 2022 21:02:15 GMT

We have an open case with Sophos regarding this (Case# 05098782-050987). This has now been escalated all the way through global escalation specialists (GES) to Development (basically as high as it gets) - Development reference number: NC-92066

We've had the issue with 18.5.2, 18.5.3 and 19.0

Pair of XGS 116 in HA. Primary stops passing traffic and HA fails over to Auxillary. If you reboot the failed XGS, the HA is restored. If you don't reboot the failed XGS, the new Primary will eventually fail, leaving you with no internet connectivity.

I would suggest those affected open a case with Sophos and reference our case number and the development reference number. If you already have a case I would suggest you pass our case details on to your current support specialist.

RE: XGS Stopped Working (18.5.3)

Andreas Hahn — Tue, 03 May 2022 21:56:06 GMT

Hi everyone, we have the same problem. We have also updated from 18.5.2 to version 18.5.3. Also another update to version 19 did not help the problem persists. We have also already opened a support ticket. (We have also dissolved the HA, this has not solved the problem either.)

RE: XGS Stopped Working (18.5.3)

Peter Mastrangelo — Tue, 03 May 2022 15:34:49 GMT

Update 2022-05-03

We spoke too soon that the rollback to 18.5.2 appeared to correct the issue. We had the issue again today.

It might be just coincidence that this issue started appearing after the 18.5.3 update. XGS was rock solid for months.

We opened a support ticket.

-Peter Mastrangelo

RE: XGS Stopped Working (18.5.3)

MCSCARRASCO — Tue, 03 May 2022 12:12:04 GMT

I have the same problem in the company. Updated XG for SFOS 18.5.3 MR-3-Build408. Equipment in Active x Passive HA. Simply 1 stops traffic and only returns after restarting. We are currently operating on the secondary XG and there have been no more problems. We have an open call with Sophos to try to identify the cause.

RE: XGS Stopped Working (18.5.3)

Peter Mastrangelo — Tue, 03 May 2022 10:29:00 GMT

Thank You for the answer.

Unfortunately the issue happened again. We had to quickly "switch to passive" because we are a K12 school dependent on Chromebooks.

I rolled back the firmware to 18.5.2 using this as a guide https://support.sophos.com/support/s/article/KB-000038017?language=en_US

The issue has not appeared again \ yet.

I did pull the logs off the device(s) and have been looking through them but haven't found anything yet.

We will be holding off any firmware updates 18.5.3 or possibly 19 until the end of the school year.

Thank You,

-Peter Mastrangelo

RE: XGS Stopped Working (18.5.3)

Excommunicado — Tue, 03 May 2022 03:50:32 GMT

Hi Peter Mastrangelo,

Did you collect logs from the firewall in a broken state? Did it reboot properly after the firmware update? Did you check the uptime or any service is down?

You might find the root cause in the system or applog log files.