This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

UTM 9.601 - RED issues!

Since upgrading all our customers to 9.601, a bigger part of them are complaining about RED's re/disconnection in a no-pattern way.

It started for all of them just the night we upgraded to 9.601, and they all are on different ISP's and located different places around the country.

Been with Sophos support for 2 hours today, and now they escalated it to higher grounds.

Will return with an update....

Suspicious entries in the log - but all connected REDs do this before connection:

2019:03:06-15:15:38 fw01-2 red_server[17509]: SELF: Cannot do SSL handshake on socket accept from 'xxx.xxx.xxx.xxx': SSL connect accept failed because of handshake problems

2019:03:06-15:15:46 fw01-2 red2ctl[12420]: Missing keepalive from reds3:0, disabling peer xxx.xxx.xxx.xxx

I know the last line is written before the tunnel disconnects, because there was no "PING/PONG" answer...

One customer has 2 x RD 50, one 1 100% stable and the other fluctuates in random intervals - we replaced this with a new RED 50, but the same thing occurs.



This thread was automatically locked due to age.
Parents
  • Same issues here after 9.601-5 UTM update. 2x RED50 Rev 1. Drop multiple ISPs at varying intervals and lengths. It was advised to re-create RED in UTM. I have performed this, but problems still persist. I was sent two replacement RED50. The first one has been replaced, a new config created, but problem persists. ISPs modems have been replaced although they were reluctant to do so. One of the REDs wont recognize the presence of ISP on WAN1 at all.

    We are losing a lot of productivity and business. We do a sizeable portion of our business via teleconferencing.

    Support Tickets#

    8710435

    8707203

    8707207

     

    The tech alluded to a potential issue with REDs after the update to 9.6.01-5.

  • My problem is resolved. There is a known issue related to unified firmware.

    from su -

    cc get red use_unified_firmware

    if value returned = 1

    cc set red use_unified_firmware 0

    reds will update and reboot

    confirm value is 0 rerunning get command above

     

    NOT A PERMANENT FIX. The issue needs to be addressed in Sophos UTM firmware permanently.

  • Hallo somi and welcome to the UTM Community!

    Based on what others have said above, I would push Sophos Support to RMA the failing RED.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Hey Bob,

    thanks, we already replaced that RED with a brand new RED 15 - same thing.

    Regards,

    Michael

  • ...it seems to be a timing issue... When i disable the RED for 5 minutes on the UTM it works fine shortly after reenabling it again.

    No 'stabelizing peers' and such in the log - just 'tunnel -up' -> PING PONG, PING PONG

    Tunnel is stable until the DSL-Line does its 24h reconnect then.
    This brings it out of 'sync'...

    Then until i disable and reenable the RED i can see -> boot -> stabelizing peers -> 5 ICMP Packets go through the tunnel (yay!) -> unstable peers -> reboot (oh no!)
    ... over and over again ...

  • Hi all,

     

    Sorry for the delay, been on vacation for a week - my nerves cound not stand it anymore ;-)

    I also did the "cc set red use_unified_firmware 0" before I left, and can confirm it solved ALL MY ISSUES.

    Had one customer with two RED 50s, one was very unstable and another was completely offline, we have setup temporary SG115's with IPSEC just to keep the customer running.

    After I have disabled the new unified firmware, both RED 50's are back and 100% stable!

    Sophos Support claims that there are no issues with this, but please, keep refering to this community string, so they can see, that there actually are problems.

    I have enabled RED debugging with suppoort, and inserted USB key for debug logging into the red 50, but nothing important was shown.

    We have the unified firmare enabled with several other customers, which have no issues with it, so it's odd, I think it looks like some ttl, ips issues, with the different ISP.

    EDIT:

    Some other issues have been located, and it seems like Sophos it looking into it:

    community.sophos.com/.../9-601-5-update-killed-red-50-home-site-dns-resolution

    -----

    Best regards
    Martin

    Sophos XGS 2100 @ Home | Sophos v20 Architect

  • I have had the same issues described.

     

    I was issued an RMA and got a new RED50 box.

     

    it bricked again within 12 hours.

  • Hello Bob,

    i am trying to connect a RED 15 to our UTM with 9.6.01 Firmware (SG115) the first time now and get the same error now:

    SELF: Cannot do SSL handshake on socket accept from 'xxx.xxx.xxx.xxx': SSL connect accept failed because of handshake problems.

    Do you know when Sophos will fix this and release a new Firmware fot UTM and RED?

    Regards, Reinhold

  • Hallo Reinhold,

    Did you try James Stoy's solution above - switch to MTU1400 and disable/enable the RED?  If that didn't work Sophos may want to RMA the 15.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Hello Bob,

    yes, not i tried it out, MTU 1400

    Its a different error in the log now, but it does stikll no connect:

    2019:04:26-08:27:33 fw red_server[4454]: SELF: (Re-)loading device configurations
    2019:04:26-08:27:36 fw red2ctl[4383]: Overflow happened on reds1:0
    2019:04:26-08:27:36 fw red2ctl[4383]: Missing keepalive from reds1:0, disabling peer  xxx.xxx.xxx.xxx
    2019:04:26-08:28:41 fw red_server[8474]: SELF: New connection from xxx.xxx.xxx.xxx with ID XXXXXXXXXXXXXX (cipher AES256-GCM-SHA384), rev1<30>Apr 26 08:28:41 red_server[8474]: XXXXXXXXXXXXXX: connected OK, pushing config
    2019:04:26-08:28:45 fw red_server[8474]: XXXXXXXXXXXXXX: command 'CON_CLOSE reason=fallback_config'
    2019:04:26-08:28:45 fw red_server[8474]: id="4202" severity="info" sys="System" sub="RED" name="RED Tunnel Down" red_id="XXXXXXXXXXXXXX" forced="1"
    2019:04:26-08:28:45 fw red_server[8474]: XXXXXXXXXXXXXX is disconnected.

    I have an open ticket at the german Sophos premium support, but still not solution.

    Need to get it running for a homeoffice start in May.

    Regards, Reinhold

  • Hi Reinhold,

    Have you tried the "disable unified firmware" fix yet? Most users who have tried the MTU 1400 fix state that disabling the unified firmware works for them.

     

    See the answer from  for more details.

    Let us know, if not I'm afraid it is an RMA.

    Kind Regards,

    James.

  • Hi James,

    yes, the Sophos Support did this first, but it did not help.

    It's very strange for me that the Sophos (Premium) Support does not know an anser and does not help me in any way.

    Looks like UTM-RED15 is a non working combination and they prefer XG.

    Regards, Reinhold

Reply
  • Hi James,

    yes, the Sophos Support did this first, but it did not help.

    It's very strange for me that the Sophos (Premium) Support does not know an anser and does not help me in any way.

    Looks like UTM-RED15 is a non working combination and they prefer XG.

    Regards, Reinhold

Children
  • Reinhold, Premium support just means (in Europe) that you can contact Support directly instead of having to go through the Sophos Partner you purchased from.  You still get the lowest level of support personnel to start with.  Thousands of RED 15s are working well with UTMs.  You will want to request escalation.

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  • Hi,

    yes, i know, it sounds better than it is.

    In the meantime the Support did point me to the new UTM Firmware 9.602-3, released tonight on the ftp site.

    But it does not help. Now its escalated.

    Regards, Reinhold

  • Hi,

    still no solution, RED15 does not connect to SG115 with Firmware 9.602-3, it trys to connect and at the end says "is disconnected"

    Sophos escalted it to special team and they work on a fix for this new bug. No infor when it will be available

    Too bad that this happens right now, i need a Homeoffice VPN for an employee since 1rst May.

    Regards, Reinhold

  • Hi,

     

    have you tried with unified firmware = 0 just to see if it's the issue or you are facing other problems with the RED?

     

    My issues with RED 50's have been solved with 9.602 AND unified firmware = 1

    -----

    Best regards
    Martin

    Sophos XGS 2100 @ Home | Sophos v20 Architect

  • Workaround for me: Just use 1x tunnel between Red and SGxxx - in redundant configuration we had the same problem. But it depends of the ISP on the site of REDs. With some ISPs we can use redundant tunnels, with other ISP we had to use just only one tunnel!

    good luck

  • BAlfson said:

    Reinhold, Premium support just means (in Europe) that you can contact Support directly instead of having to go through the Sophos Partner you purchased from.  You still get the lowest level of support personnel to start with. 

     

     

    Sadly, I can confirm this. Wish I had known before recommending that our customers here (I work for an MSP) purchase the premium option. They have paid steep fees for it and are not being taken care of very well. In this particular case of REDs disconnecting after 9.601, I initially called German support and was also told it's a known firmware issue, no fix date confirmed. The tech had me switch on the firewall's support access, but no one actually connected to it. He wrote to me a few days later and enclosed the procedure of how to manually reset the system using a config backup on a USB stick... Dankeschön und auf Wiedersehen! If I absolutely have to call in future, it will be after German business hours, when calls get routed to the US.

    Luckily there were already quite a few threads on the topic in this forum. In spite of 1000's of working REDs, it looks like a not insignificant number of installations have been affected by the issue. For us, the sure fix has been to deactivate the Red, set interface to MTU 1400, and reactivate. Thanks to whoever came up with that! 

  • __________________________________________________________________________________________________________________

  • LuCar Toni,

    I appreciate Sophos finally acknowledging the issue. We have had 2 REDs RMA'ed. Thanks for supporting us.

    One thing that would help this situation is to allow the "cc set red use_unified_firmware 0" flag change to be persistent on a further firmware upgrades once the UTM is in the 9.6 branch. As it is, this flag is reset on any further update. This was the case for me when I upgraded to 9.603001 from 9.601-5. Previously I had changed the flag to =0 but the UTM set it back to =1 during the upgrade and my new REDs reverted back to the unified firmware. This made me very nervous that I would have two more "bricks" at anytime so I reset the flag back to 0 again. Luckily the units do not "brick" during that process!!

    I can't go through the workaround of disconnecting REDs every time I need to upgrade! I upgrade firmware during late night maintenance windows (as most admins do) so there is no one at the remote office to disconnect the RED! 

    Thanks.

  • Garth's is a good question, Toni.  Do we know if it suffices to disable the RED in the UTM before Up2Dating and then enable it after having disabled unified firmware?

    Unless I'm wrong, the KB should be modified to reflect the fact that use_unified_firmware must be applied separately to each UTM in High Availability.

    NOTE about one day later: see my post below

    Cheers - Bob

     
    Sophos UTM Community Moderator
    Sophos Certified Architect - UTM
    Sophos Certified Engineer - XG
    Gold Solution Partner since 2005
    MediaSoft, Inc. USA
  •  can correct me with more information, but as far as i know, you need to set this CC switch on the master and it should be synced to the other Appliance. (like all CC switches). 

    Also this switch should be "off" after a Firmware Update. But you should check it after the firmware Update. 

     

    Sophos slowly increased the switch in the last updates.

    Currently, the unified firmware will only be applied to installations with fewer than 20 RED devices configured.

    __________________________________________________________________________________________________________________