UTM 9.601 - RED issues!

Since upgrading all our customers to 9.601, a bigger part of them are complaining about RED's re/disconnection in a no-pattern way.

It started for all of them just the night we upgraded to 9.601, and they all are on different ISP's and located different places around the country.

Been with Sophos support for 2 hours today, and now they escalated it to higher grounds.

Will return with an update....

Suspicious entries in the log - but all connected REDs do this before connection:

2019:03:06-15:15:38 fw01-2 red_server[17509]: SELF: Cannot do SSL handshake on socket accept from 'xxx.xxx.xxx.xxx': SSL connect accept failed because of handshake problems

2019:03:06-15:15:46 fw01-2 red2ctl[12420]: Missing keepalive from reds3:0, disabling peer xxx.xxx.xxx.xxx

I know the last line is written before the tunnel disconnects, because there was no "PING/PONG" answer...

One customer has 2 x RD 50, one 1 100% stable and the other fluctuates in random intervals - we replaced this with a new RED 50, but the same thing occurs.

  • In reply to Fabio Giacobbe:

    Fabio Giacobbe

    Hi Jan,

     

    but is possible to start an RMA procedure without a maintenance contract?

     

    thanks

     

    fabio

     

    When having license for reds (network protection), you should be covered ;-)

  • In reply to JanWeber:

    Jan, I think your first B. should be to 9.604, not 9.605.  See my post above and my latest PM to you.

    Cheers - Bob

  • In reply to Fabio Giacobbe:

    You are correct, Fabio, that the standard rule is that there's a 1-year warranty on REDs connected to UTMs.  I think that given that the problem was most likely caused by an Up2Date, Sophos might go ahead and replace the RED.

    If it turns out that you can't get a free replacement, my recommendation is to replace a RED 50 with an SG 115 with a Network Protection subscription.  That will give you more flexibility and will cost less over time than a RED 50 with Warranty Extensions.  You can configure a RED tunnel in your main office UTM and just replace the reds# in your existing Interface definition with the new one.  Depending on your present configuration, there might be very little needed to configure the new SG 115.

    Please let us know what you tried and the results.

    Cheers - Bob

  • In reply to BAlfson:

    Hi Bob,

    actually did not receive a PM from you, but anyway the first B is 9.605, in this scenario given that the REDs are not running the unified firmware prior to the update and are not connected during the update they will not receive a faulty unified firmware but only the fixed unified firmware of 9.605 so will not run into the problem, setting the unified firmware to 0 is actually not necessary in this case.

    The disabling of the REDs is done to prevent them from receiving a faulty firmware in the update process, ones on 9.605 that is not a problem anymore.

    Jan

  • In reply to JanWeber:

    Sorry, Jan, I don't see what I'm not understanding, but I can't reconcile your last post with:


    I just read your response to my PM, and my confusion remains.

    Cheers - Bob

  • Sorry,

     

    but it is a very angry issue! And i think nothing happend by Sophos. We upgrade an Friday one our SG105 with 1 RED50 to 9.605-1. Now the RED50 doesn´t boot, no Workaround helps. It is bullshit, sorry.

    Now i take a RMA and one of our Office ist down since Friday.

    Greetings,

     

    Ingo 

  • Hey everyone,

     

    I am still not sure what to do to ensure that our RED will not brick.

    Currently, our UTM is running 9.603-1. We have one RED50 (in total 5 RED devices) at this UTM. We never set the use_unified_firmware to 0, we managed to get the RED working with MTU 1400 and had no problem since. 

    In the KB article, I read that the problem with destroying the RED will only occure if we want to upgrade TO 9.6 - which we are already. 

    Here in this thread there are instructions for the update to 9.604 and 9.605, that seem to bit slightly different then the KB i found (https://community.sophos.com/kb/en-us/134398).

     

    So quick question: coming from 9.603-1, are the instructions in this thread the correct way to do the upgrade? And will the use_unified_firmware value stay 0 from now on, or do I have to change it later?

     

    Thanks for clarification!

     

    Regards,

     

    Tobias

  • In reply to _Tobias_:

    Just reporting that the MTU 1400 'trick' worked today for one of my remote sites that died late last week after 9.605-1 was installed on the Head Office SG230 (no issues with RED Connectivity prior to this). I found this thread in my troubleshooting after reading the Log Messages. Keeping an eye on it for stability for the rest of today (Monday). 

    So, at this stage, the only permanent fix is to SSH in and disable the unified firmware?

  • In reply to Dread:

    Nope - back down again ...

    2019:08:12-12:09:51 FW-SG230 red_server[13121]: xxidherexx: No ping for 30 seconds, exiting.
    2019:08:12-12:09:51 FW-SG230 red_server[13121]: id="4202" severity="info" sys="System" sub="RED" name="RED Tunnel Down" red_id="xxidherexx" forced="0"
    2019:08:12-12:09:51 FW-SG230 red_server[13121]: xxidherexx is disconnected.
    2019:08:12-12:09:51 FW-SG230 red_server[4659]: SELF: (Re-)loading device configurations
    2019:08:12-12:09:51 FW-SG230 red2ctl[4671]: Overflow happened on reds1:0
    2019:08:12-12:09:51 FW-SG230 red2ctl[4671]: Missing keepalive from reds1:0, disabling peer x.x.x.x
    2019:08:12-12:09:54 FW-SG230 red2ctl[4671]: Received keepalive from reds1:0, enabling peer x.x.x.x
    2019:08:12-12:11:00 FW-SG230 red2ctl[4671]: Missing keepalive from reds1:0, disabling peer x.x.x.x
    2019:08:12-12:11:40 FW-SG230 red_server[4659]: SELF: (Re-)loading device configurations
    2019:08:12-12:16:22 FW-SG230 red_server[6516]: SELF: Cannot do SSL handshake on socket accept from 'x.x.x.x': SSL connect accept failed because of handshake problems
    2019:08:12-12:16:22 FW-SG230 red_server[6526]: SELF: Cannot do SSL handshake on socket accept from 'x.x.x.x': SSL connect accept failed because of handshake problems
    2019:08:12-12:19:32 FW-SG230 red_server[7278]: SELF: Cannot do SSL handshake on socket accept from 'x.x.x.x': SSL wants a read first
    2019:08:12-12:41:54 FW-SG230 red_server[12970]: SELF: Cannot do SSL handshake on socket accept from 'x.x.x.x': SSL connect accept failed because of handshake problems
    2019:08:12-12:41:54 FW-SG230 red_server[12971]: SELF: Cannot do SSL handshake on socket accept from 'x.x.x.x': SSL connect accept failed because of handshake problems
     
    Going to SSH in and see if disabling the unified firmware will fix it now ...