UTM 9.601 - RED issues!

Since upgrading all our customers to 9.601, a bigger part of them are complaining about RED's re/disconnection in a no-pattern way.

It started for all of them just the night we upgraded to 9.601, and they all are on different ISP's and located different places around the country.

Been with Sophos support for 2 hours today, and now they escalated it to higher grounds.

Will return with an update....

Suspicious entries in the log - but all connected REDs do this before connection:

2019:03:06-15:15:38 fw01-2 red_server[17509]: SELF: Cannot do SSL handshake on socket accept from 'xxx.xxx.xxx.xxx': SSL connect accept failed because of handshake problems

2019:03:06-15:15:46 fw01-2 red2ctl[12420]: Missing keepalive from reds3:0, disabling peer xxx.xxx.xxx.xxx

I know the last line is written before the tunnel disconnects, because there was no "PING/PONG" answer...

One customer has 2 x RD 50, one 1 100% stable and the other fluctuates in random intervals - we replaced this with a new RED 50, but the same thing occurs.

  • Martin,

     

         I saw this same issue when I first tried 9.600. I worked with support over several days before they told me it was a known problem and asked me to revert to a prior firmware. I had assumed it was fixed with the latest version but it looks like that's not the case. You could try modifying the RED config and forcing a firmware update. There was another thread that mentioned that forcing a firmware upgrade fixed some issues.

     

     

    Cliff

        

  • Are the issues only with RED50 or the smaller ones too?

    Best regards

    Alex

  • An L2 from Sophos just fixed this for us. We had to ssh in and revert the firmware of all the reds. we have been stable for 45 minutes. I will come back and update if it comes back. Funny thing is it only effected one Red 50 of my 7.

  • In reply to Alexander Busch:

    Alexander Busch

    Are the issues only with RED50 or the smaller ones too?

    Best regards

    Alex

     

     

    Just RED 50 at the moment.

  • In reply to Devin Gray:

    Devin Gray

    An L2 from Sophos just fixed this for us. We had to ssh in and revert the firmware of all the reds. we have been stable for 45 minutes. I will come back and update if it comes back. Funny thing is it only effected one Red 50 of my 7.

     

     

    Glad to hear Devin, been with Sophos Support 2 hours yesterday and 1,5 hour today with L2, the talked about the new "unified firmware" could cause the troubles, but as one RED 50 is online with no issues, and the other is not, they could not confirm it.

     

    They talked about other similar incidents as mine and took a lot of snaphots from the logs and iptables and would return later on.

     

    Also funny thing, we have more than 250 UTM's in our SUM, only 4 customers have issues, and out of thoose 4, which have several REDs connected, only ONE of their REDs at each location, have issues!!! :-O

     

    They told me a "workaround" to maybe make it more stable, was to run in standard/split mode, but I still see re/disconnects.

     

    Will keep you posted!

  • In reply to twister5800:

    Working from the information above regarding support rolling back RED firmware, we had no success and are needing to replace the devices having issues. The following sequence of events in continual loop. Very strange since we updated the UTM's to 9.601 weeks ago. Two of three on the same UTM are dead and the third one is just fine....for now.

    Starting RED, Network Setup

    1. Network Setup

    2. ID A34xxxxxxxxxxxx

    3. Try wan1

    4. Firmware update 1/6 downloading

    5. Try Prov. Server

    6. Try wan2

    7. Network Setup

    8. Try wan1

    9. Try wan2

    10. Shutting down…

  • In reply to ToddCooper:

    Thats exactly the same issue I am having with my RED50.

    Tried chanching and deleting the configuration in the UTM. Doesent work.

  • Same issues here after 9.601-5 UTM update. 2x RED50 Rev 1. Drop multiple ISPs at varying intervals and lengths. It was advised to re-create RED in UTM. I have performed this, but problems still persist. I was sent two replacement RED50. The first one has been replaced, a new config created, but problem persists. ISPs modems have been replaced although they were reluctant to do so. One of the REDs wont recognize the presence of ISP on WAN1 at all.

    We are losing a lot of productivity and business. We do a sizeable portion of our business via teleconferencing.

    Support Tickets#

    8710435

    8707203

    8707207

     

    The tech alluded to a potential issue with REDs after the update to 9.6.01-5.

  • In reply to William Fraley:

    My problem is resolved. There is a known issue related to unified firmware.

    from su -

    cc get red use_unified_firmware

    if value returned = 1

    cc set red use_unified_firmware 0

    reds will update and reboot

    confirm value is 0 rerunning get command above

     

    NOT A PERMANENT FIX. The issue needs to be addressed in Sophos UTM firmware permanently.

  • In reply to twister5800:

    We have the same problem with multiple customers with RED50s and RED15s. The mentioned workaround fixed the problems for several RED50s and RED15s, only one RED50 doesn't come back at the moment.

  • In reply to SebastianRudolph:

    The workaround does work, until it doesn't. Seven out of ten were able to stabilize with the workaround, three had to be RMA'd.

    Lest anyone would get upset over the workaround not working. Should it work for you it's a win. Thanks for everyone's input to the forum, makes life less stressful in most cases.

  • In reply to William Fraley:

    this also worked for me ...

  • In reply to somi:

    Today the Tunnel of one RED was down again (with use_unified_firmware = 0). It was the one that was problematic after updating to 9.601-5

    Disabling and reenabling the RED did not fix the issue. After switching back to 'unified firmware' alle REDs are UP again (use_unified_firmware = 1)

    I think it has to bee something different as the issue still apears after one day (with or without unified firmware). ...

    However it is related to version 9.601 as i did not have issues before the update...

  • In reply to somi:

    Hallo somi and welcome to the UTM Community!

    Based on what others have said above, I would push Sophos Support to RMA the failing RED.

    Cheers - Bob

  • In reply to BAlfson:

    Hey Bob,

    thanks, we already replaced that RED with a brand new RED 15 - same thing.

    Regards,

    Michael