This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

UTM 9.601 - RED issues!

Since upgrading all our customers to 9.601, a bigger part of them are complaining about RED's re/disconnection in a no-pattern way.

It started for all of them just the night we upgraded to 9.601, and they all are on different ISP's and located different places around the country.

Been with Sophos support for 2 hours today, and now they escalated it to higher grounds.

Will return with an update....

Suspicious entries in the log - but all connected REDs do this before connection:

2019:03:06-15:15:38 fw01-2 red_server[17509]: SELF: Cannot do SSL handshake on socket accept from 'xxx.xxx.xxx.xxx': SSL connect accept failed because of handshake problems

2019:03:06-15:15:46 fw01-2 red2ctl[12420]: Missing keepalive from reds3:0, disabling peer xxx.xxx.xxx.xxx

I know the last line is written before the tunnel disconnects, because there was no "PING/PONG" answer...

One customer has 2 x RD 50, one 1 100% stable and the other fluctuates in random intervals - we replaced this with a new RED 50, but the same thing occurs.



This thread was automatically locked due to age.
Parents
  • Reporting same issues here. (2) RED50s in our setup. One in same USA state as main UTM, but different ISP. One in Hyderabad, India

    Both were working fine after upgrade to 9.601 on 2019-03-10.

    Then, as of 2019-03-30, the USA RED50 dropped offline. Troubleshooting the issue showed all of the same symptoms as reported here.

    Log:

    2019:03:30-19:51:23 astaro red_server[11907]: A34XXXXXXXXXF19: command '{"data":{"key1":"GwQ1b43erVLU4RzintYPM\/T1lczqkZWvfag7yHSQBVo=","key_active":0},"type":"SET_KEY_REQ"}'
    2019:03:30-19:51:23 astaro red_server[11907]: A34XXXXXXXXXF19: Sending json message {"data":{},"type":"SET_KEY_REP"}
    2019:03:30-19:51:24 astaro red_server[11907]: A34XXXXXXXXXF19: No ping for 30 seconds, exiting.
    2019:03:30-19:51:24 astaro red_server[11907]: id="4202" severity="info" sys="System" sub="RED" name="RED Tunnel Down" red_id="A34XXXXXXXXXF19" forced="0"
    2019:03:30-19:51:24 astaro red_server[11907]: A34XXXXXXXXXF19 is disconnected.
    2019:03:30-19:51:24 astaro red_server[4803]: SELF: (Re-)loading device configurations
    2019:03:30-19:51:25 astaro red2ctl[4813]: Overflow happened on reds2:0
    2019:03:30-19:51:25 astaro red2ctl[4813]: Missing keepalive from reds2:0, disabling peer 24.x.y.63
    2019:03:30-19:51:28 astaro red2ctl[4813]: Received keepalive from reds2:0, enabling peer 24.x.y.63
    2019:03:30-19:51:40 astaro red2ctl[4813]: Missing keepalive from reds2:0, disabling peer 24.x.y.63

    2019:03:30-19:52:25 astaro red_server[12523]: SELF: Cannot do SSL handshake on socket accept from '24.x.y.63': SSL connect accept failed because of handshake problems

    2019:03:30-19:53:06 astaro red_server[12528]: SELF: Cannot do SSL handshake on socket accept from '24.x.y.63': SSL wants a read first

     

    further on:

    2019:03:30-19:58:52 astaro red_server[13844]: SELF: New connection from 24.a.b.162 with ID A34XXXXXXXXXF19 (cipher RC4-SHA), rev1
    2019:03:30-19:58:52 astaro red_server[13844]: A34XXXXXXXXXF19: connected OK, pushing config
    2019:03:30-19:59:23 astaro red_server[13844]: A34XXXXXXXXXF19: No ping for 30 seconds, exiting.
    2019:03:30-19:59:23 astaro red_server[13844]: id="4202" severity="info" sys="System" sub="RED" name="RED Tunnel Down" red_id="A34XXXXXXXXXF19" forced="0"
    2019:03:30-19:59:23 astaro red_server[13844]: A34XXXXXXXXXF19 is disconnected.

    It then went silent.

    The LCD screen shows unit going through a similar procedure reported above by Twister5800:

    Network Setup
    ID A34xxxxxxxxxxxx
    Network Setup
    Try wan1
    Try wan2
    Firmware update 1/6 downloading
    Try Prov. Server
    Network Setup
    Try wan1
    Try wan2
    Firmware update 1/6 downloading
    Try Prov. Server
    Shutting down…

     

    We replaced the problematic RED50 with a backup RED10 and it is stable thusfar. Not sure if the RED10 uses the Unifies firmware or not? I checked our config from the console and it does show "red use_unified_firmware=1"

    We have no issue so far with the India RED50 unit. Keeping our fingers crossed and waiting for the new firmware.

  • Hi everyone,

     

    we have the same exact behaviour with one of our RED50, started exactly after the update to 9.601-5. Strangely we saw that the disconnects only happend during office hours, apperently without much traffic the RED was stable...
    We found a different and hopefully supported workaround, although I do not understand, why it works.

     

    Someone told us that sometimes the RED does not get the new firmware correctly, and you would have to force a firmware update. Either by changing some settings on the RED, forcing a reboot, or - his suggestion - changing the MTU on the RED interface to 1400 until the firmware is downloaded.

    I did that - our RED has two used interfaces with different VLAN configurations (one for data, one for VOIP network). I set the MTU on the data interface to 1400, and there were no more disconnects. I thought that now I can go back to default, so I swiched the MTU back, and the disconnects started again. I then tried different MTU values down to 1450, without success. Since a week now, we are at 1400, and not one disconnect since then.

     

    Maybe this is not at all related to the RED bug, but since it started with the update... I will see when the patch is applied - if I still cannot go back to the default MTU, there was another problem. But I still wanted to share this, since it is easier to change the MTU then changing something on the console...

     

    Regards,

     

    Tobias

Reply
  • Hi everyone,

     

    we have the same exact behaviour with one of our RED50, started exactly after the update to 9.601-5. Strangely we saw that the disconnects only happend during office hours, apperently without much traffic the RED was stable...
    We found a different and hopefully supported workaround, although I do not understand, why it works.

     

    Someone told us that sometimes the RED does not get the new firmware correctly, and you would have to force a firmware update. Either by changing some settings on the RED, forcing a reboot, or - his suggestion - changing the MTU on the RED interface to 1400 until the firmware is downloaded.

    I did that - our RED has two used interfaces with different VLAN configurations (one for data, one for VOIP network). I set the MTU on the data interface to 1400, and there were no more disconnects. I thought that now I can go back to default, so I swiched the MTU back, and the disconnects started again. I then tried different MTU values down to 1450, without success. Since a week now, we are at 1400, and not one disconnect since then.

     

    Maybe this is not at all related to the RED bug, but since it started with the update... I will see when the patch is applied - if I still cannot go back to the default MTU, there was another problem. But I still wanted to share this, since it is easier to change the MTU then changing something on the console...

     

    Regards,

     

    Tobias

Children
  • Hi All,

    Just a quick update, had this issue on a RED 15w today in a remote site office connected to a UTM SG330 running 9.601-5.

    Receiving logs as;

    2019:04:03-12:44:54 uaejltfw red_server[7052]: SELF: Cannot do SSL handshake on socket accept from 'hidden': SSL connect accept failed because of handshake problems
    2019:04:03-12:45:31 uaejltfw red_server[7066]: SELF: Cannot do SSL handshake on socket accept from 'hidden': SSL wants a read first

    I tried disabling the RED tunnel and waiting for the reconnect, then enabling it again - no success.

    I then did as Tobias stated above and changed the MTU on the interface to 1400 - no success yet.

    I then disabled the RED tunnel again, then re-enabling it again - SUCCESS.

    So it looks like the MTU 1400 change fixed it for me after the RED tried to re-connect after a timeout.

    Please note, I have NOT disabled the unified firmware in the shell as suggested above, this was going to be my last resort.

    Kind Regards,

    James.

  • Hi Tobias

     

    Had this exact problem yesterday after updating to 9.601-5 the single RED 50 that was connected to the appliance started flapping. The appliance has several RED 10 and RED 15's connected, none of them had the problem.

     

    Fortunately I stumbled on this article, so it was a quick fix. What did it for me was setting the MTU to 1400. This solved my problem immediately. Thanks for posting.

     

    I must say I am slightly shocked that this known problem hasn't been fixed by Sophos. Very frustrating!

     

     

    Ivan