UTM 9.601 - RED issues!

Since upgrading all our customers to 9.601, a bigger part of them are complaining about RED's re/disconnection in a no-pattern way.

It started for all of them just the night we upgraded to 9.601, and they all are on different ISP's and located different places around the country.

Been with Sophos support for 2 hours today, and now they escalated it to higher grounds.

Will return with an update....

Suspicious entries in the log - but all connected REDs do this before connection:

2019:03:06-15:15:38 fw01-2 red_server[17509]: SELF: Cannot do SSL handshake on socket accept from 'xxx.xxx.xxx.xxx': SSL connect accept failed because of handshake problems

2019:03:06-15:15:46 fw01-2 red2ctl[12420]: Missing keepalive from reds3:0, disabling peer xxx.xxx.xxx.xxx

I know the last line is written before the tunnel disconnects, because there was no "PING/PONG" answer...

One customer has 2 x RD 50, one 1 100% stable and the other fluctuates in random intervals - we replaced this with a new RED 50, but the same thing occurs.

  • So I had previously switched to the non Unified Firmware for my RED 15s (all my REDs are RED 15s) which helped noticeably but we were still experiencing random dropouts of random REDs.  As of now I have upgraded manually to 9.700-5.  My question is do I need to re-enable/switch back to Unified Firmware or not?

     

    Thanks,

    Tracy 

  • In reply to Tracy Carlton:

    Check the status by going to the command line and issue command "cc get red use_unified_firmware"

    If it returns a 1 then your upgrade has automatically turned the Unified firmware back on. In previous upgrades this was the default behavior. See previous posts in this thread about the best way to perform an upgrade and still retain the old RED firmware on the devices.

     

    Whether or not you "need" to be on the Unified firmware is a matter of choice. I am still seeing people here reporting big problems with the Unified firmware (note Twisters many issues). I am not using the Unified firmware until I am convinced the issues have been completely fixed.

  • In reply to garth1138:

    I checked after the upgrade and can confirm that it *DID NOT* switch back to unified firmware.  The cc get red use_unified_firmware returned a 0 on both my UTMs after the upgrade.

     

    I am going to monitor the logs closely for the next little while an see what if anything happens.

     

    Tracy

  • In reply to Tracy Carlton:

    Tracy Carlton

    I checked after the upgrade and can confirm that it *DID NOT* switch back to unified firmware.  The cc get red use_unified_firmware returned a 0 on both my UTMs after the upgrade.

     ...

    These are very good news for anybody struggling with the update to 9.7. Thank you for sharing that. It seems Sophos has learned a little bit from the updates since 9.601.

    Best regard

    Alex

  • In reply to Alexander Busch:

    It started for all of  xenderthem just th omegle e night we upgraded to 9.601, and they all are on discord  different ISP's and located different places around the country.

  • In reply to Alexander Busch:

    This could also be very bad news, since users are reporting that everything is fixed in 9.700-5. It could be that they are all still running with the non unified version of the firmware and just think that it's been fixed. 

  • In reply to John Holliday:

    Just had another occurence of the server shutting down the clients, although it rebooted the client, it didn't bring the interface up correctly, why is the server killing off the clients

     

    2019:11:06-11:34:43 sophos-2 red_server[9984]: A#########ab: command '{"data":{"seq":783},"type":"PING"}'
    2019:11:06-11:34:43 sophos-2 red_server[9984]: A#########ab: Sending json message {"data":{"seq":783},"type":"PONG"}
    2019:11:06-11:34:44 sophos-2 red_server[32198]: SELF: shutdown requested, killing clients
    2019:11:06-11:34:44 sophos-2 red_server[32198]: SELF: killing client A#########ab
    2019:11:06-11:34:44 sophos-2 red_server[9984]: id="4202" severity="info" sys="System" sub="RED" name="RED Tunnel Down" red_id="A#########ab" forced="0"
    2019:11:06-11:34:44 sophos-2 red_server[32198]: SELF: killing client A#########3a
    2019:11:06-11:34:44 sophos-2 red_server[9984]: A#########ab is disconnected.
    2019:11:06-11:34:45 sophos-2 red_server[32198]: SELF: exiting
    2019:11:06-11:34:45 sophos-1 red_server[27988]: SELF: RED10rev1 fw version set to 14
    2019:11:06-11:34:45 sophos-1 red_server[27988]: SELF: RED10rev2 local fw version set to 5214R2
    2019:11:06-11:34:45 sophos-1 red_server[27988]: SELF: RED10rev2 fw version set to 2005R2
    2019:11:06-11:34:45 sophos-1 red_server[27988]: SELF: RED15(w) fw version set to 1-424-7131d4e52-e9f0c31
    2019:11:06-11:34:45 sophos-1 red_server[27988]: SELF: RED50 fw version set to 1-424-7131d4e52-0000000
    2019:11:06-11:34:45 sophos-1 red_server[27988]: SELF: IO::Socket::SSL Version: 1.953
    2019:11:06-11:34:45 sophos-1 red_server[27988]: SELF: Startup - waiting 15 seconds ...
    2019:11:06-11:34:46 sophos-1 red2ctl[28000]: Starting REDv2 control daemon
    2019:11:06-11:34:49 sophos-2 red2ctl[32206]: Stopping REDv2 control daemon
    2019:11:06-11:35:00 sophos-1 red_server[27988]: SELF: Overlay-fw has been updated ...
    2019:11:06-11:35:00 sophos-1 red_server[28845]: UPLOAD: Uploader process starting
    2019:11:06-11:35:01 sophos-1 red_server[27988]: SELF: (Re-)loading device configurations
    2019:11:06-11:35:01 sophos-1 red_server[27988]: A#########3a: New device
    2019:11:06-11:35:01 sophos-1 red_server[27988]: A#########3a: Staging config for upload
    2019:11:06-11:35:01 sophos-1 red_server[27988]: A#########ab: New device
    2019:11:06-11:35:01 sophos-1 red_server[27988]: A#########ab: Staging config for upload
    2019:11:06-11:35:02 sophos-1 red_server[28845]: [A#########3a] Config has not changed, no need to upload to registry service
    2019:11:06-11:35:02 sophos-1 red_server[28845]: [A#########ab] Config has not changed, no need to upload to registry service
    2019:11:06-11:36:34 sophos-1 red_server[29146]: SELF: Cannot do SSL handshake on socket accept from 'xx.100.xx.xx': SSL connect accept failed because of handshake problems
    2019:11:06-11:36:37 sophos-1 red_server[29147]: SELF: New connection from xx.100.xx.xx with ID A#########ab (cipher AES256-GCM-SHA384), rev1
    2019:11:06-11:36:37 sophos-1 red_server[29147]: A#########ab: connected OK, pushing config
    2019:11:06-11:36:37 sophos-1 red_server[27988]: SELF: (Re-)loading device configurations
    2019:11:06-11:36:38 sophos-1 red_server[29147]: A#########ab: command '{"data":{"version":"0"},"type":"INIT_CONNECTION"}'
    2019:11:06-11:36:38 sophos-1 red_server[29147]: A#########ab: Initializing connection running protocol version 0
    2019:11:06-11:36:38 sophos-1 red_server[29147]: A#########ab: Sending json message {"data":{},"type":"WELCOME"}
    2019:11:06-11:36:39 sophos-1 red_server[29147]: A#########ab: command '{"data":{},"type":"CONFIG_REQ"}'
    2019:11:06-11:36:39 sophos-1 red_server[29147]: A#########ab: Sending json message {"data":