One of our UTMs has a single RED15W connected to give one of our road warriors access, till today this worked fine.
A few hours ago, I updated this UTMs to 9.601-5, the update itself did it's job, after reboot the UTM was up and running.
Unfortunately now the RED15W can't establish it's VPN to the UTM. I attached some lines of the UTMs RED log file.
Seems the UTM recognizes the REDs old firmware, but there is no update for the RED or at least it doesn't work.
The RED is trying to reconnect every 2minutes, without success, it's always the same error, "Disconnecting: Firmware update required. Trying provisioning service ..."
Any hints how to fix this (besides doing a new, clean install with an older UTM firmware)?
---------------
2019:02:19-13:46:06 mail red_server[8034]: SELF: Cannot do SSL handshake on socket accept from '<IP>': SSL connect accept failed because of handshake problems2019:02:19-13:46:06 mail red_server[8035]: SELF: Cannot do SSL handshake on socket accept from '<IP>': SSL connect accept failed because of handshake problems SSL wants a read first2019:02:19-13:46:09 mail red_server[8043]: SELF: New connection from <IP> with ID A360203FDXXXXXX (cipher AES256-GCM-SHA384), rev12019:02:19-13:46:09 mail red_server[8043]: A360203FDXXXXXX: connected OK, pushing config2019:02:19-13:46:10 mail red_server[8043]: A360203FDXXXXXX: command '{"data":{"version":"0"},"type":"INIT_CONNECTION"}'2019:02:19-13:46:10 mail red_server[8043]: A360203FDXXXXXX: Initializing connection running protocol version 02019:02:19-13:46:10 mail red_server[8043]: A360203FDXXXXXX: Sending json message {"data":{},"type":"WELCOME"}2019:02:19-13:46:11 mail red_server[8043]: A360203FDXXXXXX: command '{"data":{},"type":"CONFIG_REQ"}'2019:02:19-13:46:11 mail red_server[8043]: A360203FDXXXXXX: Sending json message {"data":{"pin":"","fullbr_dns":"","split_networks":"1.2.3.4","lan2_vids":"","lan4_vids":"","local_networks":"","tunnel_id":1,"manual2_netmask":24,"asg_cert":"[removed]","manual_address":"0.0.0.0","bridge_proto":"none","unlock_code":"tkc60a7x","password":"","manual2_defgw":"0.0.0.0","prev_unlock_code":"","manual_netmask":24,"lan3_vids":"","version_r2":"2005R2","mac_filter_type":"none","mac":"00:28:4e:14:XX:XX","dial_string":"*99#","manual2_address":"0.0.0.0","version_ng_red50":"1-330-f4c55ab8-0000000","manual_dns":"0.0.0.0","lan1_mode":"unused","username":"","activate_modem":0,"tunnel_compression_algorithm":"lzo","version_red50":"1-330-f4c55ab8-0000000","fullbr_domains":"","htp_server":"mail.hoehne.org","uplink_balancing":"failover","asg_key":"[removed]","type":"red15w","deployment_mode":"online","uplink2_mode":"dhcp","version_red15":"1-330-f4c55ab8-655eb7e","manual2_dns":"0...L15052019:02:19-13:46:12 mail red_server[8043]: A360203FDXXXXXX: command '{"data":{"message":"Firmware update required. Trying provisioning service ..."},"type":"DISCONNECT"}'2019:02:19-13:46:12 mail red_server[8043]: A360203FDXXXXXX: Disconnecting: Firmware update required. Trying provisioning service ...2019:02:19-13:46:12 mail red_server[8043]: id="4202" severity="info" sys="System" sub="RED" name="RED Tunnel Down" red_id="A360203FDXXXXXX" forced="1"2019:02:19-13:46:12 mail red_server[8043]: A360203FDXXXXXX is disconnected.
Hi all,
just encountered the same problem after updating from 9.600 to 9.601.
Since I had disabled compression in 9.600 to make the RED-tunnel work, I first tried to enable compression again.
This triggered a redeployment of the config which in turn triggered a firmwareupdate.
After the firmwareupdate, the RED tunnel came up.
So, no need to delete the config. Just modifying it should do the trick.
BR,
Michael
firmwareking said: Hi all, just encountered the same problem after updating from 9.600 to 9.601. Since I had disabled compression in 9.600 to make the RED-tunnel work, I first tried to enable compression again. This triggered a redeployment of the config which in turn triggered a firmwareupdate. After the firmwareupdate, the RED tunnel came up. So, no need to delete the config. Just modifying it should do the trick. BR, Michael
Worked for me after the last update to 9.603-1
Just modifying and after RED Comes back up just change it back...
With latest update 9.605 same thing happened. 4 RED15's refused to come back.....had to disable unified firmware, STILL didn't work. Tried a config change, STILL didn't work. Only thing that did was changed the tunnel type from Standard/Split to Unified back to Split. Changed failover to balance and back again.
Sophos really have to get their sh*t together here, i live in Arizona and manage a system based all over the UK. 21 bricked remote locations causes me a BIG headache.
Answers would be good Sophos.
Have the same problem.
Version 9.604-2 on two SG125 with UTM 9.
5x RED-15.
The workaround to disable the update works for 4 devices.
1 keeps offline loop and try to download the 5211 firmware.
Recreatong doesnt help. MTU1400 doesnt help. Switchig settings (balance/failover, compression on/off, etc.) doesnt help.
what to do now??
Hi !
was the failing device offline for a long time ?
If yes, the firmware may be to old to download the latest firmware from the update-server.
Sophos has droped support for older SSL variants. one can discuss, if this is good or bad in this case but it has bricked two of our RED50 that we kept as replacement parts in the store.
The official way is to open a ticket and get them replaced, as long as they are unter warranty. The unsupported way is to download the image by hand and trick the RED to use it instead of doawnloading it. (It's openWRT running on those devices...)
Is the recovery still working? Does anyone know?
https://community.sophos.com/kb/en-us/118843
2019-08-06 See my final version posted today.
UPDATED 2019-08-01
Hi All,
I've had several messages back and forth with Sophos folks. As Jan Weber says in a post, 9.605 fixes the problem with REDs and the only danger is updating the RED firmware when the RED is under a heavy load. I have suggested that the following instructions be added to the information about the Up2Date (I in blue dot) and the blog post about the 9.605 Up2Date:
In order to ensure that there's no problem with the update of firmware in RED devices, do the following with two planned outages:
1. Outage 1 - Up2Date to 9.604: A. In WebAdmin, disable all RED Servers for RED appliances. B. Apply Up2Dates through 9.604. C. At the command line: cc set red use_unified_firmware 0 D. In WebAdmin, enable all RED Servers for RED appliances. 2. Outage 2 - Disconnect all LAN connections from all REDs, leaving the RED online but with no connection to local clients. 3. Apply the 9.605 Up2Date. 4. After the Up2Date is complete, reconnect disconnected LAN cables to the REDs.
Cheers - Bob
Hey Bob,
sorry, but your procedure is not precise engough. At least not for me.
First things first...
1. What is the difference between "Outage 1" and "Outaged 2"?
2. Refers to your "A": By disbling the "RED Servers" you mean the devices under "RED Management\[Server] Client Management"?
3. Refers to your "B": Apply the updates to REDs? How?
4. Refers to your "C": What does this command do?
By the way...we have the same issues after updating Sophos SG from 9.604-2 to 9.605-1.
1. You could do them back-to-back, but that may not be practical for everyone.
2. Exactly.
3. REDs are not updated at this point - that will only happen when the 9.605 Up2Date is applied with the REDs connected but with no clients connected to them. Note the difference between "updates" and "Up2Dates" - the latter is new firmware updates for the UTM.
4. This disables the use of unified firmware for the REDs. Prior to the RED updates done by the UTM after the 9.605 Up2Dates, some sites had problems with REDs when use_unified_firmware was enabled.
Cheers - BobPS See Jan Weber's (a Sophos guy that really knows his stuff!) post above. His approach is a bit different, but it seems to me that it's easier to make a mistake with it.
I have a UTM running version 9.604-2. It is connected to a RED 15 in a far away country, and the people there do not have technical skills. We just switched the RED 15 to a new internet connection with a new IP address. Fortunately, it connected and appears to be working. Here are my questions:
* If we are running 9.604-2, are we already past the point of risk to brick our RED 15?
* Should I update our UTM to 9.605-1? The update says, "high priority."
* If I update to 9.605-1, do I need to follow the two outage procedure posted by Bob above?
* If I run the update at night when nobody is in the remote office and network traffic is very low (in kbytes/sec), can I skip the Outage 2 step where I disconnect the other devices from the remote LAN? I don't trust the people at the remote site to get it right.
* Is there a way to read the current RED firmware version?
* Is there a way to determine if a RED update is pending from the UTM?
* Is there a way to read the current "cc set red use_unified_firmware" setting?
* Does the "cc set red use_unified_firmware" setting change when there is a UTM update? In other words, if you run "cc set red use_unified_firmware", does it revert to the correct setting on its own after updates are complete?
I haven't heard of 15s bricking, just 50s. If you're on 9.604, follow my post above to go from there to 9.605. I don't think your 15 would be permanently damaged, but better-safe-than-sorry, so disconnect the clients behind it or, if you're sure someone won't leave a download running overnight, doing it at night "should" work. I'm sure there's a way to get inside the RED and see its version, but, according to several Sophos folks, it's the RED update with 9.605 that permanently fixes this RED issue. I'm not aware of a way to know what's in an Up2Date except for looking at the details in the Up2Date blog. cc get red use_unified_firmware - a 0 result tells you it's disabled.
The answer to your last question is "sometimes," but according to Sophos folks, either setting should work after applying the 9.605 Up2Date with the REDs attached and enabled.
Updated 4.5 hours later
Final version.
If you don't have any RED appliances attached, apply all Up2Dates through 9.605 now.
It has come to my attention that those of you at 9.510 or below will not have an option to go to 9.604 as once you get to 9.510, the only available Up2Date you will see in WebAdmin is the one straight to 9.605. In fact the problem with REDs only began with 9.601.
If you are at 9.510 or below and you see no available Up2Dates between 9.510 and 9.605, disregard my earlier instructions below and go ahead and Up2Date directly to 9.605, regardless of whether you have REDs attached.
It's important to get to 9.604 at least as soon as possible, so I've broken up the process into two outages. If you have RED appliances, please do the following:
I. Outage 1 - Up2Date to 9.604: 1. In WebAdmin, '[Server] Client Management' disable all RED Servers for RED appliances. 2. Apply Up2Dates through 9.604. 3. At the command line as root: cc set red use_unified_firmware 0 4. In WebAdmin, enable all RED Servers for RED appliances. II. Outage 2 - Up2Date to 9.605 1. Disconnect all LAN connections from all REDs, leaving the REDs online with no local clients. 2. Apply the 9.605 Up2Date. 3. After the Up2Date is complete, reconnect disconnected LAN cables to the REDs.
Ok, thank you very much for your support here.
But what to do when you are already on 9.605?
Can i bring my remote REDs somehow back to life? For example directly attach to Sophos SG or something like that?
Based on what others have said in this thread and other places, RED 15s can usually be brought back online by simply powering them off/on. If that doesn't work, try deleting the Server in WebAdmin and adding it back again - be sure to note the Unlock code before you delete the Server! If that doesn't work, get a ticket open with Sophos Support.
You can try the same thing with RED 50s, but there have been instances of some "bricking" - if Sophos Support doesn't want to give you an RMA for a bricked RED 50 that's out of warranty (1 year), request escalation. I expect that Sophos will already have relaxed enforcement of that for devices that were killed by a bad Up2Date.