We have a Sophos XG 210 running firmware 18.0.4 MR-4 with quite a few VPN tunnels and so far 8 RED devices attached running firmware 3.0.004. The REDs are working well unless we have the machines behind the RED configured with DHCP. This is a guess at this point because we're struggling getting to the bottom of the issue. Below is the log.
We do know that pulling the power on the RED fixes the issue even though it appears to be connected prior to the power reset. We've since added email alerts on a disconnected RED, but I suspect the connection is there but it's not allowing some traffic through.
The REDs are all bridged into a single interface, so a single DHCP pool serves all the devices. The REDs themselves are configured with a static public IP address and we only have 2-3 devices behind each one.
Has anyone else had DHCP problems with RED devices?
Fri Feb 19 18:04:25 2021 REDD INFO SerialNumber/LocationName transfered bytes TX: 9235344 RX: 2165780Fri Feb 19 18:04:33 2021 REDD INFO command '{"data":{"seq":7922},"type":"PING"}'Fri Feb 19 18:04:33 2021 REDD INFO Sending json message {"type":"PONG","data":{"seq":7922}}Fri Feb 19 18:09:26 2021 REDD INFO SerialNumber/LocationName transfered bytes TX: 19535120 RX: 4083252Fri Feb 19 18:09:37 2021 REDD INFO command '{"data":{"seq":7941},"type":"PING"}'Fri Feb 19 18:09:37 2021 REDD INFO Sending json message {"data":{"seq":7941},"type":"PONG"}Fri Feb 19 18:14:09 2021 REDD INFO Sending json message {"data":{"seq":7958},"type":"PONG"}Fri Feb 19 18:14:27 2021 REDD INFO SerialNumber/LocationName transfered bytes TX: 144096 RX: 39916Fri Feb 19 18:19:27 2021 REDD INFO command '{"data":{"poe_chip_status":{"type":"chip","id":46,"totalPower":0,"totalPowerReg":0,"temperature":57.48,"volt":53.46027,"totalPowerCalc":0,"maxTotalPower":34,"firmware":12}},"type":"STATUS"}'Fri Feb 19 18:19:28 2021 REDD INFO SerialNumber/LocationName transfered bytes TX: 194240 RX: 39984Fri Feb 19 18:21:51 2021 REDD INFO command '{"data":{"key_active":1,"key0":"Very_Long_Key"},"type":"SET_KEY_REQ"}'Fri Feb 19 18:21:51 2021 REDD INFO Sending json message {"type":"SET_KEY_REP","data":{}}Fri Feb 19 20:12:21 2021 REDD INFO No ping for 30 seconds, exiting.Fri Feb 19 20:12:21 2021 REDD INFO SerialNumber/LocationName is now disconnectedFri Feb 19 20:12:21 2021 REDD INFO device is disconnected.Fri Feb 19 20:14:36 2021 REDD INFO server: New connection from IP_Number (cipher ECDHE-RSA-AES256-GCM-SHA384), rev1Fri Feb 19 20:14:37 2021 REDD INFO connected OK, pushing configFri Feb 19 20:14:37 2021 REDD INFO command '{"data":{"version":"0"},"type":"INIT_CONNECTION"}'Fri Feb 19 20:14:37 2021 REDD INFO Initializing connection running protocol version 0Fri Feb 19 20:14:37 2021 REDD INFO Sending json message {"data":{},"type":"WELCOME"}Fri Feb 19 20:14:38 2021 REDD INFO command '{"data":{"poe_port1_status":{"current":0,"FETok":true,"detectionOn":false,"pdstate":0,"port":1,"pdclass":-3,"classificationOn":false,"classFail":false,"TPPL":0,"good":false,"type":"port","priority_str":"high","PMoff":false,"MSCCcap":false,"priority":0,"mode":0,"pdstate_str":"unknown","volt":0,"PPL":0,"pdclass_power_limit":0,"FEToverTemp":false,"isAT":false,"power":false,"watt":0,"mode_str":"shutdown"},"poe_port2_status":{"current":0,"FETok":true,"detectionOn":false,"pdstate":0,"priority_str":"low","pdclass":-3,"classificationOn":false,"classFail":true,"TPPL":0,"good":false,"type":"port","port":2,"PMoff":false,"MSCCcap":false,"watt":0,"power":false,"PPL":0,"FEToverTemp":false,"isAT":false,"pdclass_power_limit":0,"volt":0,"pdstate_str":"unknown","mode":0,"priority":2,"mode_str":"shutdown"},"poe_chip_status":{"type":"chip","id":46,"totalPower":0,"totalPowerReg":0,"temperature":52.68,"volt":53.4486,"totalPowerCalc":0,"maxTotalPower":34,"firmware":12}},"type":"STATUS"}'Fri Feb 19 20:14:39 2021 REDD INFO command '{"data":{},"type":"CONFIG_REQ"}'Fri Feb 19 20:14:39 2021 REDD INFO Sending json message {"type":"CONFIG_REP","data":{"manual2_address":"","poe_port2":0,"htp_port":"4444","hub_hostname":"HUB_IP","uplink_mode":"manual","unlock_code":"46ctci38","lanport_mode":"switch","red_id":"SerialNumber","lan4_mode":"unused","tunnel_id":9,"manual2_dns":"","manual_defgw":"RED_IP","tunnel_compression":0,"lan1_vids":"","htp_server":"HUB_IP","lan3_mode":"unused","hub2_hostname":"","mobile_network":"gsm","apn":"","fullbr_dns":"","prev_unlock_code":"","manual2_defgw":"","debug_level":0,"route_mode":"default","asg_ca":"[removed]","asg_key":"[removed]","manual_address":"IP_Number","bridge_proto":"none","lan2_vids":"","lan2_mode":"unused","redinterface":"reds9","manual_dns":"DNS_IP","responsivity":"low","bridge_netmask":24,"pin":"NULL","bridge_address":"0.0.0.0","mac":"MAC_ADDRESS","split_networks":"1.2.3.4","tunnel_compression_algorithm":"lzo","version_ng_red60":"1-1117-3aa1e7992-fda4803","asg_cert":"[removed]","branchname":"LocationName","fullbr_domains":"","lan4_vids":"","uplink_balancing":"failover","lan3_vids":"","manual2_netmask":"","password":"","version_red60":"1-1117-3aa1e7992-fda4803","activate_modem":0,"hostname_balancing":"failover","poe_port1":0,"mac_filter_list":"","mac_filter_type":"none","manual_netmask":27,"dial_string":"*99#","username":"","deployment_mode":"online","lan1_mode":"unused","uplink2_mode":"dhcp","type":"red60"}}Fri Feb 19 20:14:42 2021 REDD INFO command '{"data":{"key1":"Very_Long_Key","key0":"Very_Long_Key","key_active":0},"type":"SET_KEY_REQ"}'Fri Feb 19 20:14:42 2021 REDD INFO Sending json message {"data":{},"type":"SET_KEY_REP"}Fri Feb 19 20:14:43 2021 REDD INFO command '{"data":{"seq":0},"type":"PING"}'Fri Feb 19 20:14:43 2021 REDD INFO SerialNumber/LocationName is now re-connected after 173000 msFri Feb 19 20:14:43 2021 REDD INFO Sending json message {"type":"PONG","data":{"seq":0}}Fri Feb 19 20:14:44 2021 REDD INFO command '{"data":{"switch_port_status_v2":{"lan3":"1Gb\/s","lan1":"Down","lan4":"1Gb\/s","lan2":"1Gb\/s"},"wan1_ip":"IP_Number","uplink":"WAN1","uplink_state":"0"},"type":"STATUS"}'Fri Feb 19 20:14:51 2021 REDD INFO SerialNumber/LocationName transfered bytes TX: 405280 RX: 171756Fri Feb 19 20:14:59 2021 REDD INFO command '{"data":{"seq":1},"type":"PING"}'Fri Feb 19 20:14:59 2021 REDD INFO Sending json message {"data":{"seq":1},"type":"PONG"}Fri Feb 19 20:15:08 2021 REDD INFO command '{"data":{"poe_chip_status":{"type":"chip","id":46,"totalPower":0,"totalPowerReg":0,"temperature":54.6,"volt":53.477775,"totalPowerCalc":0,"maxTotalPower":34,"firmware":12}},"type":"STATUS"}'Fri Feb 19 20:15:15 2021 REDD INFO command '{"data":{"seq":2},"type":"PING"}'Fri Feb 19 20:15:15 2021 REDD INFO Sending json message {"type":"PONG","data":{"seq":2}}Fri Feb 19 20:15:31 2021 REDD INFO command '{"data":{"seq":3},"type":"PING"}'Fri Feb 19 20:15:31 2021 REDD INFO Sending json message {"type":"PONG","data":{"seq":3}}
Update: I was able to confirm the desktops behind the RED is set to static IP, so it's not a DHCP issue. The RED simply stops traffic until a reboot. It appears to be online, but does not allow traffic…
Hello Brandon,
Thank you for contacting the Sophos Community.
What mode is the RED deployed?
Depending on your above answer, does the traffic stops even for traffic directed to the internet or only to resources on the XG side?
Can you share a screenshot of your Bridge RED configuration?
If you a Packet Capture on the GUI og the XG when the issue is present, do you see traffic arriving to the XG?
Regards,
The RED is configured in Standard/unified mode.
We know the machine behind the RED is offline, so there is no traffic either to internal or internet.
We'll need to capture packets once it fails again. Unfortunately, each time it stops working someone is waiting to get work done we're in a hurry to get it working again. But considering all the connections in the screenshot, we've only had one location with a problem, it's been two different devices at that location. All the other locations have been ok. I realize this points to something at the location, but I'm at a loss to what it is considering the RED has a direct connection to the Coax.
It looks like my issue is related to the same issue in this thread. https://community.sophos.com/xg-firewall/f/discussions/124690/urgent-sophos-xg-18-0-3-mr-3-red60-loses-connectivity-no-dhcp/
Do we have an issue with version 18 or possibly RED firmware. We are running 3.0.002 and I see there is 3.0.003 available.
Emmanuel, It looks like the SD-RED 60 lockup is a common issue with multiple discussions in the community. I was able to confirm we are actually running 3.0.003 (not unified) firmware on the RED. We are switching to unified tonight. At this point we have a site with 11 SD-RED 60 units and will be adding another 15 soon. We are considering not using the REDs with the stability issues we've seen.
Is there a different XG firmware or pattern update we can try?
References:
https://community.sophos.com/xg-firewall/f/discussions/124690/urgent-sophos-xg-18-0-3-mr-3-red60-loses-connectivity-no-dhcp/
https://community.sophos.com/xg-firewall/b/blog/posts/sd-red-firmware-3-0-003-pattern-update?CommentId=69a7772c-bc81-47bd-baea-556008aeb482
Thank you for the follow-up?
Do you have any Case ID open with us, that you could share? If not I would recommend you to get one open.
MR5 is bringing a fix for an issue with REDs where a configuration change causes some RED tunnels to reload (that of course causes traffic to stop passing while that happens), however, for the logs, you provided I don't see you are being affected by that.
Were you able to do the change of Firmware and do the tcpdump test, when the issue is happening?
I do. 03696572. We are watching closely but I think we made some progress last night.
Some progress sounds good. We got an identical issue woth RED SD 60 3.0.004 and XGFirewall
18.0.4.MR 4
Only power off helps solving the issue every morning
Regards
Chris
Thank you for the Case ID, I see your case is already escalated to GES, they have applied a patch to potentially help fix this, and they’re waiting on your feedback.
Hello Christopher,
I see my co-worker Harsh, asked you to share your Case ID with him, please do so, so he can follow-up accordingly.
We have the exact same issue, XG125 running SFOS 18.0.4 MR-4 and RED SD 60 running 3.0.0.004. I opened a support case as well, case number 03708276. Can we get in line for the patch? We don't want to replace the device with something else if a patch will resolve the issue.
Hello Jason,
Thank you for the Case ID, I have left a note in the case asking the engineer to escalate the case so you can get the patch applied.