XG managed APX 740 APs randomly going offline and dropping all clients

Hello Jamie,

can you please specify wich fixed channels do you selected?

Kind Regards,

Suzzyx

Hi Suzzyx,

for 2ghz (3 APs):

1, 6, 11

for 5ghz (2 APs):

36, 44

side note: I could re-enable auto selection (I only ever used auto on 5ghz) and see if it happens.  Still no new AP offline entries in the log.

Could you show us a little network diagram with your APs? 

Are you using VLANs or are the APs directly attached? 

Happened again:

Yes I can.  It’s a small test network with roughly 50 devices, most wireless.  I’m not using VLANs although I have a test VLAN configured on all 48 switch ports (tagged) and a VLAN interface on the firewall but it’s not used (it is configured on the only LAN interface I have however).  The DHCP network for that VLAN is turned off as well.  Much of this was in place to test central wireless.  I’ll put a straw diagram together today and post.

didn’t get a chance to toss a diagram together. I will tomorrow. Today got unexpectedly busy.  But, it happened again today (that’s twice now today)

Again

The switch port the AP is connected to is bouncing when these entries occur in the firewall log. No errors on the switch.  Appears the AP is actually reloading but I’ll move switch ports later just in case and the replace the cable although this happens to the other AP as well, it doesn’t happen near as frequent. This the the log from my switch:

I moved from g17 to g18 on the switch FYI 

Ian,

I went back at your original suggestion regarding DHCP and noticed the lease settings for the APs are 24 hours and correlate to the times they go offline.  Would it be better to statically reserve these or elongate the lease period? Or is there something unexpected going on here? I would not expect the AP to lose connection when it renews it’s IP lease.  Come to think of it, the WAN interface on the XG sometimes does the same thing.

No symptoms or log messages since I statically reserved the IPs in DHCP.  Last event was 1/13.  Still too soon for me to be "comfy" though.

Network drawing (basic):

Hi James,

at one stage there was an issue with the DHCP server, so I blew all my lease times out and I do make my IPs use a static assignment.

The WAN interface sounds like your ISP might be performing network maintenance, the changes occur at night?

Ian

I do not have any of the APX series APs only the previous models.

Hi James,

at one stage there was an issue with the DHCP server, so I blew all my lease times out and I do make my IPs use a static assignment.

The WAN interface sounds like your ISP might be performing network maintenance, the changes occur at night?

Ian

I do not have any of the APX series APs only the previous models.

I found the culprit on the WAN side and it turns out there's a node problem in the neighborhood.  so, you're right, it's carrier related but it was one of those deals where they didn't know there was a problem in their infrastructure.

I have not had an issue since I statically reserved the IPs for the APX APs but I'd like to see a few more days go by to feel "good" if that makes sense.  I find it odd that statically reserving solves this because it's still a renew, just ensuring the same IP.  Seems it's masking an underlying issue.  I may do the same as you across the DHCP scope.

Just my 2 cents right now.

Still solid uptime since the static reservations, but memory is now at 78%.  Concerning albeit unrelated to the AP issue.  Uptime is 10+ days.  Starts around 36% at boot.  I’ll watch it but worried there’s a memory leak at play.

Just happened to my disappointment.  There must be some logging on the access points that can be looked at.  I do not have power supplies for these access points (they doing ship with them) but I would expect a PoE related log entry on my switch.  Otherwise, I’d move them to the ports directly on the XG to rule out the switch.  I would imagine Sophos would want to get to the bottom of this but it doesn’t feel like it.  May consider rolling back to 17 and see if this follows as I’m running out of options. Keep in mind, no other devices are doing this.  At first it was all access points.  Now, it’s only the access point that’s running the 5ghz SSID. A while back I removed the 5ghz SSID from the other APs because devices weren’t rehousing properly and I’d prefer the outage sadly.

System logs from the XG:

Switch logs show only that port bouncing (and this is a new port and cable to the wall jack):

no errors on the switch port (3rd column is error received count):

Hi James,

you are describing an overheating issue. Try turning the 2.4ghz SSID on the failing AP off.

Ian

I’ll give it a shot.  I’ll do it today.  Bit strange that using both radios causes them to overheat (I understand your point though).  The APs are in a well cooled area (sitting on a furniture surface - that’s it).  Given that all the APs were doing this when they all had both SSIDs enabled  and now only the AP that has both radios enabled is doing this, you could be on to something.  That’s a design flaw to me.

But, as I said, it’s worth a try to identify the source.  I’ll report back. Thx Ian.  

Done.  2 APs with only 2.4ghz and one AP with only 5ghz

Quick update:

10 days of uptime and frankly better AP Association by endpoints.  I’m going to call it identified and remediated after a month but not resolved as this is a product defect in my opinion.  This happened with the TX power turned down, too.  Honestly, the access point that is now dedicated to just 5ghz is running warm to the touch.  I now can’t run guest on all access points as 2+5 and I need additional APs to fill in 5ghz coverage gaps now. Nevertheless, it would appear you were correct, Ian.