Thread Info
  • Locked Locked
  • Replies 27 replies
  • Subscribers 10 subscribers
  • Views 29385 views
  • Users 0 members are here
Options
Suggested
This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

9.400 bricks connected APs!

ZdenekCizek

Hello,

just a heads up warning regarding the soft released 9.400.


Just installed it on the gateway and all connected APs that were using VLAN tagging and bridging communication to VLANs got bricked by the AP firmware update.

If the AP is not using VLAN tagging and wireless networks just use bridging to AP LAN then it works. If VLAN tagging is used then after the firmware update the AP never finishes booting and gets stuck while trying to get IP address from DHCP in an never ending loo (AP50). Got a confirmation from a customer where it bricked their AP30s. For AP50 and 30 it is possible to use the flash tool to get them back to working state. Unfortunately AP55c gets bricked totally as there is no flash tool for it and after it gets the 9.400 firmware update and is deployed with VLAN tagging enabled it basically dies and never finishes booting or does not even get to a point of requesting the IP address from DHCP and gets stuck rebooting over and over again.

After rolling back to 9.355 (restore from tape) and reflashing the APs using the recovery tool (when possible) wireless works fine.

Looks like some nasty bug that slipped through QA... again :(



This thread was automatically locked due to age.
Parents

  • Hello Zdenek,

    we just did some investigation regarding this regression you mentioned, and indeed there is a regression, but it is a bit different then you think it is. First of all the AP is probably not bricked (even if it looks like). What is broken is the fallback mechanism, which is probably used in your setup as default due to the connecting VLAN being untagged (coming later to it). When you configure your AP with the VLAN tagging it tries to connect the UTM over the specified VLAN, if it can not do so it will after some time fallback to a default LAN behaviour this means it contacts the UTM without using a VLAN tag. And this fallback is currently broken in the 9.400.

    If you use this vlantagging option and meanwhile configure your switch to use the specified VLAN with 'untag', the AP will first try to connect the UTM with the specified VLAN. This leads to the packets going out from the AP with the vlantag, the switch will forward them to the UTM, but as soon as the answer from the UTM comes to the switch it will get the vlantag removed. Thus the AP is not able to match the answer (without VLANTAG) to his request (with VLANTAG), then after some time the AP will go to the fallback and work in this fallback mechanism, the bridge 2 VLAN networks are not effected by this fallback, so they still work as expected, it is just the way the AP contacts the UTM.

    We are working on the fix for this regression, but for now if you want to get the APs back running which don't show up anymore you could provide the VLANTAG as 'tagged' from the switch to them, then they should come up working again.

    Regards,
    Emanuel

  • in reply to etaube

    Hello Emanuel,

    thank you for the response. The switch port the AP is connected to is configured with the AP control VLAN being both tagged and untagged on it plus the bridged VLANs are tagged there as well. So just like in past where it works, the AP should be able to reach the UTM via tagged or untagged way.

    With AP50 I can see that the AP boots and gets stuck in the infinite loop while trying to get IP from DHCP server (AP sends request, server provides offer but AP never acks it). When I deploy the AP50 without VLAN tagging and using just some testing SSID which bridges the traffic to AP LAN all works fine. AP connects to the UTM, flashes the firmware, reboots and wifi starts working. After I change the configuration to VLAN tagged setup it gets stuck in the DHCP cycle.

    However, the AP55 once the VLAN tagging config was pushed to it, it no longer finishes booting and does not even attempt to send DHCP request and just reboots over and over again in random intervals. Unfortunately there is no flash tool for the new APs so right now it is a brick.

    Right now I'm not able to do any more tests as I had to rollback the 9.400  back to 9.355 where all works as WiFi with VLANs is critical and I have to keep the business going.

    Cheers,

    Zdenek

  • in reply to ZdenekCizek

    Hello Zdenek,


    for my understanding, let's say you have the vlantag 500 configured as the AP control VLAN, and the switch has VLAN 500 tagged and untagged on the port the AP is plugged in. Now a packet with VLAN 500 arrives at the switch (which is for the AP) what does/should the switch do? Send the packet to the AP with the VLAN or send it without the VLAN? My expectation would be to send it without the VLAN.

    So my understanding is that if you have the switch configured VLAN 500 as untagged and tagged then both is allowed, so when the AP sends a packet with vlantag 500 it gets forwarded to the utm and when the AP sends a packet without any vlantag it also gets forwarded to the utm, but for the answer path there needs to be a decision. So I would expect the behaviour previously described with the fallback. (which is broken since 9.400)

    Anyways the problem you describe with the AP55 sounds a bit different since it does not even send the DHCP request on the VLAN, we will continue looking into that one. Was the AP55 also configured in 9.35 and then after update it went in the state, or did you configure it directly in 9.4?


    Regards,

    Emanuel

  • in reply to etaube

    Sorry for late reaction., The AP55 was not connected to 9.3. It has been connected directly to 9.400 (it was a fresh AP55 out of the box). It appeared in the pending list, was accepted, firmware update was pushed to it and it never came back up.

Reply Children
No Data
Unfiltered HTML