VLAN and Virtual XG Behaviors

Question

So I'll start off by letting you know I'm currently running pfSense, with no issue. The configuration is set as expected. I'm finding some odd behavior though in trying to MATCH the configuration with the Sophos XG. I can validate that with the UTM9 platform, I'm able to see traffic behave as expected as well. But for some reason, the XG is causing some very odd results, and not working in setting up my VLAN's appropriately.

2 Main issues I've run int:

I have an Asus RT-N66U running Tomato Shibby firmware. This enables the VLAN tagging/trunking. I've setup Port1 as Trunk with VLAN 10/12. On XG, I've configured Port1 with VLAN sub-interfaces (10/12). If I connect Port1 to LAN (vPortGroup, which is on a vSwitch with 1 uplink to Port1 of Asus), there is no connectivity. I can not from a device on the VLAN's ping the interfaces (either the main Port1 or Port1.10/Port1.12).
Using a virtual environment, I've added 3 vNIC's to the VM (VM-Net/Wan/DMZ). My goal when setup properly, is to attach the LAN network in place of VM-Net. For testing and connectivity though (due to issues above), I've added a 4th vNIC connected to Lan. If I set Port1(VM-Net) and Port3(LAN), I can ping both IP's on the interfaces. If I switch Port1(LAN) and Port3(VM-Net), I can not reach EITHER interface IP's.

My setup with some basic outline below:

Cable Modem <---> vmNIC1 ESXi <---> vSwitch3 <---> vPortGroupWAN <---> <[XG Firewall]> <---> vPortGroupLAN [VLAN 4095 (trunk)] <---> vSwitch2 <---> Asus Port1 (VL10/12 Trunk) <---> br0 (VL10) Local LAN ports & WiFi / br1 (VL12) Guest WiFi

In VMware, the way to pass a trunk port through to a vNIC interface, is to set the PortGroup VLAN to 4095 for passing ALL VLAN's into the VM. This essentially leaves all TAGS intact, and passes them through to the attached vNIC interfaces. As mentioned above, this works perfectly fine with pfSense where I've setup the LAN/GUEST interfaces to VLAN 10/12 - the configuration on the Asus is sending packets being flagged properly with the correct VLAN tags and processing traffic appropriately. And the end results of segregating the two subnets through my pfSense box (aka strong firewall, qos, snort, etc rules) is working perfectly.

I'd heard a lot of people talking about how XG isn't ready and stick with the UTM 9 instead. I decided to attempt seeing if the same issue existed there, and it looks at a base level like the issues is not present in UTM9. I did a quick setup, configured VLAN's, and tested communication from the UTM to the different VLAN subnets, and it seems to work ok. I'd rather not have to go UTM9, as the interface is getting outdated, and likely will be a retired product in just a few short years. XG is clearly the future, but I'm EXTREMELY frustrated at this point not being able to switch over.

Any help is GREATLY appreciated. Also if there are ways I can further isolate/troubleshoot to find exactly where the issue is, that would be helpful as well.

This thread was automatically locked due to age.

pumpkinpower · Answer

Over the last few days I have finally got some time to try a migration from the UTM9 which I have used and loved for some years now to the new XG Firewall. I too have the same setup as you with the UTM and XG running in ESX and found the same problem but was aware of using 4095 to send all the vlans through to the UTM. What I did was to resolve this in the short term was leave Port1 as a dedicated Out of Band management port, Port 2 as WAN and Port 3 as all my network vlans which worked perfectly. Up until this point I was also getting very frustrated because whilst I could add all the vlans onto Port1 the result was there was NO WAN connectivity at all, I couldn't find a reason why. 
 Rebuilding the firewall VM again, I did it as default and vanilla as I could and everything worked with a single WAN internet connection and single non vlan Port 1 LAN network. As soon as you add a vlan to Port 1, WAN was dead. This is not how the UTM9 worked as I have WAN, LAN (all vlans) only and it was perfect, this took me a while to work out. Putting all vlans on Port3 had the same effect and no problems encountered. 
 Hope that helps, not ideal it means wasting a port but you dont have to bind Port 1 to a physical vmnic via a vswitch anyway so you can just ignore it in a way.