XG Home Edition on ESXi 6.7 with 4+ vNICs - NICs order

Found the root of the issue - VMXNET3 adapters. After swithing to E1000 issue was been fixed.

Hi Ivan,

That's very odd, VMXNET3 are the recommended virtual hardware and not E1000 due to performance and stability.

I would recommend being logged in via the console and doing "ifconfig eth0" and working out what happens to them, you may find it could be the nic ordering is just getting messed up. Compare the mac address from the ifconfig to what you have assigned in the VMWare machine config.

Emile

Hi Emile,

Thank you for your reply!

Is there any real issue I can face if I leave E1000 as network adapters?

I suspect that you are correct regarding NIC order issue as I found exactly same issue with pfSense. But actually I prefer to have NICs ordered in the same way in both XG and ESXi, so it is not an option anyway...

I found better fix then E1000:

In ESXi go to Edit VM -> VM Options -> Advanced -> Edit Configuration

Then switch values for ethernetX.pciSlotNumber parameters to reorder NICs. For my case (6 NICs) following fixed the ordering issue:

Initially it was:

ethernet0.pciSlotNumber = 192
ethernet1.pciSlotNumber = 224
ethernet2.pciSlotNumber = 256
ethernet3.pciSlotNumber = 1184
ethernet4.pciSlotNumber = 1216
ethernet5.pciSlotNumber = 1248

I changed it to following:

ethernet0.pciSlotNumber = 1184
ethernet1.pciSlotNumber = 192
ethernet2.pciSlotNumber = 1216
ethernet3.pciSlotNumber = 224
ethernet4.pciSlotNumber = 1248
ethernet5.pciSlotNumber = 256

Also found that this is consist between VMs and *nix OS - same fix work for pfSense as well (at least for 6 NICs)

It's well known VMXNET3  adapter are not ideal with firewalls ... And that's when they work.

Same kind of problems with virtualized Checkpoint firewalls ...

Paul Jr

Isnt VMXNET3 a basic "Mainstream OS" network adapter?

https://kb.vmware.com/s/article/1001805

"Because operating system vendors do not provide built-in drivers for this card, you must install VMware Tools to have a driver for the VMXNET network adapter available"

So basically you need working VMware Tools and also be always up2date.

And Vmware most likely prioritize the most popular OS like Windows (server) and other servers like Debian etc. 

When you select VMXNET3, you will need VMWare Tools to install VMXNET3 drivers on the VM OS.  It has optimization features that are, at my opinion, meant mostly for Windows.  It is more stable and predictable with 1000 or 1000e.

VMXNET3 on firewalls, proxies, and routers is asking for troubles with no real performance gain.

Paul Jr

Hi all,

The XG has drivers for VMXNET3 as the XG has a versions of VMWare tools.

E1000 is emulated and has a sizable overhead, VMXNET3 is a fully virtualised hypervisor NIC.

Always use VMXNET3 unless you have compatibikity issues.

See example below:

VMXNET3 is not optimised better for one OS over another no more than a hardware Intel NIC (well some could be but thats by the by). It is the best NIC available for performance.

"VMware: Do not use flexible NICs. We advise to avoid using e1000 and instead use VMXNET3. It is not recommended to add or remove vNICs after installation as this can result in changes in NIC order"

From KB:

Emile

I do not care much what Google gives me when I'm faced with real life problems because of cybernetic champions' recommendations.

VMXNET3 (VMXNET2, VMXNET) will be ok when it works, but I had so many problems on so many platform, that I do not bother to even try it anymore. HP, IBM, Dell, name it !!!  All of them.  And on all version of VMWare on which that appeared. It is just not worth the hassle.

I repeat, VMXNET3 is an excellent way to find problems only for REAL-LIFE gains barely noticeable, if any.  Meaning I do not care what the chart says.  It's totally synthetic and artificial.  Much like CPU benchmarks.

There's also doors open to well know memory hacking techniques because of all additional complexities brought by those "offloading" drivers.  These hacking technics have been recently dig-out and discussed.  In short, virtual environments are already not that safe.

Oh.  And by the way. Good luck when you'll be moving your VM onto another hardware host.  I have hit the wall often with Checkpoint and PFSense.  They won't warn you about that on Google.

Paul Jr

Hello Paul,

You are making quite serious claims based on anecdotal evidence.

I have installed hundreds of VMs using VMXNET3 amd have never heard of the issues you are describing (that's my anecdotal evidence). Do you have any source material on the matter? I have read about some TCP issues in the past with certain versions of the linux kernel, anything recent?

Of the two security issues i can remember, one was windows only and one has been patched which allowed VM escape.

I do appreciate personal experience but i vastly appreciate research more, even though i understand you don't give a hoot about tests and graphs:

^that's got some intetesting pointers

^he even did a cycle/perf comparison.

If you deploy E1000, you are limiting the performance unnecessarily. I can find little in the way of compatibility issues and only have had issues on unstable hosts not the guests themselves.

I feel like this is going to be an immovable object vs unstoppable force situation however I would like to point out VMWare officially recommend VMXNET3 on top of this.

Emile