XG Home Edition on ESXi 6.7 with 4+ vNICs - NICs order

 Hello!

Im trying to install Sophos XG on my ESXi 6.7 using ISO (as OVA has some issues on my ESXi when trying to import VM from it). I need 6 interfaces so I added 6 network adapters to VM. Installation finished without any issues and also I was able to finish Initial Setup. But after reboot XG stops to answer on both LAN and WAN interfaces (other networks does not contain any hosts yet so I checked only LAN and WAN).

I found that XG works well if VM has only 3 interfaces. If add one more this issue appears again.

Is there any kind of port count limit for Home Edition or what else may cause this?

  • Found the root of the issue - VMXNET3 adapters. After swithing to E1000 issue was been fixed.

  • In reply to Ivan Osipov:

    Hi Ivan,

    That's very odd, VMXNET3 are the recommended virtual hardware and not E1000 due to performance and stability.

    I would recommend being logged in via the console and doing "ifconfig eth0" and working out what happens to them, you may find it could be the nic ordering is just getting messed up. Compare the mac address from the ifconfig to what you have assigned in the VMWare machine config.

    Emile

  • In reply to EmileBelcourt:

    Hi Emile,

    Thank you for your reply!

    Is there any real issue I can face if I leave E1000 as network adapters?

    I suspect that you are correct regarding NIC order issue as I found exactly same issue with pfSense. But actually I prefer to have NICs ordered in the same way in both XG and ESXi, so it is not an option anyway...

  • I found better fix then E1000:

    In ESXi go to Edit VM -> VM Options -> Advanced -> Edit Configuration

    Then switch values for ethernetX.pciSlotNumber parameters to reorder NICs. For my case (6 NICs) following fixed the ordering issue:

    Initially it was:

    ethernet0.pciSlotNumber = 192
    ethernet1.pciSlotNumber = 224
    ethernet2.pciSlotNumber = 256
    ethernet3.pciSlotNumber = 1184
    ethernet4.pciSlotNumber = 1216
    ethernet5.pciSlotNumber = 1248

    I changed it to following:

    ethernet0.pciSlotNumber = 1184
    ethernet1.pciSlotNumber = 192
    ethernet2.pciSlotNumber = 1216
    ethernet3.pciSlotNumber = 224
    ethernet4.pciSlotNumber = 1248
    ethernet5.pciSlotNumber = 256

    Also found that this is consist between VMs and *nix OS - same fix work for pfSense as well (at least for 6 NICs)

  • In reply to Ivan Osipov:

    It's well known VMXNET3  adapter are not ideal with firewalls ... And that's when they work.

    Same kind of problems with virtualized Checkpoint firewalls ...

    Paul Jr

  • In reply to Big_Buck:

    Isnt VMXNET3 a basic "Mainstream OS" network adapter?

    https://kb.vmware.com/s/article/1001805

    "Because operating system vendors do not provide built-in drivers for this card, you must install VMware Tools to have a driver for the VMXNET network adapter available"

    So basically you need working VMware Tools and also be always up2date.

    And Vmware most likely prioritize the most popular OS like Windows (server) and other servers like Debian etc. 

  • In reply to LuCar Toni:

    When you select VMXNET3, you will need VMWare Tools to install VMXNET3 drivers on the VM OS.  It has optimization features that are, at my opinion, meant mostly for Windows.  It is more stable and predictable with 1000 or 1000e.

    VMXNET3 on firewalls, proxies, and routers is asking for troubles with no real performance gain.

    Paul Jr

  • In reply to Big_Buck:

    Hi all,

    The XG has drivers for VMXNET3 as the XG has a versions of VMWare tools.

    E1000 is emulated and has a sizable overhead, VMXNET3 is a fully virtualised hypervisor NIC.

    Always use VMXNET3 unless you have compatibikity issues.

    See example below:

    VMXNET3 is not optimised better for one OS over another no more than a hardware Intel NIC (well some could be but thats by the by). It is the best NIC available for performance.

    "VMware: Do not use flexible NICs. We advise to avoid using e1000 and instead use VMXNET3. It is not recommended to add or remove vNICs after installation as this can result in changes in NIC order"

    From KB:

    Emile

  • In reply to EmileBelcourt:

    I do not care much what Google gives me when I'm faced with real life problems because of cybernetic champions' recommendations.

    VMXNET3 (VMXNET2, VMXNET) will be ok when it works, but I had so many problems on so many platform, that I do not bother to even try it anymore. HP, IBM, Dell, name it !!!  All of them.  And on all version of VMWare on which that appeared. It is just not worth the hassle.

    I repeat, VMXNET3 is an excellent way to find problems only for REAL-LIFE gains barely noticeable, if any.  Meaning I do not care what the chart says.  It's totally synthetic and artificial.  Much like CPU benchmarks.

    There's also doors open to well know memory hacking techniques because of all additional complexities brought by those "offloading" drivers.  These hacking technics have been recently dig-out and discussed.  In short, virtual environments are already not that safe.

    Oh.  And by the way. Good luck when you'll be moving your VM onto another hardware host.  I have hit the wall often with Checkpoint and PFSense.  They won't warn you about that on Google.

    Paul Jr

  • In reply to Big_Buck:

    Hello Paul,

    You are making quite serious claims based on anecdotal evidence.

    I have installed hundreds of VMs using VMXNET3 amd have never heard of the issues you are describing (that's my anecdotal evidence). Do you have any source material on the matter? I have read about some TCP issues in the past with certain versions of the linux kernel, anything recent?

    Of the two security issues i can remember, one was windows only and one has been patched which allowed VM escape.

    I do appreciate personal experience but i vastly appreciate research more, even though i understand you don't give a hoot about tests and graphs:

    ^that's got some intetesting pointers

    ^he even did a cycle/perf comparison.

    If you deploy E1000, you are limiting the performance unnecessarily. I can find little in the way of compatibility issues and only have had issues on unstable hosts not the guests themselves.

    I feel like this is going to be an immovable object vs unstoppable force situation however I would like to point out VMWare officially recommend VMXNET3 on top of this.

    Emile

  • In reply to EmileBelcourt:

    I too have installed hundreds on very different hardware/OS platforms for close to 4 decades.  I virtualized before VMWare existed.  Heck !!! I was learning to virtualize before Microsoft existed !!!  On PDP-11 and CDC-835.

    The point I bring here is not a piss contest.  Problems with VMXNET3 are just real.  And known.  And up to recently, documentations from Checkpoint and Symantec were crystal clear and would recommend to use 1000 or 1000E for compatibility and security reasons. Only recently, since version 10.6, Symantec Messaging Gateway would start support for VMXnet3.  And of course, since then, bugs raised.  And sure enough, the workaround was to go back with e1000e.  Since version 10.7.1, few weeks ago, things are back to normal apparently.

    That said. You don't get problems with with VMXNet after Windows 2012.

    On other platforms, VMXNet, whatever speed gain you will get, was up to recently a common source of problems.  Not just an anecdote.

    VMWare may recommend this, but in the end, the only important recommendation comes from the supplier. Sophos, CheckPoint, Symantec and all.

    Very recently VMXnet had some security bulletins, so sweet, that allows a hacker to escape the VM, and run commands on the host.  That was only few months ago ...  Any other virtual adapters were not affected.

    I would not recommend blindly to use this.  Particularly when most virtual firewalls are overpowered ...

    Paul Jr

  • In reply to Big_Buck:

    Hi Paul,

    If we are taking vendors recommendations then Sophos recommends VMXNET3 and that has been the way for a while.

    As we are speaking the context of Sophos NSG appliances virtualised in VMWare then we should be recommending what the vendor does, as you say.

    Emile

  • In reply to EmileBelcourt:

    Yes, but from what I know, those Sophos recomendations date BEFORE vulnerabilities regarding VMXnet3 drivers where found.  Few months ago, I failed to find Sophos literature regarding this.  All I could find was regarding Astaro.  So ... way too old.  Maybe I should check again.

    Paul Jr Robitaille

  • In reply to EmileBelcourt:

    Hello Emile

    Have you got by chance found any statement or declaration from Sophos regarding VMXNet drivers vulnerabilities. (as a reminder "ies" here, means "many")  As well as SCSI Some of them were addressed starting at ESXi670-201811401-BG.  But I have yet to find any technical paper from Sophos confirming or clarifying anything regarding those critical issues.

     

    Paul Jr