This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

XG VM crashes and performance

Gents,

3rd day running XG on ESXi 6.7, and after dealing with licensing issue, let me ask another set of newbie questions:
1) XG VM is installed on SSD datastore, with 4 vCPUs and 8 GB or RAM, 4x VXMNET3 vNICs (which is recommended NIC type AFAIK). CPU never goes beyond 20%, memory (according to XG stats) 54%.

2) as soon as I increase LAN interface MTU to 9000 - XG  crashes / reboots in cycle. Nothing in /var/cores, syslog before reboot only shows tons of
Mar 3 10:16:36 (none) user.err kernel: [ 357.728901] 996.769636 [ 317] vmxnet3_netmap_rxsync 155 skipped! rx_idx 256
Mar 3 10:16:36 (none) user.err kernel: [ 357.728901] 996.769636 [ 317] vmxnet3_netmap_rxsync 156 skipped! rx_idx 256

Why MTU change kills it ? And where to find crash logs, as they appear on VM screen only momentarily, I can't really capture any details

3) Performance is kind of low. Speedtest from laptop shows 100Mbps down, 80-90 up, but PPPoE line is 600Mbps. (ESXi host is good enough, even heavy-weight PANOS VM gets nearly linerate performance on it). Initial access to  sites is damn slow, it looks like when I first open website - it takes like 10 seconds or more to load, but then navigation gets better. Any hints/best practices how to improve XG throughput/performance ?



BR,
Alex



This thread was automatically locked due to age.
  • Hi Alex,

    all you figures seem very high, what is the cpu? With a new installation you should be running about  30 to 40% ram. Sounds like a configuration issue. My hardware system runs 55% ram and idles about 5% cpu but does get excited occasionally.

    did you lock the resources to the XG only?

    XG115W - v20 GA - Home

    XG on VM 8 - v20 GA

    If a post solves your question please use the 'Verify Answer' button.

  • Well, in average idle state it's 3% CPU (20% is max value VMWare ever saw) and 54% RAM
    Host CPU is 4 CPUs x Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz, and apart from XG runs few low-volume Linux workloads, so host CPU avg usage is 5%. Host RAM is 65% free, so resources shouldn't be an issue 

  • Hi Alex,

    the XG requires exclusive use of CPU, RAM and disk. If it is sharing then you will have performance and stability issues.

    Ian

    XG115W - v20 GA - Home

    XG on VM 8 - v20 GA

    If a post solves your question please use the 'Verify Answer' button.

  • ,

    I reinstalled VM using recommended Home Edition image, hard-locked CPU and RAM resources for it, but that hasn't changed anything in jumbo frame-related FW crash/reboot. My subjective observation - it depends on traffic level, i.e. in "live" subnet FW reboots even before Web UI comes up, in default 172.16.16.0 "empty" subnet it stays up long enough to allow MTU change from GUI, although if you don't touch it - crashes/reboots anyway. 
    I'd really love Sophos to check that issue, jumbo frame support is pretty much baseline these days, and in any cases FW should go nuts about it and die ;) 

    In any case, it appears to be VXNET3-related, so question: will there be any performance penalty if I switch to E1000e vNIC type ?


    This time I used serial port to log crash so I can add full brach log here:
    [ 108.003919] watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [swapper/2:0]
    [ 108.005107] Modules linked in: nf_conntrack_ipslb nfnetmap_queue(O) xt_xfrmpolicy ah4 vfp_firewall(O) debug_cntrs(O) xt_addrtype nf_nat_ftp nf_conntrack_ftp xt_CT ebtable_filter ebtable_nat ebtables ip6t_MASQUERADE xt_muser xt_conntrack xt_LBS ip6table_filter iptable_filter xt_DNAT xt_SNAT nf_nat_masquerade_ipv6 xt_nat_lookup xt_UST xt_ust xt_firewall nat_rules sfos_rules_framework firewall ip_set_hash_mlmwsticky ip_set_hash_sslvpn iptable_mangle ip_set_hash_mac ip_set_hash_bw nf_conntrack_dns nf_nat_sip nf_conntrack_sip nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_h323 nf_conntrack_h323 nf_nat_pptp nf_conntrack_pptp sunkbd xtkbd usbhid hid_generic hid ohci_pci ohci_hcd xhci_pci xhci_hcd uhci_hcd ehci_pci ehci_hcd fw_handle_ngfw_notification fp2sp_api fp_notifier bonding cifs
    [ 108.015979] red red2 appdev nf_conntrack_netlink nf_nat_proto_gre nf_conntrack_proto_gre set_sessiontbl sessiontbl ip_gre gre ipcomp xfrm_ipcomp esp4 xfrm4_mode_transport xfrm4_mode_tunnel xfrm4_tunnel xfrm_user af_key xfrm_algo pcbc aesni_intel glue_helper aes_x86_64 crypto_simd cryptd cls_u32 act_mirred sch_ingress ifb sch_hfsc sch_leafprio sch_headprio sch_sfq sch_htb xt_MULTISET xt_MLM xt_SRCNETMAP xt_MARKROUTE xt_CONTINUE xt_LOGDROP xt_ULOG xt_TCPMSS xt_REDIRECT nf_nat_redirect ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_OUT_OUTDEV ip6t_rpfilter ipt_rpfilter ebt_nflog ebt_pkttype xt_serviceset xt_appset xt_hostset xt_pkttype xt_recent xt_state xt_status xt_cet xt_OUTDEV xt_iprange xt_limit xt_hashlimit xt_tcpudp xt_multiport nf_conntrack_relate xt_IPMACFILTER xt_RANGENAT xt_VHDNAT ip_set_bitmap_vhost
    [ 108.027137] xt_FWSET xt_set ip_set_hash_maciface_fp ip_set_hash_ipiface_fp ip_set_bitmap_hotspotuser ip_set_hash_hotspotmac ip_set_bitmap_tlsrule ip_set_bitmap_appset ip_set_bitmap_fwrule ip_set_bitmap_ctrxss ip_set_bitmap_user sp2fp_api ip_set_bitmap_userpolicy ip_set_hash_ipuser ip_set_bitmap_service ip_set_bitmap_host ip_set_hash_ipmaciface ip_set_hash_l2mac ip_set_hash_ipmac ip_set_hash_ip ip_set arptable_filter arp_tables via_velocity crc_ccitt via_rhine tehuti sunhme sungem sungem_phy niu cassini stmmac_pci dwmac_generic stmmac_platform stmmac sis900 sis190 sc92031 sfc r8169 r8168 8139too 8139cp r6040 qlge qlcnic qla3xxx netxen_nic yellowfin hamachi forcedeth vxge s2io ns83820 natsemi myri10ge ksz884x ks8851_mll mlx4_en mlx4_core devlink sky2 skge jme ixgbevf ixgbe_nm(O) ixgb igbvf igb_nm(O)
    [ 108.038286] i40e_nm(O) e1000e_nm(O) e1000_nm(O) e100 hp100 fealnx ethoc be2net dnet sundance dl2k winbond_840 uli526x dmfe de4x5 de2104x enic bna tg3 cnic bnx2x zlib_inflate bnx2 b44 libphy atl2 atl1 atl1e atl1c pcnet32 amd8111e acenic ne2k_pci 8390 typhoon 3c59x tulip cn virtio_net_nm(O) i2c_algo_bit i2c_core hwmon ptp pps_core mdio vmxnet3_nm(O) netmap(O) ip6table_nat nf_nat_ipv6 ip6table_mangle ip6table_raw iptable_nat iptable_raw nf_nat_ipv4 xt_dscp nf_nat ip6_tables ip_tables tun af_packet 8021q nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 ip6_tunnel tunnel6 sit ip_tunnel tunnel4 ppdev parport_pc parport nf_conntrack lineartable bitmap_api br_netfilter bridge nf_defrag_ipv4 ipv6 stp llc x_tables nfnetlink button evdev [last unloaded: nfnetmap_queue]
    [ 108.049216] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O 4.14.38 #2
    [ 108.050418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
    [ 108.052169] task: ffff8801b8937000 task.stack: ffff8801b8954000
    [ 108.053129] RIP: 0010:console_unlock+0x279/0x440
    [ 108.053857] RSP: 0000:ffff8801bfd03ca8 EFLAGS: 00000283 ORIG_RAX: ffffffffffffff10
    [ 108.055068] RAX: 0000000000000000 RBX: 00000000033f8898 RCX: ffffffff82038018
    [ 108.056245] RDX: 0000000000000001 RSI: 0000000000000082 RDI: 0000000000000283
    [ 108.057393] RBP: 0000000000000283 R08: 00000000033f8898 R09: 0000000000000002
    [ 108.058544] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff820d9ff0
    [ 108.059729] R13: 0000000000000001 R14: 0000000000000000 R15: 00000000033f8898
    [ 108.060862] FS: 0000000000000000(0000) GS:ffff8801bfd00000(0000) knlGS:0000000000000000
    [ 108.062154] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
    [ 108.063096] CR2: 00007f8eb68a9880 CR3: 0000000002018005 CR4: 00000000001606e0
    [ 108.064374] Call Trace:
    [ 108.064786] <IRQ>
    [ 108.065123] vprintk_emit+0x327/0x390
    [ 108.065727] printk+0x3e/0x46
    [ 108.066243] ? __getnstimeofday64+0x36/0xc0
    [ 108.066961] 0xffffffffa01ff67d
    [ 108.067476] netmap_ioctl_legacy+0x5d5/0x690 [netmap]
    [ 108.068348] 0xffffffffa0202082
    [ 108.068887] ? netmap_notify+0x15/0x7b0 [netmap]
    [ 108.069640] ? 0xffffffffa0200313
    [ 108.070151] ? 0xffffffffa0202082
    [ 108.070705] 0xffffffffa0202f5c
    [ 108.071210] net_rx_action+0xd8/0x270
    [ 108.071800] __do_softirq+0xcb/0x1f8
    [ 108.072454] irq_exit+0xb9/0xc0
    [ 108.072966] do_IRQ+0x76/0xc0
    [ 108.073427] common_interrupt+0x7a/0x7a
    [ 108.074041] </IRQ>
    [ 108.074427] RIP: 0010:native_safe_halt+0x2/0x10
    [ 108.075192] RSP: 0000:ffff8801b8957f20 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff4e
    [ 108.076407] RAX: 0000000080000000 RBX: ffffffff820d9ff0 RCX: 000000000001b380
    [ 108.077539] RDX: 0000000000047a32 RSI: ffff8801bfd1b380 RDI: 0000000000000000
    [ 108.078607] RBP: ffff8801b8937000 R08: 0000000000000002 R09: 000000000002f780
    [ 108.079712] R10: ffff8801b8957eb0 R11: 0000000000000001 R12: 0000000000000000
    [ 108.080907] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
    [ 108.082103] default_idle+0x5/0x10
    [ 108.082655] do_idle+0x94/0x100
    [ 108.083193] cpu_startup_entry+0x14/0x20
    [ 108.083890] secondary_startup_64+0xa5/0xb0
    [ 108.084631] Code: 48 c7 c7 2c 38 74 82 e8 b6 19 76 00 4c 8b 3d bf 30 70 01 48 8b 1d c8 30 70 01 c6 05 bd 03 6c 01 00 e8 2c 1d 00 00 48 89 ef 57 9d <0f> 1f 44 00 00 49 39 df 74 16 e8 98 f9 ff ff 85 c0 75 0d e8 cf
    [ 108.087582] Kernel panic - not syncing: softlockup: hung tasks
    [ 108.088538] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G O L 4.14.38 #2
    [ 108.089624] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
    [ 108.091242] Call Trace:
    [ 108.091654] <IRQ>
    [ 108.091977] dump_stack+0x5c/0x78
    [ 108.092553] panic+0xd0/0x228
    [ 108.093068] watchdog_timer_fn+0x219/0x220
    [ 108.093721] __hrtimer_run_queues+0xab/0x110
    [ 108.094399] hrtimer_interrupt+0xa1/0x1d0
    [ 108.095065] smp_apic_timer_interrupt+0x4c/0x80
    [ 108.095780] apic_timer_interrupt+0x7a/0x80
    [ 108.096419] RIP: 0010:console_unlock+0x279/0x440
    [ 108.097183] RSP: 0000:ffff8801bfd03ca8 EFLAGS: 00000283 ORIG_RAX: ffffffffffffff10
    [ 108.098436] RAX: 0000000000000000 RBX: 00000000033f8898 RCX: ffffffff82038018
    [ 108.099572] RDX: 0000000000000001 RSI: 0000000000000082 RDI: 0000000000000283
    [ 108.100724] RBP: 0000000000000283 R08: 00000000033f8898 R09: 0000000000000002
    [ 108.101895] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff820d9ff0
    [ 108.103015] R13: 0000000000000001 R14: 0000000000000000 R15: 00000000033f8898
    [ 108.104142] vprintk_emit+0x327/0x390
    [ 108.104732] printk+0x3e/0x46
    [ 108.105207] ? __getnstimeofday64+0x36/0xc0
    [ 108.105949] 0xffffffffa01ff67d
    [ 108.106439] netmap_ioctl_legacy+0x5d5/0x690 [netmap]
    [ 108.107254] 0xffffffffa0202082
    [ 108.107770] ? netmap_notify+0x15/0x7b0 [netmap]
    [ 108.108521] ? 0xffffffffa0200313
    [ 108.109084] ? 0xffffffffa0202082
    [ 108.109621] 0xffffffffa0202f5c
    [ 108.110120] net_rx_action+0xd8/0x270
    [ 108.110697] __do_softirq+0xcb/0x1f8
    [ 108.111271] irq_exit+0xb9/0xc0
    [ 108.111758] do_IRQ+0x76/0xc0
    [ 108.112245] common_interrupt+0x7a/0x7a
    [ 108.112832] </IRQ>
    [ 108.113192] RIP: 0010:native_safe_halt+0x2/0x10
    [ 108.113970] RSP: 0000:ffff8801b8957f20 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff4e
    [ 108.115175] RAX: 0000000080000000 RBX: ffffffff820d9ff0 RCX: 000000000001b380
    [ 108.116384] RDX: 0000000000047a32 RSI: ffff8801bfd1b380 RDI: 0000000000000000
    [ 108.117477] RBP: ffff8801b8937000 R08: 0000000000000002 R09: 000000000002f780
    [ 108.118595] R10: ffff8801b8957eb0 R11: 0000000000000001 R12: 0000000000000000
    [ 108.119661] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
    [ 108.120788] default_idle+0x5/0x10
    [ 108.121313] do_idle+0x94/0x100
    [ 108.121826] cpu_startup_entry+0x14/0x20
    [ 108.122476] secondary_startup_64+0xa5/0xb0
    [ 108.123197] Kernel Offset: disabled
    [ 108.125262] Rebooting in 3 seconds..
    [ 111.147982] ACPI MEMORY or I/O RESET_REG.

  • Did complete reinstall and manual reconfiguration - same crap.
    Changed vNIC type to E1000e - at least it doesn't crash now, but performance is still 110-130 Mbps

  • Hello there,

    When you make the change do you see something under /var/cores?

    Regards,


     
    Emmanuel (EmmoSophos)
    Technical Team Lead, Global Community Support
    Sophos Support VideosProduct Documentation  |  @SophosSupport  | Sign up for SMS Alerts
    If a post solves your question use the 'Verify Answer' link.
  • Emmanuel,

    no, there are no core dumps. 
    Reboot is driven by kernel soft lock, and by default linux doesn't run kdump for it.