[9.194-5][BUG]Intel NIC crashes under load

This possibly a duplicate report.
I have been trying to do some performance reporting using two different sets of hardware built with the latest ISO.

One the NIC goes off line under load, kernel log shows a problem. BarryG advises there is an issue with the current driver.

I would provide the log entries, but that period of testing in now full of *, wrong UTM. Will need to bring the other UTM up to extract the log entries.

I changed the the NIC for another of the same type and model and got the same results.

Ian


2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   TDH                  
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   TDT                  
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   next_to_use          
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   next_to_clean        
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] buffer_info[next_to_clean]:
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   time_stamp           
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   next_to_watch        
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   jiffies              
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   next_to_watch.status 
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] MAC Status             
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] PHY Status             
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] PHY 1000BASE-T Status  
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] PHY Extended Status    
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] PCI Status             
2014:02:08-14:37:57 Cats-speed kernel: [ 2179.944779] e1000e 0000:00:19.0 eth0: Reset adapter unexpectedly
2014:02:08-14:38:00 Cats-speed kernel: [ 2183.235199] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

Parents

0 Tinkerbell over 10 years ago

Hi Barry/Ian/Martin,

I know this issue is here for a while. I thought would be fixed by the driver update in mantis id 29775 [ https://community.sophos.com/products/unified-threat-management/astaroorg/f/81/t/65355 ], which is already in the current ISO.
@Ian can you please provide "lspci -tv" output and what kind of appliance you are using? I am gathering all the infos in the mantis id.

Thanks guys for checking!
Bianca
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

0 ECKEROTH over 10 years ago in reply to Tinkerbell

Hi Bianca

Short status:
TSO  = OFF
LRO  = OFF
GRO  = ?
ASPM = ?

GOOGLE:\> network ASPM e1000e -> 1570 results [:O]

fw:/sbin/ethtool -k eth1 (e1000e)



Features for eth1:

rx-checksumming: on

tx-checksumming: on

        tx-checksum-ipv4: off [fixed]

        tx-checksum-ip-generic: on

        tx-checksum-ipv6: off [fixed]

        tx-checksum-fcoe-crc: off [fixed]

        tx-checksum-sctp: off [fixed]

scatter-gather: on

        tx-scatter-gather: on

        tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: off

        tx-tcp-segmentation: off [requested on]

        tx-tcp-ecn-segmentation: off [fixed]

        tx-tcp6-segmentation: off [requested on]

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: off [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: on

loopback: off [fixed]

rx-fcs: off

rx-all: off

And I don't have 20-nic.rules in /lib/udev/rules.d/

// Martin

Reply

0 ECKEROTH over 10 years ago in reply to Tinkerbell

Hi Bianca

Short status:
TSO  = OFF
LRO  = OFF
GRO  = ?
ASPM = ?

GOOGLE:\> network ASPM e1000e -> 1570 results [:O]

fw:/sbin/ethtool -k eth1 (e1000e)



Features for eth1:

rx-checksumming: on

tx-checksumming: on

        tx-checksum-ipv4: off [fixed]

        tx-checksum-ip-generic: on

        tx-checksum-ipv6: off [fixed]

        tx-checksum-fcoe-crc: off [fixed]

        tx-checksum-sctp: off [fixed]

scatter-gather: on

        tx-scatter-gather: on

        tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: off

        tx-tcp-segmentation: off [requested on]

        tx-tcp-ecn-segmentation: off [fixed]

        tx-tcp6-segmentation: off [requested on]

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: off [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: on

loopback: off [fixed]

rx-fcs: off

rx-all: off

And I don't have 20-nic.rules in /lib/udev/rules.d/

// Martin

Children

0 Tinkerbell over 10 years ago in reply to ECKEROTH

And I don't have 20-nic.rules in /lib/udev/rules.d/

// Martin

Martin ,

The file is in /etc/udev/ not in lib/....
Just run a vim /etc/udev/rules.d/20-nic.rules and open directly the file.
thx
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel
0 ECKEROTH over 10 years ago in reply to Tinkerbell
Martin ,

The file is in /etc/udev/ not in lib/....
Just run a vim /etc/udev/rules.d/20-nic.rules and open directly the file.
thx

Hi Bianca

Found it: TSO is OFF. Do you want me to enable?

When trying to run nic-rules command i've get this error:
fw:/home/login # /lib/udev/nic-disable-tso
/lib/udev/nic-disable-tso: line 8: DEVPATH: parameter null or not set
fw:/home/login # /lib/udev/nic-disable-gro
/lib/udev/nic-disable-gro: line 7: DEVPATH: parameter null or not set
fw:/home/login #

// Martin
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel
0 Tinkerbell over 10 years ago in reply to ECKEROTH

Hi martin,

no no dont enable tso.

gro and gso are shown as ON from the output. Lets disable it and run the load test again.
please run: /sbin/ethtool -K eth1 gso off gro off

i saw eth 1 is the one that crashes on ure machine.
the whole purpose is to have those parameters mentioned in my post disabled.

best
bianca
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

0 ECKEROTH over 10 years ago in reply to Tinkerbell

This is the output:

fw:/home/login # /sbin/ethtool -K eth1 gso off gro off

Actual changes:

tcp-segmentation-offload: on

        tx-tcp-segmentation: on

        tx-tcp6-segmentation: on

generic-segmentation-offload: off

generic-receive-offload: off

fw:/home/login #

Status on eth1 is now:

fw:/home/login # /sbin/ethtool -k eth1

Features for eth1:

rx-checksumming: on

tx-checksumming: on

        tx-checksum-ipv4: off [fixed]

        tx-checksum-ip-generic: on

        tx-checksum-ipv6: off [fixed]

        tx-checksum-fcoe-crc: off [fixed]

        tx-checksum-sctp: off [fixed]

scatter-gather: on

        tx-scatter-gather: on

        tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: off

        tx-tcp-segmentation: off

        tx-tcp-ecn-segmentation: off [fixed]

        tx-tcp6-segmentation: off

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: off

generic-receive-offload: off

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: off [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: on

loopback: off [fixed]

rx-fcs: off

rx-all: off

fw:/home/login #

// Martin