[9.194-5][BUG]Intel NIC crashes under load

This possibly a duplicate report.
I have been trying to do some performance reporting using two different sets of hardware built with the latest ISO.

One the NIC goes off line under load, kernel log shows a problem. BarryG advises there is an issue with the current driver.

I would provide the log entries, but that period of testing in now full of *, wrong UTM. Will need to bring the other UTM up to extract the log entries.

I changed the the NIC for another of the same type and model and got the same results.

Ian


2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] e1000e 0000:00:19.0 eth0: Detected Hardware Unit Hang:
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   TDH                  
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   TDT                  
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   next_to_use          
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   next_to_clean        
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] buffer_info[next_to_clean]:
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   time_stamp           
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   next_to_watch        
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   jiffies              
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536]   next_to_watch.status 
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] MAC Status             
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] PHY Status             
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] PHY 1000BASE-T Status  
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] PHY Extended Status    
2014:02:08-14:37:56 Cats-speed kernel: [ 2178.936536] PCI Status             
2014:02:08-14:37:57 Cats-speed kernel: [ 2179.944779] e1000e 0000:00:19.0 eth0: Reset adapter unexpectedly
2014:02:08-14:38:00 Cats-speed kernel: [ 2183.235199] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

Parents

0 Tinkerbell over 10 years ago

Hi Barry/Ian/Martin,

I know this issue is here for a while. I thought would be fixed by the driver update in mantis id 29775 [ https://community.sophos.com/products/unified-threat-management/astaroorg/f/81/t/65355 ], which is already in the current ISO.
@Ian can you please provide "lspci -tv" output and what kind of appliance you are using? I am gathering all the infos in the mantis id.

Thanks guys for checking!
Bianca
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Reply

0 Tinkerbell over 10 years ago

Hi Barry/Ian/Martin,

I know this issue is here for a while. I thought would be fixed by the driver update in mantis id 29775 [ https://community.sophos.com/products/unified-threat-management/astaroorg/f/81/t/65355 ], which is already in the current ISO.
@Ian can you please provide "lspci -tv" output and what kind of appliance you are using? I am gathering all the infos in the mantis id.

Thanks guys for checking!
Bianca
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

Children

0 ECKEROTH over 10 years ago in reply to Tinkerbell

Hi Bianca

Short status:
TSO  = OFF
LRO  = OFF
GRO  = ?
ASPM = ?

GOOGLE:\> network ASPM e1000e -> 1570 results [:O]

fw:/sbin/ethtool -k eth1 (e1000e)



Features for eth1:

rx-checksumming: on

tx-checksumming: on

        tx-checksum-ipv4: off [fixed]

        tx-checksum-ip-generic: on

        tx-checksum-ipv6: off [fixed]

        tx-checksum-fcoe-crc: off [fixed]

        tx-checksum-sctp: off [fixed]

scatter-gather: on

        tx-scatter-gather: on

        tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: off

        tx-tcp-segmentation: off [requested on]

        tx-tcp-ecn-segmentation: off [fixed]

        tx-tcp6-segmentation: off [requested on]

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: on

generic-receive-offload: on

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: off [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: on

loopback: off [fixed]

rx-fcs: off

rx-all: off

And I don't have 20-nic.rules in /lib/udev/rules.d/

// Martin

0 BarryG over 10 years ago in reply to Tinkerbell

I know this issue is here for a while. I thought would be fixed by the driver update in mantis id 29775 [ https://community.sophos.com/products/unified-threat-management/astaroorg/f/81/t/65355 ], which is already in the current ISO.

Hi Bianca,

Are you saying I need to re-install from the 9.19x ISO, or is installing the up2dates after installing from the 9.165 ISO OK?

Thanks,
Barry
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel
0 RFCat_vk_01 over 10 years ago in reply to BarryG

Hi Barry,
I built my two utms I was testing ips on from the latest ISO and both misbehaved.

Bianca, I am logged in as root and can't get this stuff to run.
No ethtool in /usr/sbin.
While I have the 20-nic.rules in your specified directory, I can't make them execute. Permissions denied.

I will connect a display and keyboard directly into the box and try again.
My UTM that I am doing this on is
Intel i3-4130
114gb ssd
Onboard Intel ethernet controller
Intel gigabit CT desktop in the expansion slot.
The motherboard is Gigabyte GA-Z87N-wifi mini itx with dual onboard NICs.

8gb ram

Ian

I have attached 3 txt files with the output from the commands
- ether.txt
- View
- Hide
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel
0 Tinkerbell over 10 years ago in reply to BarryG

Hi Bianca,

Are you saying I need to re-install from the 9.19x ISO, or is installing the up2dates after installing from the 9.165 ISO OK?

Thanks,
Barry

The Intel driver updates are merged since 9.190.
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel
0 Dr. ECKEROTH over 10 years ago in reply to RFCat_vk_01

Hi Barry,
I built my two utms I was testing ips on from the latest ISO and both misbehaved.

Bianca, I am logged in as root and can't get this stuff to run.
No ethtool in /usr/sbin.
While I have the 20-nic.rules in your specified directory, I can't make them execute. Permissions denied.

I will connect a display and keyboard directly into the box and try again.
My UTM that I am doing this on is
Intel i3-4130
114gb ssd
Onboard Intel ethernet controller
Intel gigabit CT desktop in the expansion slot.
The motherboard is Gigabyte GA-Z87N-wifi mini itx with dual onboard NICs.

8gb ram

Ian

I have attached 3 txt files with the output from the commands

Hi Ian

ethtool is located in /sbin/ethtool
To make the 20-nic.rules exe you have do: chmod 755 20-nic.rules

// Martin
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel
0 RFCat_vk_01 over 10 years ago in reply to Dr. ECKEROTH

Hi Martin,
I saw you amended commands.
Thank you for the change cmd which I will try out shortly.

Ian
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel
0 Tinkerbell over 10 years ago in reply to ECKEROTH

And I don't have 20-nic.rules in /lib/udev/rules.d/

// Martin

Martin ,

The file is in /etc/udev/ not in lib/....
Just run a vim /etc/udev/rules.d/20-nic.rules and open directly the file.
thx
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel
0 ECKEROTH over 10 years ago in reply to Tinkerbell
Martin ,

The file is in /etc/udev/ not in lib/....
Just run a vim /etc/udev/rules.d/20-nic.rules and open directly the file.
thx

Hi Bianca

Found it: TSO is OFF. Do you want me to enable?

When trying to run nic-rules command i've get this error:
fw:/home/login # /lib/udev/nic-disable-tso
/lib/udev/nic-disable-tso: line 8: DEVPATH: parameter null or not set
fw:/home/login # /lib/udev/nic-disable-gro
/lib/udev/nic-disable-gro: line 7: DEVPATH: parameter null or not set
fw:/home/login #

// Martin
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel
0 Tinkerbell over 10 years ago in reply to ECKEROTH

Hi martin,

no no dont enable tso.

gro and gso are shown as ON from the output. Lets disable it and run the load test again.
please run: /sbin/ethtool -K eth1 gso off gro off

i saw eth 1 is the one that crashes on ure machine.
the whole purpose is to have those parameters mentioned in my post disabled.

best
bianca
Cancel
Vote Up 0 Vote Down

Sign in to reply

Verify Answer

Cancel

0 ECKEROTH over 10 years ago in reply to Tinkerbell

This is the output:

fw:/home/login # /sbin/ethtool -K eth1 gso off gro off

Actual changes:

tcp-segmentation-offload: on

        tx-tcp-segmentation: on

        tx-tcp6-segmentation: on

generic-segmentation-offload: off

generic-receive-offload: off

fw:/home/login #

Status on eth1 is now:

fw:/home/login # /sbin/ethtool -k eth1

Features for eth1:

rx-checksumming: on

tx-checksumming: on

        tx-checksum-ipv4: off [fixed]

        tx-checksum-ip-generic: on

        tx-checksum-ipv6: off [fixed]

        tx-checksum-fcoe-crc: off [fixed]

        tx-checksum-sctp: off [fixed]

scatter-gather: on

        tx-scatter-gather: on

        tx-scatter-gather-fraglist: off [fixed]

tcp-segmentation-offload: off

        tx-tcp-segmentation: off

        tx-tcp-ecn-segmentation: off [fixed]

        tx-tcp6-segmentation: off

udp-fragmentation-offload: off [fixed]

generic-segmentation-offload: off

generic-receive-offload: off

large-receive-offload: off [fixed]

rx-vlan-offload: on

tx-vlan-offload: on

ntuple-filters: off [fixed]

receive-hashing: on

highdma: on [fixed]

rx-vlan-filter: off [fixed]

vlan-challenged: off [fixed]

tx-lockless: off [fixed]

netns-local: off [fixed]

tx-gso-robust: off [fixed]

tx-fcoe-segmentation: off [fixed]

fcoe-mtu: off [fixed]

tx-nocache-copy: on

loopback: off [fixed]

rx-fcs: off

rx-all: off

fw:/home/login #

// Martin