Skip to contentSkip to conversation

Strange problem with e1000 driver - ping packet loss

DISCUSSION Srivatsa Vaddagiri 2 participants 3 messages Jun 18, 2008 – Jun 19, 2008

lore
Share & tools
lore
#0 lore raw

Hi,
I happened to look at a system which was exhibiting poor ping
performance with e1000 driver (in 2.6.25) and had some questions regarding that.

Ping test was done between the system and a laptop, which were connected using a straight ethernet cable. Ping reported round trip times running into seconds (!) and also packet loss.

Upon some investigation, I found that the interrupt count field in /proc/interrupts (associated with eth1) is not incrementing as fast as it should. Moreover eth1 interrupt line is shared with the hard disk interrupt (ata_piix) as below:

# cat /proc/interrupts

..

10: 2296 XT-PIC-XT ata_piix, eth0, eth1

..

IRQ10 is thus being shared by both the hard disk and eth0/eth1.

Here's the strange observation I made:

When I initiate some disk activity (ex: dd if=/dev/zero of=/tmp/file), ping performance suddently shot up (round trip time in double digits ms, 0% packet loss)! I presume this is because that e1000 intr handler is called whenever there was a interrupt from hard disk on IRQ10, which polled NIC and processed packets immediately.

As soon as I kill the background disk-write intensive job, ping performance again dropped.

This meant that e1000 NIC is having trouble interrupting the OS.

Before I could jump up and say this is a hardware issue, I was told that Windows works just fine on the server (and as well as 2.4 kernel, which I couldnt verify) :(

Some more observations:

1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0, TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of them helped.

2. When ping performance was poor, readprofile showed that system is mostly idle. This confirms that OS is not getting very frequenty interrupts from eth1 and hence idling.

3. When ping performance was poor, ethtool -S eth1 showed that rx_bytes was incrementing at a good pace, showing that the NIC was receiving ping responses back, but not handing them over to OS for further processing

4. e1000 chipset is 82546GB

5. e1000e driver didnt work at all (it doesnt recognize the cards).

Any advice on how to fix this problem?

Signature · 2 lines
Regards,
vatsa
#1 lore raw
Quoted 6 lines · Srivatsa Vaddagiri

Hi,
I happened to look at a system which was exhibiting poor ping
performance with e1000 driver (in 2.6.25) and had some questions regarding that.
...

Quoted 7 lines

Upon some investigation, I found that the interrupt count field in /proc/interrupts (associated with eth1) is not incrementing as fast as it should. Moreover eth1 interrupt line is shared with the hard disk interrupt (ata_piix) as below:

# cat /proc/interrupts
 10:       2296    XT-PIC-XT        ata_piix, eth0, eth1

whats wrong with your system that you can't use acpi and/or apic? It would probably orthoginally solve the problem by unsharing your interrupt.

Quoted 1 line

IRQ10 is thus being shared by both the hard disk and eth0/eth1.

bad for performance but should really work okay.

Quoted 6 lines

Here's the strange observation I made:

When I initiate some disk activity (ex: dd if=/dev/zero
...

This meant that e1000 NIC is having trouble interrupting the OS.

you're correct here, there appears to be some problem on your system either with interrupt delivery or with the driver masking off interrupts and leaving them disabled.

Quoted 3 lines

Before I could jump up and say this is a hardware issue, I was told that Windows works just fine on the server (and as well as 2.4 kernel, which I couldnt verify) :(

well it might be a bios issue, but would likely be solved by using boot option acpi=force and/or lapci (see kernel-parameters.txt

Quoted 5 lines

Some more observations:

1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0, TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of them helped.

these won't help you get an interrupt delivered or re-enabled

Quoted 3 lines

2. When ping performance was poor, readprofile showed that system is mostly idle. This confirms that OS is not getting very frequenty interrupts from eth1 and hence idling.

expected, thanks for checking.

Quoted 4 lines

3. When ping performance was poor, ethtool -S eth1 showed that rx_bytes was incrementing at a good pace, showing that the NIC was receiving ping responses back, but not handing them over to OS for further processing

also expected for an interrupt problem.

Quoted 3 lines

4. e1000 chipset is 82546GB

5. e1000e driver didnt work at all (it doesnt recognize the cards).

expected, this is a PCI-X adapter.

Quoted 1 line

Any advice on how to fix this problem?

try the boot options first, then if that doesn't work for you, download ethregs from e1000.sourceforge.net download area and compile/run it and send me the output in private email.

if you have a spare moment, you can try the e1000-8.X driver from sourceforge and let me know if it works okay, that would imply we just need to patch the in-kernel driver to fix an already known issue.

Jesse

#2 lore raw
Quoted 6 lines · Jesse
Quoted 2 lines
# cat /proc/interrupts
 10:       2296    XT-PIC-XT        ata_piix, eth0, eth1

whats wrong with your system that you can't use acpi and/or apic? It would probably orthoginally solve the problem by unsharing your interrupt.

Nothing wrong with acpi/apic. It just wasnt helping solve the problem. I booted with noapic to check if it helps resolve, but found it didnt.

Quoted 3 lines
Quoted 1 line

IRQ10 is thus being shared by both the hard disk and eth0/eth1.

bad for performance but should really work okay.

Is there a way we can force unsharing of the IRQ (between harddisk and eth1) in software?

Quoted 9 lines
Quoted 6 lines

Here's the strange observation I made:

When I initiate some disk activity (ex: dd if=/dev/zero
...

This meant that e1000 NIC is having trouble interrupting the OS.

you're correct here, there appears to be some problem on your system either with interrupt delivery

Note that other interrupts (timer, hard disk) are fine. Even eth1 interrupt "works", just that it comes lazily (once in few seconds - when I am pumping potentially hundreds of ping packets to it every second).

	# watch "grep 10: /proc/interrupts"

shows the interrupt count associated with eth1 increment at the rate of 1-2 every 2-3 seconds (<1 interrupt per second).

Is there some interrupt-related statistics that we can obtain from e1000 card which shows how many times e1000 NIC tried "interrupting" the system?

Quoted 1 line

or with the driver masking off interrupts and leaving them disabled.

Hmm ..shouldnt that affect ata disk functionality too? hard disk I/O works fine when ping performance is bad.

Quoted 5 lines
Quoted 3 lines

Before I could jump up and say this is a hardware issue, I was told that Windows works just fine on the server (and as well as 2.4 kernel, which I couldnt verify) :(

well it might be a bios issue,

Again, if it was a bios issue, the question i am faced with is "how is Windows working fine on it?".

Quoted 2 lines

but would likely be solved by using boot
option acpi=force and/or lapci (see kernel-parameters.txt

I had tried these other boot options in vain: noapic, acpi=off, acpi=noirq, pci=noacpi If you recommend any other boot option, we'd be glad to try it out.

Quoted 7 lines
Quoted 5 lines

Some more observations:

1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0, TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of them helped.

these won't help you get an interrupt delivered or re-enabled

ok.

Quoted 17 lines
Quoted 3 lines

2. When ping performance was poor, readprofile showed that system is mostly idle. This confirms that OS is not getting very frequenty interrupts from eth1 and hence idling.

expected, thanks for checking.

Quoted 4 lines

3. When ping performance was poor, ethtool -S eth1 showed that rx_bytes was incrementing at a good pace, showing that the NIC was receiving ping responses back, but not handing them over to OS for further processing

also expected for an interrupt problem.

Quoted 3 lines

4. e1000 chipset is 82546GB

5. e1000e driver didnt work at all (it doesnt recognize the cards).

Quoted 8 lines

expected, this is a PCI-X adapter.

Quoted 1 line

Any advice on how to fix this problem?

try the boot options first, then if that doesn't work for you, download ethregs from e1000.sourceforge.net download area and compile/run it and send me the output in private email.

sure ..i will send you them after sometime.

Quoted 3 lines

if you have a spare moment, you can try the e1000-8.X driver from sourceforge and let me know if it works okay, that would imply we just need to patch the in-kernel driver to fix an already known issue.

ok, we will test that out as well.

Thanks for all your inputs!

Signature · 2 lines
Regards,
vatsa