Strange problem with e1000 driver - ping packet loss
Hi,
I happened to look at a system which was exhibiting poor ping
performance with e1000 driver (in 2.6.25) and had some questions regarding that.
Ping test was done between the system and a laptop, which were connected using a straight ethernet cable. Ping reported round trip times running into seconds (!) and also packet loss.
Upon some investigation, I found that the interrupt count field in /proc/interrupts (associated with eth1) is not incrementing as fast as it should. Moreover eth1 interrupt line is shared with the hard disk interrupt (ata_piix) as below:
# cat /proc/interrupts
..
10: 2296 XT-PIC-XT ata_piix, eth0, eth1
..
IRQ10 is thus being shared by both the hard disk and eth0/eth1.
Here's the strange observation I made:
When I initiate some disk activity (ex: dd if=/dev/zero of=/tmp/file), ping performance suddently shot up (round trip time in double digits ms, 0% packet loss)! I presume this is because that e1000 intr handler is called whenever there was a interrupt from hard disk on IRQ10, which polled NIC and processed packets immediately.
As soon as I kill the background disk-write intensive job, ping performance again dropped.
This meant that e1000 NIC is having trouble interrupting the OS.
Before I could jump up and say this is a hardware issue, I was told that Windows works just fine on the server (and as well as 2.4 kernel, which I couldnt verify) :(
Some more observations:
1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0, TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of them helped.
2. When ping performance was poor, readprofile showed that system is mostly idle. This confirms that OS is not getting very frequenty interrupts from eth1 and hence idling.
3. When ping performance was poor, ethtool -S eth1 showed that rx_bytes was incrementing at a good pace, showing that the NIC was receiving ping responses back, but not handing them over to OS for further processing
4. e1000 chipset is 82546GB
5. e1000e driver didnt work at all (it doesnt recognize the cards).
Any advice on how to fix this problem?
Signature · 2 lines
Regards, vatsa
Quoted 6 lines · Srivatsa Vaddagiri
Hi,
I happened to look at a system which was exhibiting poor ping
performance with e1000 driver (in 2.6.25) and had some questions regarding that.
...
Quoted 7 lines
Upon some investigation, I found that the interrupt count field in
/proc/interrupts(associated with eth1) is not incrementing as fast as it should. Moreover eth1 interrupt line is shared with the hard disk interrupt (ata_piix) as below:# cat /proc/interrupts 10: 2296 XT-PIC-XT ata_piix, eth0, eth1
whats wrong with your system that you can't use acpi and/or apic? It would probably orthoginally solve the problem by unsharing your interrupt.
Quoted 1 line
IRQ10 is thus being shared by both the hard disk and eth0/eth1.
bad for performance but should really work okay.
Quoted 6 lines
Here's the strange observation I made:
When I initiate some disk activity (ex: dd if=/dev/zero
...This meant that e1000 NIC is having trouble interrupting the OS.
you're correct here, there appears to be some problem on your system either with interrupt delivery or with the driver masking off interrupts and leaving them disabled.
Quoted 3 lines
Before I could jump up and say this is a hardware issue, I was told that Windows works just fine on the server (and as well as 2.4 kernel, which I couldnt verify) :(
well it might be a bios issue, but would likely be solved by using boot option acpi=force and/or lapci (see kernel-parameters.txt
Quoted 5 lines
Some more observations:
1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0, TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of them helped.
these won't help you get an interrupt delivered or re-enabled
Quoted 3 lines
2. When ping performance was poor, readprofile showed that system is mostly idle. This confirms that OS is not getting very frequenty interrupts from eth1 and hence idling.
expected, thanks for checking.
Quoted 4 lines
3. When ping performance was poor, ethtool -S eth1 showed that
rx_byteswas incrementing at a good pace, showing that the NIC was receiving ping responses back, but not handing them over to OS for further processing
also expected for an interrupt problem.
Quoted 3 lines
4. e1000 chipset is 82546GB
5. e1000e driver didnt work at all (it doesnt recognize the cards).
expected, this is a PCI-X adapter.
Quoted 1 line
Any advice on how to fix this problem?
try the boot options first, then if that doesn't work for you, download ethregs from e1000.sourceforge.net download area and compile/run it and send me the output in private email.
if you have a spare moment, you can try the e1000-8.X driver from sourceforge and let me know if it works okay, that would imply we just need to patch the in-kernel driver to fix an already known issue.
Jesse
Quoted 6 lines · Jesse
Quoted 2 lines
# cat /proc/interrupts 10: 2296 XT-PIC-XT ata_piix, eth0, eth1whats wrong with your system that you can't use acpi and/or apic? It would probably orthoginally solve the problem by unsharing your interrupt.
Nothing wrong with acpi/apic. It just wasnt helping solve the problem. I booted with noapic to check if it helps resolve, but found it didnt.
Quoted 3 lines
Quoted 1 line
IRQ10 is thus being shared by both the hard disk and eth0/eth1.
bad for performance but should really work okay.
Is there a way we can force unsharing of the IRQ (between harddisk and eth1) in software?
Quoted 9 lines
Quoted 6 lines
Here's the strange observation I made:
When I initiate some disk activity (ex: dd if=/dev/zero
...This meant that e1000 NIC is having trouble interrupting the OS.
you're correct here, there appears to be some problem on your system either with interrupt delivery
Note that other interrupts (timer, hard disk) are fine. Even eth1 interrupt "works", just that it comes lazily (once in few seconds - when I am pumping potentially hundreds of ping packets to it every second).
# watch "grep 10: /proc/interrupts"
shows the interrupt count associated with eth1 increment at the rate of 1-2 every 2-3 seconds (<1 interrupt per second).
Is there some interrupt-related statistics that we can obtain from e1000 card which shows how many times e1000 NIC tried "interrupting" the system?
Quoted 1 line
or with the driver masking off interrupts and leaving them disabled.
Hmm ..shouldnt that affect ata disk functionality too? hard disk I/O works fine when ping performance is bad.
Quoted 5 lines
Quoted 3 lines
Before I could jump up and say this is a hardware issue, I was told that Windows works just fine on the server (and as well as 2.4 kernel, which I couldnt verify) :(
well it might be a bios issue,
Again, if it was a bios issue, the question i am faced with is "how is Windows working fine on it?".
Quoted 2 lines
but would likely be solved by using boot
option acpi=force and/or lapci (see kernel-parameters.txt
I had tried these other boot options in vain: noapic, acpi=off, acpi=noirq, pci=noacpi If you recommend any other boot option, we'd be glad to try it out.
Quoted 7 lines
Quoted 5 lines
Some more observations:
1. I tried setting e1000 parameters (RxIntDelay=0, RxAbsIntDelay=0, TxIntDelay=0, TxAbsIntDelay=0, InterruptThrottleRate=0). None of them helped.
these won't help you get an interrupt delivered or re-enabled
ok.
Quoted 17 lines
Quoted 3 lines
2. When ping performance was poor, readprofile showed that system is mostly idle. This confirms that OS is not getting very frequenty interrupts from eth1 and hence idling.
expected, thanks for checking.
Quoted 4 lines
3. When ping performance was poor, ethtool -S eth1 showed that
rx_byteswas incrementing at a good pace, showing that the NIC was receiving ping responses back, but not handing them over to OS for further processingalso expected for an interrupt problem.
Quoted 3 lines
4. e1000 chipset is 82546GB
5. e1000e driver didnt work at all (it doesnt recognize the cards).
Quoted 8 lines
expected, this is a PCI-X adapter.
Quoted 1 line
Any advice on how to fix this problem?
try the boot options first, then if that doesn't work for you, download ethregs from e1000.sourceforge.net download area and compile/run it and send me the output in private email.
sure ..i will send you them after sometime.
Quoted 3 lines
if you have a spare moment, you can try the e1000-8.X driver from sourceforge and let me know if it works okay, that would imply we just need to patch the in-kernel driver to fix an already known issue.
ok, we will test that out as well.
Thanks for all your inputs!
Signature · 2 lines
Regards, vatsa