New: driver "sunhme" experiences corrupt packets if machine has more than 2GB of memory
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]/
Quoted 63 lines · On Sun, 25 May 2008 04:10:04 -0700 (PDT) [email protected] wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=10790
Summary: driver "sunhme" experiences corrupt packets if machine
has more than 2GB of memory
Product: Drivers
Version: 2.5
KernelVersion: 2.6.22.17-0.1
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Network
AssignedTo: [email protected]
ReportedBy: [email protected]
CC: [email protected]Latest working kernel version: unknown
Earliest failing kernel version: unknown
Distribution: OpenSUSE 10.3 (also tested on vanilla kernel) - filed on Novell bugzilla at https://bugzilla.novell.com/show_bug.cgi?id=376831Hardware Environment:
System 1: Q6600 cpu, 4GB mem, Asus P5N32-E SLI Plus, Sun QFE card System 2: Q6600 cpu, 4GB mem, MSI P35 NEO-F (P35), Sun QFE cardSoftware Environment: OpenSUSE 10.3, both SUSE & vanilla kernels
Problem Description:
Network interfaces on the QFE card experience corrupted packets when the machine has more than 2GB of memory. The NIC can successfully obtain a DHCP address from a server, but attempts to ping or ssh to the address give unexpected results
hazard@mustang:~/hme-debug> ping 192.168.2.90
PING 192.168.2.90 (192.168.2.90) 56(84) bytes of data. 64 bytes from 192.168.2.90: icmp_seq=1 ttl=64 time=2.20 ms wrong data byte #54 should be0x36but was0xba
#16 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27
28 29 2a 2b 2c 2d 2e 2f
#48 30 31 32 33 34 35 ba cc
64 bytes from 192.168.2.90: icmp_seq=2 ttl=64 time=0.223 ms wrong data byte #54 should be0x36but was0x64
#16 10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f 20 21 22 23 24 25 26 27
28 29 2a 2b 2c 2d 2e 2f
#48 30 31 32 33 34 35 64 d8hazard@mustang:~/hme-debug> ssh 192.168.2.90
Received disconnect from 192.168.2.90: 2: Corrupted MAC on input. hazard@mustang:~/hme-debug> ssh 192.168.2.90
Received disconnect from 192.168.2.90: 2: Bad packet length 1869967922. hazard@mustang:~/hme-debug> ssh 192.168.2.90
Received disconnect from 192.168.2.90: 2: Bad packet length 1869967922. hazard@mustang:~/hme-debug> ssh 192.168.2.90
Received disconnect from 192.168.2.90: 2: Bad packet length 1869967922.
Quoted 5 lines
Steps to reproduce:
Plumb an interface on a Sun QFE card in a machine with > 2GB memory. Ping or ssh to the IP of the interface.
Note that this patch from Karsten:
https://bugzilla.novell.com/attachment.cgi?id=217968 did not work.
Stoopid question: if it fails at 2GB, should we try DMA_31BIT_MASK?
David Miller
maintainerFrom: Andrew Morton <[email protected]>
Date: Sun, 25 May 2008 11:45:48 -0700
Quoted 2 lines
Note that this patch from Karsten:
https://bugzilla.novell.com/attachment.cgi?id=217968 did not work.
That patch is unlikely to be the issue, right.
Quoted 1 line
Stoopid question: if it fails at 2GB, should we try
DMA_31BIT_MASK?
I've scoured all of the docs and the chip itself and the PCI front-end it is connected to support full 32-bit PCI addressing.
The bug report says platform "All" but the report specifically mentions Q6600 cpus, is this x86_64 by chance? I wonder if the problem is that we use GFP_ATOMIC/GFP_KERNEL for allocations of SKB data, and we end up with buffers above 4GB but the IOMMU mapping operation doesn't cope with that correctly.
In the arch/x86/kernel/pci-nommu.c case, this would result in kernel log messages from check_addr():
if (*hwdev->dma_mask >= DMA_32BIT_MASK) printk(KERN_ERR "nommu_%s: overflow %Lx+%zu of device mask %Lx\n", name, (long long)bus, size, (long long)*hwdev->dma_mask);
Quoted 8 lines · David Miller
From: Andrew Morton <[email protected]>
Date: Sun, 25 May 2008 11:45:48 -0700Quoted 2 lines
Note that this patch from Karsten:
https://bugzilla.novell.com/attachment.cgi?id=217968 did not work.That patch is unlikely to be the issue, right.
No, this patch was a experiment to fix the issue, it is not in our kernel by default.
Quoted 10 lines
Quoted 1 line
Stoopid question: if it fails at 2GB, should we try
DMA_31BIT_MASK?I've scoured all of the docs and the chip itself and the PCI front-end it is connected to support full 32-bit PCI addressing.
The bug report says platform "All" but the report specifically mentions Q6600 cpus, is this
x86_64by chance? I wonder if the problem is that we use GFP_ATOMIC/GFP_KERNEL for allocations of SKB data, and we end up with buffers above 4GB but the IOMMU mapping operation doesn't cope with that correctly.
Yes it is x86_64 and I think, that if you put more as 2G in this machine, some physical addresses are behind the 4 GB border.
Quoted 9 lines
In the
arch/x86/kernel/pci-nommu.ccase, this would result in kernel log messages fromcheck_addr():if (*hwdev->dma_mask >= DMA_32BIT_MASK) printk(KERN_ERR "nommu_%s: overflow %Lx+%zu of device mask %Lx\n", name, (long long)bus, size, (long long)*hwdev->dma_mask);
Signature · 4 lines
Karsten Keil SuSE Labs ISDN and VOIP development SUSE LINUX Products GmbH, Maxfeldstr.5 90409 Nuernberg, GF: Markus Rex, HRB 16746 (AG Nuernberg)
David Miller
maintainerFrom: Karsten Keil <[email protected]>
Date: Wed, 4 Jun 2008 10:21:21 +0200
Quoted 2 lines
Yes it is
x86_64and I think, that if you put more as 2G in this machine, some physical addresses are behind the 4 GB border.
If that is the case then I think there might be some bug in the dma_sync_single_for_cpu() implementation on whatever IOMMU implementation is being used on this x86_64 system.
sunhme has two cases:
1) If the packet is greater than 256 bytes the original packet is unmapped. So, if soft-iommu uses a bounce buffer, for example, this should do the necessary copy from the bounce buffer to the actual SKB data area.
2) If the packet is less than or equal to 256 bytes, the packet is DMA sync'd using pci_dma_sync_single_for_cpu(), which should cause soft-iommu or similar to copy from any bounce buffer to the real buffer, and then we copy the SKB data into a new freshly allocated SKB and reuse the original one after calling pci_dma_sync_single_for_device().
I've validated that the DMA address and lengths used by those calls in the sunmhe driver are correct, so really my top suspect would be the IOMMU implementation being used on this system for these specific kinds of cases.