Skip to contentSkip to conversation

X86_64 + VIA + 4g problems

DISCUSSION Robert Hancock 2 participants 5 messages Jan 2, 2006 – Jan 4, 2006 5 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw
Quoted 2 lines · Andi Kleen

When you not compile in the SKGE network driver does everything else work? skge supports 64bit DMA, so it shouldn't use any IOMMU.

Are you sure this is skge? Anyway, even if the driver does support 64-bit DMA, I would be surprised if a desktop motherboard had the Ethernet chip connected via a 64-bit PCI bus.

This brings up something I've been wondering. It's possible to run most 64-bit capable PCI devices in a 32-bit slot (i.e. with the 64-bit part hanging out of the slot). In this configuration the device will not be able to use 64-bit DMA (unless it supports dual address cycle). However, who is supposed to detect this and know to not try to use DMA addresses above 4GB on that device? Is the driver supposed to know this and use a 32-bit DMA mask, or is the PCI subsystem able to figure this out somehow? What if the driver has no way to detect this? I don't think I've seen any code in the kernel that would figure out if the current configuration of the card is actually capable of 64-bit DMA or not.

Signature · 3 lines
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/
#3 lore raw
Quoted 5 lines · Robert Hancock
Quoted 2 lines · Andi Kleen

When you not compile in the SKGE network driver does everything else work? skge supports 64bit DMA, so it shouldn't use any IOMMU.

Are you sure this is skge?

I grepped for the output in that badly truncated logfile and skge hit.

Quoted 3 lines

Anyway, even if the driver does support
64-bit DMA, I would be surprised if a desktop motherboard had the Ethernet chip connected via a 64-bit PCI bus.

I meant 64bit DMA, aka double address cycle, not a 64bit wide PCI-X bus. That works on a 32bit PCI bus too. On modern boards it's likely PCI Express anyways for on board devices.

Quoted 7 lines

This brings up something I've been wondering. It's possible to run most 64-bit capable PCI devices in a 32-bit slot (i.e. with the 64-bit part hanging out of the slot). In this configuration the device will not be able to use 64-bit DMA (unless it supports dual address cycle). However, who is supposed to detect this and know to not try to use DMA addresses above 4GB on that device?

64bit PCI-X bus has nothing to do with the addressing capability. It just gives you more bandwidth.

-Andi

#4 lore raw
Quoted 5 lines · Andi Kleen
Quoted 1 line

This brings up something I've been wondering. It's possible to run most 64-bit capable PCI devices in a 32-bit slot (i.e. with the 64-bit part hanging out of the slot). In this configuration the device will not be able to use 64-bit DMA (unless it supports dual address cycle). However, who is supposed to detect this and know to not try to use DMA addresses above 4GB on that device?

64bit PCI-X bus has nothing to do with the addressing capability. It just gives you more bandwidth.

Indeed, I misunderstood how this worked.. Accesses to addresses >4GB must always use DAC addressing, the address can just be decoded faster if the extra 64-bit data lines are connected.

I think there could still be an issue if the PCI bus does not support DAC and the device tries to use PCI DAC addressing causing the bus to choke or the wrong RAM to get accessed. Apparently this is a real problem according to MS:

http://www.microsoft.com/whdc/system/platform/server/PAE/PAEdrv.mspx

"Unfortunately, Microsoft is finding that not all PCI buses on a system board support DAC, which is required for a 32-bit PCI adapter to address more than 4 GB of memory. Furthermore, there is no way for a DAC-capable PCI device (or its associated driver) to know that it is running on a non-DAC-capable bus."

Microsoft's solution is to disable all memory above 4GB if non-DAC-capable buses are present - largely due to their API limitations and the number of broken drivers that don't call proper DMA functions. Haha to them, I guess..

I'm not sure if this would be an issue with Linux. I would suspect that it could be, if there are indeed such buses that can't support DAC.. In this case the right thing to do would be to reject requests for DMA masks larger than 4GB for devices located on such a bus.

Signature · 3 lines
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/
#5 lore raw
Quoted 4 lines · Robert Hancock

I'm not sure if this would be an issue with Linux. I would suspect that it could be, if there are indeed such buses that can't support DAC.. In this case the right thing to do would be to reject requests for DMA masks larger than 4GB for devices located on such a bus.

Well, from the original messages in this thread which you so shamelessly hijacked it is possible VIA PCI bridges might have this problem. If yes it can be handled in the PCI-DMA API by disallowing DAC for devices behind such bridges. But before taking such drastic action we need first need more testing results.

-Andi

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 11 lines · Carsten Otto
Quoted 2 lines · Andi Kleen

Alternatively you can try with the appended patch. If that helps then the chipset or the BIOS likely has some fundamental issue with >4GB.

Based on the first few minutes of testing this works perfectly! Thanks a lot!

PS: Does it make sense to contact my shop and/or Abit and/or VIA regarding this problem? I don't want to pay money for broken hardware.

I suspect VIA may be the best place to complain to, it is most likely their breakage. I don't know how responsive they are to such things however.

Signature · 3 lines
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/