Skip to contentSkip to conversation

PCI-DMA: Out of IOMMU space on x86-64 (Athlon64x2), with solution

DISCUSSION Robert Hancock 4 participants 5 messages Mar 7, 2006 – Apr 2, 2006 4 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 8 lines · Michael Monnerie
Quoted 5 lines · On Freitag, 3. März 2006 23:23 Jeff Garzik wrote:

I'll happen but not soon. Motivation is low at NV and here as well, since newer NV is AHCI. The code in question, "NV ADMA", is essentially legacy at this point -- though I certainly acknowledge the large current installed base. Just being honest about the current state of things...

I'd like to raise motivation a lot because most MB sold here (central Europe) are Nforce4 with Athlon64x2 at the moment. It would be nice from vendors if they support OSS developers more, as it's their interest to have good drivers.

I second that.. It appears that nForce4 will continue to be a popular chipset even after the Socket AM2 chips are released, so the demand for this (and for NCQ support as well, likely) will only increase.

Signature · 3 lines
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from [email protected]
Home Page: http://www.roberthancock.com/
#5 lore raw
Quoted 17 lines · Robert Hancock
Quoted 9 lines · Michael Monnerie
Quoted 5 lines · On Freitag, 3. März 2006 23:23 Jeff Garzik wrote:

I'll happen but not soon. Motivation is low at NV and here as well, since newer NV is AHCI. The code in question, "NV ADMA", is essentially legacy at this point -- though I certainly acknowledge the large current installed base. Just being honest about the current state of things...

Quoted 4 lines

I'd like to raise motivation a lot because most MB sold here (central Europe) are Nforce4 with Athlon64x2 at the moment. It would be nice from vendors if they support OSS developers more, as it's their interest to have good drivers.

Quoted 4 lines

I second that.. It appears that nForce4 will continue to be a popular chipset even after the Socket AM2 chips are released, so the demand for this (and for NCQ support as well, likely) will only increase.

I'll third it. Just had another machine blow up it's RAID5 set because of this bug. Tyan S2895 board, 4 500GB Hitachi SATA drives in RAID5. I suppose I could buy a 3ware controller I suppose but that's a few hundred dollars per machine.

These machines are running SUSE 9.3 or SUSE 10, I've tried kernel.org kernels as well as the iommu=memaper=3 cmdline option.

Any other advice greatly appreciated.

I saw a lot of patches come through by Muli but am not sure they address this issue, do they?

--Joerg

#6 lore raw
Quoted 2 lines · Joerg Bashir

I saw a lot of patches come through by Muli but am not sure they address this issue, do they?

No, I'm afraid not - our patches are to support a different IOMMU, and it looks like this problem is gart specific. I'll go dig through the archive, but is there consensus on how to solve this bug and it's just a question of doing the work, or is the root cause unknown?

Cheers,
Muli

Signature · 2 lines
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/
#7 lore raw
Quoted 7 lines · Muli Ben-Yehuda
Quoted 2 lines · Joerg Bashir

I saw a lot of patches come through by Muli but am not sure they address this issue, do they?

Quoted 5 lines

No, I'm afraid not - our patches are to support a different IOMMU, and it looks like this problem is gart specific. I'll go dig through the archive, but is there consensus on how to solve this bug and it's just a question of doing the work, or is the root cause unknown?

I'm unsure. Here is an earlier message from this thread with info and a glimmer of hope.

------------------------------------

Quoted 11 lines · Andi Kleen
Quoted 5 lines · Allen Martin

nForce4 has 64 bit (40 bit AMD64) DMA in the SATA controller. We gave the docs to Jeff Garzik under NDA. He posted some non functional driver code to linux-ide earlier this week that has the 64 bit registers and structures although it doesn't make use of them. Someone could pick this up if they wanted to work on it though.

Quoted 1 line

Thanks for the correction. Sounds nice - hopefully we'll get a driver

soon.

Quoted 1 line

I guess it's in good hands with Jeff for now.

I'll happen but not soon. Motivation is low at NV and here as well, since newer NV is AHCI. The code in question, "NV ADMA", is essentially legacy at this point -- though I certainly acknowledge the large current installed base. Just being honest about the current state of things...

Jeff

Quoted 3 lines

Cheers,
Muli

#8 lore raw
In reply to Joerg Bashir
Quoted 23 lines · Joerg Bashir
Quoted 17 lines · Robert Hancock
Quoted 9 lines · Michael Monnerie
Quoted 5 lines · On Freitag, 3. März 2006 23:23 Jeff Garzik wrote:

I'll happen but not soon. Motivation is low at NV and here as well, since newer NV is AHCI. The code in question, "NV ADMA", is essentially legacy at this point -- though I certainly acknowledge the large current installed base. Just being honest about the current state of things...

Quoted 4 lines

I'd like to raise motivation a lot because most MB sold here (central Europe) are Nforce4 with Athlon64x2 at the moment. It would be nice from vendors if they support OSS developers more, as it's their interest to have good drivers.

Quoted 4 lines

I second that.. It appears that nForce4 will continue to be a popular chipset even after the Socket AM2 chips are released, so the demand for this (and for NCQ support as well, likely) will only increase.

Quoted 7 lines

I'll third it. Just had another machine blow up it's RAID5 set because of this bug. Tyan S2895 board, 4 500GB Hitachi SATA drives in RAID5. I suppose I could buy a 3ware controller I suppose but that's a few hundred dollars per machine.

These machines are running SUSE 9.3 or SUSE 10, I've tried kernel.org kernels as well as the iommu=memaper=3 cmdline option.

You either have an IOMMU leak or very deep block device queues that overflow the aperture. You can either increase it even more or try to reduce the block queue lengths using the sysfs knobs (/sys/block/*/queue/) The maximum memory pinned by all the block queues must be smaller than the IOMMU aperture minus some slack for other devices.

If it's a leak someone has to debug it.

Or alternatively someone fixes the driver that can run the NForce4 controller without 4GB limits. If you still have a leak somewhere that won't help of course.

Queue overflow is more likely.

-Andi