32-bit dma allocations on 64-bit platforms
A message that is not in the archive (sent off-list or not yet received): [email protected]
Terence Ripperda <[email protected]> writes:
[sending again with linux-kernel in cc]
Quoted 1 line
I'm working on cleaning up some of our dma allocation code to properly allocate 32-bit physical pages for dma on 64-bit platforms. I think our first pass at supporting em64t is sub-par. I'd like to fix that by using the correct kernel interfaces.
I get from this that your hardware cannot DMA to >32bit.
Quoted 12 lines
the physical addressing of memory allocations seems to boil down to the behavior of
GFP_DMAandGFP_NORMAL. but there seems to be some disconnect between what these mean for each architecture and what various drivers expect them to mean.based on each architecture's
paging_initroutines, the zones look like this:x86: ia64: x86_64: ZONE_DMA: < 16M < ~4G < 16M ZONE_NORMAL: 16M - ~1G > ~4G > 16M
ZONE_HIMEM: 1G+an example of this disconnect is
vmalloc_32. this function is obviously intended to allocate 32-bit addresses (this was specifically mentioned in a comment in 2.4.x header files). butvmalloc_32is an inline routine that calls__vmalloc(GFP_KERNEL). based on the above zone descriptions, this will do the correct thing for x86, but not for ia64 orx86_64. on ia64, a driver could just useGFP_DMAfor the desired behavior, but this doesn't work well forx86_64.
Quoted 7 lines
AMD's
x86_64provides remapping > 32-bit pages through the iommu, but obviously Intel's em64t provides no such ability. based on the above zonings, these leaves us with the options of either relying on the swiotlb interfaces for dma, or relying on the isa memory for dma.the last day or two, I've been experimenting with using the
pci_alloc_consistentinterface, which uses the later (note attached patch to fix an apparent memory leak in thex86_64pci_alloc_consistent). unfortunately, this provides very little dma-able memory. In theory, up to 16 Megs, but in practice I'm only getting about 5 1/2 Megs.I was rather surprised by these limitations on allocating 32-bit addresses. I checked through the dri and bttv drivers, to see if they had dealt with these issues, and they did not appear to have done so. has anyone tested these drivers on ia64/x86_64/em64t platforms w/ 4+ Gigs of memory?
are these limitations on allocating 32-bit addresses intentional and known? is there anything we can do to help improve this situation? help work on development?
First vmalloc_32 is a rather broken interface and should imho just be removed. The function name just gives promises that cannot be kept. It was always quite bogus. Please don't use it.
The situation on EM64T is very unfortunate I agree. There was a reason we asked AMD to add an IOMMU and it's quite bad that the Intel chipset people ignored that wisdom and put us into this compatibility mess.
Failing that it would be best if the other PCI DMA hardware could just address enough memory, but that's less realistic than just fixing the chipset.
The x86-64 port had decided early to keep the 16MB GFP_DMA zone to get maximum driver compatibility and because the AMD IOMMU gave us an nice alternative over bounce buffering.
In theory I'm not totally against enlarging GFP_DMA a bit on x86-64. It would just be difficult to find a good value. The problem is that that there may be existing drivers that rely on the 16MB limit, and it would not be very nice to break them. We got rid of a lot of them by disallowing CONFIG_ISA, but there may be some left. So before doing this it would need a full driver tree audit to check any device
The most prominent example used to be the floppy driver, but the current floppy driver seems to use some other way to get around this.
There seem to be quite some sound chipsets with DMA limits < 32bit; e.g. 29 bits seems to be quite common, but I see several 24bit PCI ones too.
I must say I'm somewhat reluctant to break an working in tree driver. Especially for the sake of an out of tree binary driver. Arguably the problem is probably not limited to you, but it's quite possible that even the in tree DRI drivers have it, so it would be still worth to fix it.
I see two somewhat realistic ways to handle this:
- We enlarge GFP_DMA and find some way to do double buffering for these sound drivers (it would need an PCI-DMA API extension that always calls swiotlb for this)
For sound that's not too bad, because they are relatively slow. It would require to reserve bootmem memory early for the bounces, but I guess requiring the user to pass a special boot time parameter for these devices would be reasonable.
If yes someone would need to do this work.
Also the question would be how large to make GFP_DMA Ideally it should not be too big, so that e.g. 29bit devices don't require the bounce buffering.
- We introduce multiple GFP_DMA zones and keep 16MB GFP_DMA and GFP_BIGDMA or somesuch for larger DMA.
The VM should be able to handle this, but it may still require some tuning. It would need some generic changes, but not too bad. Still would need a decision on how big GFP_BIGDMA should be. I suspect 4GB would be too big again.
Comments?
-Andi
Quoted 4 lines · Andi Kleen
The VM should be able to handle this, but it may still require some tuning. It would need some generic changes, but not too bad. Still would need a decision on how big
GFP_BIGDMAshould be. I suspect 4GB would be too big again.
What is the problem again, can't the driver us the dynamic pci mapping API which does allow more memory to be mapped even on crippled machines without iommu ?
And isn't this a problem that will vanish since PCI Express and PCI X both *require* support for 64 bit addressing, so all higher speed cards are going to be ok in principle ?
Quoted 10 lines · Arjan van de Ven
Quoted 4 lines · Andi Kleen
The VM should be able to handle this, but it may still require some tuning. It would need some generic changes, but not too bad. Still would need a decision on how big
GFP_BIGDMAshould be. I suspect 4GB would be too big again.What is the problem again, can't the driver us the dynamic pci mapping API which does allow more memory to be mapped even on crippled machines without iommu ?
In theory one could fix pci_alloc_consistent from the swiotlb pool yes, the problem is just that this pool is completely preallocated. If enough memory is needed that would be quite nasty, because you suddenly lose 1 or 2GB RAM.
Quoted 3 lines
And isn't this a problem that will vanish since PCI Express and PCI X both *require* support for 64 bit addressing, so all higher speed cards are going to be ok in principle ?
There are EM64T systems with AGP only and not all PCI-Express cards seem to follow this. PCI-Express unfortunately discouraged the AGP aperture too, so not even that can be used on those Intel systems.
-Andi
Quoted 6 lines · Arjan van de Ven
What is the problem again, can't the driver us the dynamic pci mapping API which does allow more memory to be mapped even on crippled machines without iommu ?
And isn't this a problem that will vanish since PCI Express and PCI X both *require* support for 64 bit addressing, so all higher speed cards are going to be ok in principle ?
Well, PCI-X may require it, but there certainly are PCI-X devices that don't do 64 bit addressing, or if they do, it's a crippled implementation (e.g. top 32 bits have to be constant).
Jesse
correct. I checked with my contacts here on the PCI express requirements. Apparently the spec says "A PCI Express Endpoint operating as the Requester of a Memory Transaction is required to be capable of generating addresses greater than 4GB", but my contact claims this is a "soft" requirement.
but even if all PCI-X and PCI-E devices properly addressed the full 64-bits, legacy 32-bit PCI devices can be plugged into the motherboards as well. my Intel em64t boards have mostly PCI-X, but 1 PCI slot and my amd x86_64 have all PCI slots (aside from the main PCI-E slot).
also, at least one motherboard manufacturer claims PCI-E + AGP, but the AGP is really just an AGP form-factor slot on the PCI bus.
Thanks,
Terence
Quoted 17 lines · On Thu, Jun 24, 2004 at 06:48:07AM -0700, [email protected] wrote:
Quoted 2 lines · Arjan van de Ven
What is the problem again, can't the driver us the dynamic pci mapping API which does allow more memory to be mapped even on crippled
machines
Quoted 3 lines
without iommu ?
And isn't this a problem that will vanish since PCI Express and PCI X both *require* support for 64 bit addressing, so all higher speedcards
Quoted 1 line
are going to be ok in principle ?
Well, PCI-X may require it, but there certainly are PCI-X devices that don't
do 64 bit addressing, or if they do, it's a crippled implementation (e.g. top
32 bits have to be constant).Jesse
Quoted 4 lines · Terence Ripperda
but even if all PCI-X and PCI-E devices properly addressed the full 64-bits, legacy 32-bit PCI devices can be plugged into the motherboards as well. my Intel em64t boards have mostly PCI-X, but 1 PCI slot and my amd
x86_64have all PCI slots (aside from the main PCI-E slot).
For the older AGP devices you can always map the data through the AGP aperture, no ? (It also has a size limit, but that can be usually increased in the BIOS setup)
This won't work for graphic cards put into PCI slots, but these can probably tolerate some performance degradation.
For AMD all PCI IO can be done through the aperture anyways, only Intel is more crippled in this regard.
Quoted 2 lines
also, at least one motherboard manufacturer claims PCI-E + AGP, but the AGP is really just an AGP form-factor slot on the PCI bus.
With no aperture I guess?
-Andi