64 Bits DMA Addresses for Alloc Consistent Interfaces.
Subsystems (3)
Quoted 6 lines · James Bottomley
Quoted 5 lines · Colin Ngam
Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.
Quoted 8 lines
Quoted 4 lines
This extention will not break current usages of these interface, and as driver changes, this extension will enable platforms to support all of it's Physical Memory via the above 2 Interfaces without exhausting limited system mapping resources.
The intent of the
dma_alloc_.. was to follow the PCI spec (i.e. lower 32 bits). However, it does take theGFP_flags, and thus could be adjusted to use the region specifier portion of those.
Hi James,
Do you mean to use the flag parameter to advice that we can return a 64Bit DMA Address? What is a region specifier?
Thanks.
colin
Quoted 6 lines
We're currently debating this very issue, since it came up in the context of the 2.6 must fix list.
James
Quoted 9 lines · Colin Ngam
Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.
This extention will not break current usages of these interface, and as driver changes, this extension will enable platforms to support all of it's Physical Memory via the above 2 Interfaces without exhausting limited system mapping resources.
The intent of the dma_alloc_.. was to follow the PCI spec (i.e. lower 32 bits). However, it does take the GFP_ flags, and thus could be adjusted to use the region specifier portion of those.
We're currently debating this very issue, since it came up in the context of the 2.6 must fix list.
James
Quoted 13 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Colin Ngam
Colin> Anyway, to make a long story short, we like to "extend" these Colin> interfaces to return DMA addresses of 64 bits if and when the Colin> driver has the full capabality to support DAC - via it's DMA Colin> mask capability setting.
Colin> This extention will not break current usages of these Colin> interface, and as driver changes, this extension will enable Colin> platforms to support all of it's Physical Memory via the Colin> above 2 Interfaces without exhausting limited system mapping Colin> resources.
Quoted 4 lines
James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (
GFP_DMA32or some such).
Hi David,
I actually like the flag as a parameter because it allows platform specific extensions to pass in platform specific capabilities that can be ignored by platform that does not support these capabilities.
I do not know whether using the same set of flags i.e.GFP_XXX would overload it's meanings to other MM routines but I really like the a flag parameter, that will provide capability for Platform specific hints.
Thanks.
Colin
Quoted 3 lines
--davidQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Colin Ngam
Colin> Anyway, to make a long story short, we like to "extend" these Colin> interfaces to return DMA addresses of 64 bits if and when the Colin> driver has the full capabality to support DAC - via it's DMA Colin> mask capability setting.
Colin> This extention will not break current usages of these Colin> interface, and as driver changes, this extension will enable Colin> platforms to support all of it's Physical Memory via the Colin> above 2 Interfaces without exhausting limited system mapping Colin> resources.
James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (GFP_DMA32 or some such).
--david
Quoted 4 lines · David Mosberger
James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (
GFP_DMA32or some such).
This is still being hashed out. It looks like adding a the _mask variants may still be a viable runner for 2.6. If it turns out not to be, the GFP_DMA32 will almost definitely be there.
James
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Colin Ngam
Colin> I do not know whether using the same set of flags i.e.GFP_XXX Colin> would overload it's meanings to other MM routines but I Colin> really like the a flag parameter, that will provide Colin> capability for Platform specific hints.
Overloading the GFP flags with platform-specific DMA stuff is certainly not an option.
If you want to argue for yet-another-flags argument, go ahead and do so. Be prepared for having to make a VERY strong argument...
--david
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · James Bottomley
Quoted 4 lines · David Mosberger
Quoted 4 lines
James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (
GFP_DMA32or some such).
James> This is still being hashed out. It looks like adding a the James> _mask variants may still be a viable runner for 2.6.
That would be good.
--david
Quoted 10 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · James Bottomley
Quoted 4 lines · David Mosberger
Quoted 4 lines
James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (
GFP_DMA32or some such).James> This is still being hashed out. It looks like adding a the James>
_maskvariants may still be a viable runner for 2.6.
David,
Can you provide a little detail on how the _mask vriant is used and passed into the call?
Thanks.
colin
Quoted 5 lines
That would be good.
--david
Quoted 7 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Colin Ngam
Colin> I do not know whether using the same set of flags i.e.GFP_XXX Colin> would overload it's meanings to other MM routines but I Colin> really like the a flag parameter, that will provide Colin> capability for Platform specific hints.
Quoted 5 lines
Overloading the GFP flags with platform-specific DMA stuff is certainly not an option.
If you want to argue for yet-another-flags argument, go ahead and do so. Be prepared for having to make a VERY strong argument...
David,
I was argueing for using the flag arguement and define it's own set of values versus using the GFP_XXX defines. And not an additional flag arguement.
Thanks.
colin
Quoted 3 lines
--davidQuoted 4 lines · David Mosberger
James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (
GFP_DMA32or some such).
For Streaming DMA, I'd rather stick with the current scheme. The main reason is 64-bit addressing support will be hard coded in the each driver as appropriate. ie it's not a parameter than needs to be passed in with each call to dma_map_single().
I'm assuming there is some performance impact to adding another argument to the existing dma_* calls. (more instructions and another register consumed). That might be a bad assumption for ia64.
Since Consistent DMA allocations aren't in the performance path normally, implementation details here don't matter as much (to me).
One of the original goals for my talk at OLS was to propose extensions to 2.4.x to provide "DMA Hints". This would help chipset specific DMA support pick "optimal" settings for DMA operations. I was thinking of redefining "direction" parameter to be "hints". It would continue to support Read or Write "hint" but then use upper bits as flags for other things (don't want to completely spoil my talk). But I can't justify the change based on the performance differences I've measured to date and probably will not offer a proposal. I haven't gotten to more complex testing (only single card tests so far) and HPUX certainly saw performance benefits in a similar scheme on PARISC (using PCI, not PCI-X).
hth,
grant
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Grant Grundler
Grant> For Streaming DMA, I'd rather stick with the current scheme.
Nobody said anything about changing streaming DMA. The discussion was purely about coherent/non-coherent allocations.
--david
Quoted 6 lines · Colin Ngam
Hi,
The Altix PCI-X Bridge chipset expects devices that runs in PCI-X mode to always be 64Bit DMA Address capable(DAC). This requirement is directly in conflict with both the
pci_alloc_consistent()anddma_alloc_coherent()interfaces.
I don't (yet) see the conflict.
Can't the 32-bit address can be extended to 64-bits as described in the dma_alloc_coherent() interface? Are you running out of 32-bit memory?
Quoted 4 lines
Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.
[email protected] proposed this before. On April 23, 2002, davem wrote:
| %99 of PCI chips out there do not support DAC addressing for things | like descriptor tables etc. So it's not a matter of "well" it's | a matter of "at all".
If you want the complete thread, check ia64-linux 2002 mail archives.
hth,
grant
Quoted 3 lines · Grant Grundler
Can't the 32-bit address can be extended to 64-bits as described in the
dma_alloc_coherent()interface? Are you running out of 32-bit memory?
I found the answer to my question here:
https://lists.linuxia64.org/archives//linux-ia64/2002-April/003418.html
Quoted 1 line
If you want the complete thread, check ia64-linux 2002 mail archives.
/me happened to take his own advice this time...
grant
ps. "cygnus.com" is a dead domain. IIRC, rth is at redhat now. trying that.
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 08:17:58 -0500
Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.
All devices I am aware of do not support 64-bit addresses for their descriptors, only 32-bit.
There is a huge difference between using a "64-bit address" and requiring to generate a DAC cycle, the latter of which is probably what you need and that is independant of what kind of address pci_alloc_consistent() is giving you.
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 10:05:28 -0500
I actually like the flag as a parameter because it allows platform specific extensions to pass in platform specific capabilities that can be ignored by platform that does not support these capabilities.
I hate this concept. I can't think of any attribute that all platforms can't take advantage of in some way and we'd actually want drivers to use.
Here be the way to private platform-specific interfaces, which is totally against the reason we have these interfaces in the first place.
From: Grant Grundler <[email protected]>
Date: Thu, 15 May 2003 12:06:22 -0700
This would help chipset specific DMA support pick "optimal" settings for DMA operations.
I vehemently will fight any such rediculious ideas.
The whole concept behind these portable DMA apis is to avoid platforms specific junk, it all must be generic and apply to everyone.
I was thinking of redefining "direction" parameter to be "hints". It would continue to support Read or Write "hint" but then use upper bits as flags for other things (don't want to completely spoil my talk).
Do not mix "must" parameters such as read/write protections with "it would be nice if" flag bits.
Quoted 10 lines · David S. Miller
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 08:17:58 -0500Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.
All devices I am aware of do not support 64-bit addresses for their descriptors, only 32-bit.
PCI-X devices are required to support 64 bit addrs. They'll only generate SAC if the top 32 bits are 0, but our platform is incapable of handling this, since the IOMMU can't be used in PCI-X mode (i.e. we need high order bits for PCI-X devices).
Quoted 4 lines
There is a huge difference between using a "64-bit address" and requiring to generate a DAC cycle, the latter of which is probably what you need and that is independant of what kind of address
pci_alloc_consistent()is giving you.
Unfortunately, we need both for PCI-X on Altix. How about this:
drivers that support PCI-X cards are required to program the full 64 bits of a pci_alloc_consistent DMA address, the top 32 bits of which can be zero for most platforms. That means that the device will only generate a SAC for most platforms, but on platforms like ours, the top 32 bits will contain useful information, so a DAC will be generated.
Thanks,
Jesse
Quoted 2 lines · David S. Miller
Do not mix "must" parameters such as read/write protections with "it would be nice if" flag bits.
R/W "hints" is not required for ia64/parisc IOMMUs. AFAIK, only swiotlb requires it and ZX1 IOMMU support has a one performance optimization which makes use of R/W hint.
In short, "must" only applies to some subset of working platforms.
But in general I agree, hints should be useful to most architectures. Having too many platform specific hints becomes hell for driver writers.
grant
Quoted 7 lines · Jesse Barnes
Quoted 2 lines
All devices I am aware of do not support 64-bit addresses for their descriptors, only 32-bit.
PCI-X devices are required to support 64 bit addrs. They'll only generate SAC if the top 32 bits are 0, but our platform is incapable of handling this, since the IOMMU can't be used in PCI-X mode (i.e. we need high order bits for PCI-X devices).
I think you're missing Dave's point. While the card is required to support DAC that doesn't mean the device on the card is necessarily capable of being handed a full 64-bit address to DMA into.
From the way he's talking, I'd surmise tg3 is a good example -- I suspect the ring buffers only support 32-bit addresses.
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Quoted 8 lines · David S. Miller
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 08:17:58 -0500Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.
Quoted 7 lines
All devices I am aware of do not support 64-bit addresses for their descriptors, only 32-bit.
There is a huge difference between using a "64-bit address" and requiring to generate a DAC cycle, the latter of which is probably what you need and that is independant of what kind of address
pci_alloc_consistent()is giving you.
David,
I am quite confuse with what you are saying.
pci_alloc_consistent(), as documented must return a 32bits DMA in the dma mapped address in "dma_addr_t".
Right now, on our platform, the Upper 32 Bits of this address is always 0s and the Lower 32 Bits has the IOMMU mapped DMA address. When this address is given to the card, SAC is generated because the Upper 32 Bits are 0s - if the whole 64bits are given to the card.
Now, for PCI-X devices, we need to generate an IOMMU mapped DMA address that has the Upper 32 bits non 0s. When the driver gives this address to the card, it will generate a DAC for the DMA.
From what I understand from the pci_alloc_consistant() interface, we cannot return a "dma_addr_t" that has the Upper 32bits NON 0s. Is that TRUE? If it is, our platform cannot deal with this restriction .. therefore we are requesting
for this extension.
Thanks.
colin
Quoted 11 lines · James Bottomley
Quoted 10 lines · Colin Ngam
Right now, on our platform, the Upper 32 Bits of this address is always 0s and the Lower 32 Bits has the IOMMU mapped DMA address. When this address is given to the card, SAC is generated because the Upper 32 Bits are 0s - if the whole 64bits are given to the card.
Now, for PCI-X devices, we need to generate an IOMMU mapped DMA address that has the Upper 32 bits non 0s. When the driver gives this address to the card, it will generate a DAC for the DMA.
Quoted 1 line
From what I understand from the
pci_alloc_consistant()interface, we cannot
Quoted 8 lines
Quoted 4 lines
return a "
dma_addr_t" that has the Upper 32bits NON 0s. Is that TRUE? If it is, our platform cannot deal with this restriction .. therefore we are requesting
for this extension.OK, now you've confused me. I thought you were saying that your PCI hardware generates two type of addresses: one for DAC that *doesn't* go through the IOMMU and one for SAC that does?
Yes, you are correct .. we have the capability to generate a SAC or DAC address in PCI Mode only.
Quoted 9 lines
In which case, SAC should be able to go through the IOMMU to reach anwhere in memory, and thus
pci_alloc_consistentshould work for you now.Or are you saying it's more efficient to go via the DAC non-iommu cycle and you'd like to do that preferentially if you could (which is a feature request, not a hard requirement)?
However, in PCI-X mode, we do not have the capability to generate SAC addresses. And yes, you are also correct that it is very much efficient on our architecture to generate DAC for all(PCI/PCI-X)
So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.
Hope that is clear ..
Thanks.
colin
Quoted 3 lines
James
Quoted 9 lines · Colin Ngam
Right now, on our platform, the Upper 32 Bits of this address is always 0s and the Lower 32 Bits has the IOMMU mapped DMA address. When this address is given to the card, SAC is generated because the Upper 32 Bits are 0s - if the whole 64bits are given to the card.
Now, for PCI-X devices, we need to generate an IOMMU mapped DMA address that has the Upper 32 bits non 0s. When the driver gives this address to the card, it will generate a DAC for the DMA.
Quoted 5 lines
Quoted 1 line
From what I understand from the
pci_alloc_consistant()interface, we cannotreturn a "
dma_addr_t" that has the Upper 32bits NON 0s. Is that TRUE? If it is, our platform cannot deal with this restriction .. therefore we are requesting
for this extension.
OK, now you've confused me. I thought you were saying that your PCI hardware generates two type of addresses: one for DAC that *doesn't* go through the IOMMU and one for SAC that does?
In which case, SAC should be able to go through the IOMMU to reach anwhere in memory, and thus pci_alloc_consistent should work for you now.
Or are you saying it's more efficient to go via the DAC non-iommu cycle and you'd like to do that preferentially if you could (which is a feature request, not a hard requirement)?
James
Quoted 7 lines · Colin Ngam
However, in PCI-X mode, we do not have the capability to generate SAC addresses. And yes, you are also correct that it is very much efficient on our architecture to generate DAC for all(PCI/PCI-X)
So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.
Hope that is clear ..
Actually, no. This seems to imply that your PCI-X cards can *never* access the first 4GB of memory (because they can't use an IOMMU and they have to generate a DAC cycle with non zero in bits 63--32).
James
Quoted 5 lines · James Bottomley
Quoted 4 lines · Colin Ngam
However, in PCI-X mode, we do not have the capability to generate SAC addresses. And yes, you are also correct that it is very much efficient on our architecture to generate DAC for all(PCI/PCI-X)
Quoted 7 lines
Quoted 3 lines
So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.
Hope that is clear ..
Actually, no. This seems to imply that your PCI-X cards can *never* access the first 4GB of memory (because they can't use an IOMMU and they have to generate a DAC cycle with non zero in bits 63--32).
Actually, our system does not have Physical Memory Address smaller than 48Bits - give or take couple bits ... Hence, 32Bits Mapped DMA address maps to 48Bits System Physical Address.
64Bits Mapped DMA address maps directly to System Physical Memory with Bits 63-49 containing transfer attributes like hardware support for byte swapping .. if needed(e.g. Big Endian Cards).
Thanks.
colin
Quoted 3 lines
James
From: Jesse Barnes <[email protected]>
Date: Thu, 15 May 2003 13:24:12 -0700
How about this: drivers that support PCI-X cards are required to program the full 64 bits of a pci_alloc_consistent DMA address, the top 32 bits of which can be zero for most platforms.
How about this: you put the device attributes where they belong, in the device struct.
Quoted 8 lines · David S. Miller
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 17:14:11 -0500Now, for PCI-X devices, we need to generate an IOMMU mapped DMA address that has the Upper 32 bits non 0s.
Then add a
consistent_dma_maskattribute to the device struct, problem solved.
Hi David,
I do not understand what you are saying .. how would a consistent_dma_mask attribute in the device struct helps pci_alloc_consistent() to return a dma hanlde
that can have the Upper 32 Bits Non 0's, and, moreover, these Upper 32 Bits are not constants, but can vary greatly, and is part of the 64 bit Physical System
Memory Address?
Thanks.
colin
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin Ngam From: Matthew Wilcox <[email protected]>
Date: Thu, 15 May 2003 22:29:18 +0100
From the way he's talking,
My main point is that device attributes belong IN THE DEVICE not in some umpteenth new parameter to the DMA interfaces.
Just put consistent_dma_mask into the device struct next to the existing dma_mask, voila problem solved.
I have two major problems with the generic DMA interfaces and how they are being handled in general:
1) Their use is entirely inappropriate in unary-bus devices. There is no way someone can justify some PCI driver continually asking itself whether it is a PCI device or not on each DMA API call.
2) Additions to the parameters in the interfaces is not given enough thought. Each new parameter you add is a discrete failure of the API. It was built to be simple, and new parameters do nothing but add complexity.
I cannot even fathom that someone would suggest to pass device attributes into the DMA interfaces. That is just garbage and unnecessary bloat.
Finally, I see nobody tackling the real hard bugs we have in the DMA API. These should be fixed first before adding random DMA behaviorial attributes. For example, pci_dma_sync_*() sucks because it only syncs in one direction, device to cpu, there is nothing available to sync things back from cpu-->device. This prevents MIPS from using the devices that need to do syncs without unmaps (eepro100 is but one example, USB host controller drivers want this too).
Quoted 11 lines · David S. Miller
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:25:54 -0500Actually, our system does not have Physical Memory Address smaller than 48Bits - give or take couple bits ... Hence, 32Bits Mapped DMA address maps to 48Bits System Physical Address.
If you support mapping 32-bit DMA bus spaces to the full 48-bit physical memory spaces, then why do you even need >4GB addressing?
Saying that PCI-X devices can only generate DAC cycles is bogus,
I did not say that. it is very obvious that PCI-X devices can generate either SAC or DAC.
What I did say is that, on our platform, in PCI-X mode, our PCI-X Bridge Chipset do NOT support 32-Bit DMA Bus Space.
This does not break any PCI-X specs.
However, the pci_alloc_consistent() interface is documented to ONLY return a 32-Bit DMA Bus Space .. we like this to be extended so that PCI-X device drivers or PCI drivers can consumed 64-Bit DMA Bus Space, if they have the capability.
Thanks.
colin
Quoted 10 lines
I have PCI-X tg3 cards in my systems and they function just fine. Or does this rule only apply when the device is on a PCI-X bus?
This all sounds seriously broken to me, I thought the issue was that
a) the addresses could not be translated, it was direct phys-->virt and b) you can't guarentee that the physical address of memory allocated on any particular node is <4GB.
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 17:14:11 -0500
Now, for PCI-X devices, we need to generate an IOMMU mapped DMA address that has the Upper 32 bits non 0s.
Then add a consistent_dma_mask attribute to the device struct, problem solved.
From: James Bottomley <[email protected]>
Date: 15 May 2003 17:47:04 -0500
OK, now you've confused me.
Their hardware platform cannot guarentee low-32bit memory allocations to devices because each node can have physical memory starting at
Quoted 1 line
4GB etc etc.
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 17:44:57 -0500
However, in PCI-X mode, we do not have the capability to generate SAC addresses.
The device can generate addresses that have the top-32bits 0 though.
You problem is rather that you cannot guarentee to be able to allocate physical memory which is <4GB on each and every node.
That is very different from a "PCI-X issue". It is a "this NUMA platform" issue.
Quoted 8 lines · David S. Miller
From: James Bottomley <[email protected]>
Date: 15 May 2003 17:47:04 -0500OK, now you've confused me.
Their hardware platform cannot guarentee low-32bit memory allocations to devices because each node can have physical memory starting at
Quoted 1 line
4GB etc etc.
Hi,
Something else I do not understand .. in the PCI world, what does Physical Memory Address got to do with 32Bit Mapped DMA Bus Address?
Not only do we not guarantee that we cannot generate 32 Bit Memory Allocations .. we plain old do not have Memory Addresses smaller below 32Bits .. well, below 48 Bits for that matter .. I have not seen any of the drivers broken yet - e.g. qlXXX, acenic, tg3 etc.
Thanks.
colin
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:25:54 -0500
Actually, our system does not have Physical Memory Address smaller than 48Bits - give or take couple bits ... Hence, 32Bits Mapped DMA address maps to 48Bits System Physical Address.
If you support mapping 32-bit DMA bus spaces to the full 48-bit physical memory spaces, then why do you even need >4GB addressing?
Saying that PCI-X devices can only generate DAC cycles is bogus, I have PCI-X tg3 cards in my systems and they function just fine. Or does this rule only apply when the device is on a PCI-X bus?
This all sounds seriously broken to me, I thought the issue was that
a) the addresses could not be translated, it was direct phys-->virt and b) you can't guarentee that the physical address of memory allocated on any particular node is <4GB.
Quoted 9 lines · David S. Miller
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:32:31 -0500I do not understand what you are saying .. how would a
consistent_dma_maskattribute in the device struct helpspci_alloc_consistent()to return a dma hanlde that can have the Upper 32 Bits Non 0's, and, moreover, these Upper 32 Bits are not constants, but can vary greatly, and is part of the 64 bit Physical System Memory Address?
Quoted 6 lines
The mask says that the consistent address "can" have the upper 32-bits non-zero. Then you define a
consistent_dma_attrsbitmask that can define things like "requires >4GB address" etc.Again, all of this is device attribute stuff and not an issue concerning the DMA api calls.
Hi David,
Implementation details aside, you are agreeing that the pci_alloc_consistent() interface can return an address > 4GB and we can document as such?
Thanks.
colin
Quoted 4 lines
But I am ever further confused, how do things like tg3 cards even function TODAY on any PCI-X systems.
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:32:31 -0500
I do not understand what you are saying .. how would a consistent_dma_mask attribute in the device struct helps pci_alloc_consistent() to return a dma hanlde that can have the Upper 32 Bits Non 0's, and, moreover, these Upper 32 Bits are not constants, but can vary greatly, and is part of the 64 bit Physical System Memory Address?
The mask says that the consistent address "can" have the upper 32-bits non-zero. Then you define a consistent_dma_attrs bitmask that can define things like "requires >4GB address" etc.
Again, all of this is device attribute stuff and not an issue concerning the DMA api calls.
But I am ever further confused, how do things like tg3 cards even function TODAY on any PCI-X systems.
Quoted 10 lines · David S. Miller
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:45:42 -0500Something else I do not understand .. in the PCI world, what does Physical Memory Address got to do with 32Bit Mapped DMA Bus Address?
Many platforms direct map, that is what I assumed was happening here and causing the problems.
Quoted 7 lines
I have not seen any of the drivers broken yet - e.g. qlXXX, acenic, tg3 etc.
It only works because these devices happen to allow >32-bit DMA addresses for descriptors. You will find eventually that many comodity pieces of PCI-X hardware will opt to only support 32-bit DMA addressing for descriptor tables and whatnot.
Hi David,
It is in the PCI-X Spec that all PCI-X devices in PCI-X Mode have to support DAC.
It did not say that it cannot support SAC or should not support SAC, but it has to support DAC.
Thanks.
colin
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:40:22 -0500
What I did say is that, on our platform, in PCI-X mode, our PCI-X Bridge Chipset do NOT support 32-Bit DMA Bus Space.
Sounds like you can't use a whole swath of devices then. What an entirely unfortunate design decision.
Anyways, now that I understand the problem, merely adding the consistent_dma_mask attribute to the device struct will allow drivers to indicate this capability just fine. The driver goes:
if (pci_set_consistent_dma_mask(pdev, ~(u64)0)) goto probe_error;
and then the platform pci_alloc_consistent() can do whatever is appropriate.
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:45:42 -0500
Something else I do not understand .. in the PCI world, what does Physical Memory Address got to do with 32Bit Mapped DMA Bus Address?
Many platforms direct map, that is what I assumed was happening here and causing the problems.
I have not seen any of the drivers broken yet - e.g. qlXXX, acenic, tg3 etc.
It only works because these devices happen to allow >32-bit DMA addresses for descriptors. You will find eventually that many comodity pieces of PCI-X hardware will opt to only support 32-bit DMA addressing for descriptor tables and whatnot.
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:51:35 -0500
Implementation details aside, you are agreeing that the pci_alloc_consistent() interface can return an address > 4GB and we can document as such?
I can return an address that matches the requirements specified by dev->consistent_dma_mask which is set by default to 0xffffffff and must be changed by a device driver if it wants to use a non-default value.
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:57:09 -0500
It is in the PCI-X Spec that all PCI-X devices in PCI-X Mode have to support DAC.
Having to support DAC cycles does not mean that it must support full 64-bit DMA addresses in it's descriptors.
A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.
Quoted 9 lines · David S. Miller
From: Jesse Barnes <[email protected]>
Date: Thu, 15 May 2003 13:24:12 -0700How about this: drivers that support PCI-X cards are required to program the full 64 bits of a
pci_alloc_consistentDMA address, the top 32 bits of which can be zero for most platforms.How about this: you put the device attributes where they belong, in the device struct.
That's fine, it'll work for us. I just wanted to make sure that dma_alloc_consistent could return a 64 bit DMA addr. Having a consistent_dma_mask will allow us to do that for devices that support it. We'll have to switch devices that can't into PCI mode for them to operate correctly on our hw.
Also, are you saying that tg3 doesn't support all 64 bits of the DMA address? If so, I'm not sure how it's working on our platform either, unless we're forcing it into PCI mode; I'll have to check.
Thanks,
Jesse
Quoted 5 lines · David S. Miller
Having to support DAC cycles does not mean that it must support full 64-bit DMA addresses in it's descriptors.
A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.
This is certainly a property of the aic79xx driver (only using 32 bit descriptor tables). It has this nice code in the driver for doing it:
aic79xx.c:
/* * Although we can dma data above 4GB, our * "consistent" memory is below 4GB for * space efficiency reasons (only need a 4byte * address). For this reason, we have to reset * our dma mask when doing allocations. */ if (ahd->dev_softc != NULL) ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF); *vaddr = pci_alloc_consistent(ahd->dev_softc, dmat->maxsize, &map->bus_addr); if (ahd->dev_softc != NULL) ahd_pci_set_dma_mask(ahd->dev_softc, ahd->platform_data->hw_dma_mask);
And it comes in PCI-X flavours.
James
From: Jesse Barnes <[email protected]>
Date: Thu, 15 May 2003 17:29:05 -0700
Also, are you saying that tg3 doesn't support all 64 bits of the DMA address? If so, I'm not sure how it's working on our platform either, unless we're forcing it into PCI mode; I'll have to check.
I believe tg3 does, but some cost-sensitive PCI-X devices might decide not to.
Quoted 27 lines · James Bottomley
Quoted 5 lines · David S. Miller
Having to support DAC cycles does not mean that it must support full 64-bit DMA addresses in it's descriptors.
A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.
This is certainly a property of the aic79xx driver (only using 32 bit descriptor tables). It has this nice code in the driver for doing it:
aic79xx.c:/* * Although we can dma data above 4GB, our * "consistent" memory is below 4GB for * space efficiency reasons (only need a 4byte * address). For this reason, we have to reset * our dma mask when doing allocations. */ if (ahd->dev_softc != NULL) ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF); *vaddr = pci_alloc_consistent(ahd->dev_softc, dmat->maxsize, &map->bus_addr); if (ahd->dev_softc != NULL) ahd_pci_set_dma_mask(ahd->dev_softc, ahd->platform_data->hw_dma_mask);
Quoted 1 line
And it comes in PCI-X flavours.
Yes agreed. And I think this is fine. But let's not have the interface prevents 64bits drivers that wants to do DAC. that's all I am asking ..
Thanks.
colin
Quoted 3 lines
James
From: James Bottomley <[email protected]>
Date: 15 May 2003 19:31:20 -0500
if (ahd->dev_softc != NULL) ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF); *vaddr = pci_alloc_consistent(ahd->dev_softc, dmat->maxsize, &map->bus_addr);
Funny how whoever wrote this didn't understand that pci_set_dma_mask() has zero effect on pci_alloc_consistent() behavior.
Someone please tell Justin to fix this broken bit of code.
Quoted 8 lines · David S. Miller
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:57:09 -0500It is in the PCI-X Spec that all PCI-X devices in PCI-X Mode have to support DAC.
Having to support DAC cycles does not mean that it must support full 64-bit DMA addresses in it's descriptors.
Hi David,
I do not believe it is within spec to perform DAC when the Upper 32bits are 0s. Current usage of pci_alloc_consistent(), even if you want to do DAC, you cannot, because the interface is defined to return only 32bits worth of addresses.
However, I think we are beyond this point now.
Thanks.
colin
Quoted 4 lines
A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.
Quoted 23 lines · James Bottomley
Quoted 2 lines
A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.
This is certainly a property of the aic79xx driver (only using 32 bit descriptor tables). It has this nice code in the driver for doing it:
aic79xx.c:/* * Although we can dma data above 4GB, our * "consistent" memory is below 4GB for * space efficiency reasons (only need a 4byte * address). For this reason, we have to reset * our dma mask when doing allocations. */ if (ahd->dev_softc != NULL) ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF); *vaddr = pci_alloc_consistent(ahd->dev_softc, dmat->maxsize, &map->bus_addr); if (ahd->dev_softc != NULL) ahd_pci_set_dma_mask(ahd->dev_softc, ahd->platform_data->hw_dma_mask);
Quoted 1 line
And it comes in PCI-X flavours.
Ah, ok (set_dma_mask vs. alloc_consistent bugs aside). I think there was some confusion about the fact that the PCI-X spec. says that cards must support DAC vs. them actually supporting 64 bit DMA addresses. They all have to do the former, but not the latter. Also, using an IOMMU is different than using direct mapping. Our hw can do both for PCI cards that only handle 32 bit DMA addresses. IOMMU mapping allows you to map pages anywhere in memory, while direct mapping gives you a static 2GB window into system memory.
Hope this clears things up a little.
Thanks,
Jesse
Quoted 10 lines · David S. Miller
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 19:36:55 -0500I do not believe it is within spec to perform DAC when the Upper 32bits are 0s.
That is an illogical requirement in that it means PCI-X mode is illegal in the presence of such addresses. It means that PCI-X devices cannot work in x86 systems with <4GB of memory which is obviously not true.
No that is not the case.
If you have an address that only has the Lower 32bits set, your card do a SAC and not a DAC.
SAC is not prohibited in PCI-X mode. But DAC has to be supported by the card.
Someone please correct me if I misread.
Thanks.
colin
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 19:36:55 -0500
I do not believe it is within spec to perform DAC when the Upper 32bits are 0s.
That is an illogical requirement in that it means PCI-X mode is illegal in the presence of such addresses. It means that PCI-X devices cannot work in x86 systems with <4GB of memory which is obviously not true.
Quoted 2 lines · David S. Miller
But I am ever further confused, how do things like tg3 cards even function TODAY on any PCI-X systems.
They work just fine on most platforms with 32-bit addresses. The issue is with what the SGI platform can support, not what the PCI-X devices can support.
BTW, adding consistent_dma_mask to struct pci_dev and a corresponding pci_set_consistent_dma_mask() is a good idea. I think it would simplify our SBA code a bit.
thanks,
grant
From: Grant Grundler <[email protected]>
Date: Thu, 15 May 2003 18:20:51 -0700
BTW, adding consistent_dma_mask to struct pci_dev and a corresponding pci_set_consistent_dma_mask() is a good idea. I think it would simplify our SBA code a bit.
Great, I'm glad this is resolved. :-)
Quoted 3 lines · Colin Ngam
I do not understand what you are saying .. how would a
consistent_dma_maskattribute in the device struct helpspci_alloc_consistent()to return a dma hanlde that can have the Upper 32 Bits Non 0's...
Colin,
davem has offered a counter proposal that makes way more sense that adding parameters to the dma_*() API. It won't change existing code or existing behaviors. And yes, Documentation/DMA_mapping.txt (API documentation) has to change to reflect that.
Device Drivers SGI cares about will have to be modified to call pci_set_consistent_dma_mask() before calling pci_alloc_consistent().
hth,
grant
Quoted 11 lines · Grant Grundler
Quoted 3 lines · Colin Ngam
I do not understand what you are saying .. how would a
consistent_dma_maskattribute in the device struct helpspci_alloc_consistent()to return a dma hanlde that can have the Upper 32 Bits Non 0's...Colin,
davem has offered a counter proposal that makes way more sense that adding parameters to the dma_*() API. It won't change existing code or existing behaviors. And yes,Documentation/DMA_mapping.txt(API documentation) has to change to reflect that.
Quoted 5 lines
Device Drivers SGI cares about will have to be modified to call
pci_set_consistent_dma_mask()before callingpci_alloc_consistent().hth,
grant
Hi Grant,
That would be very fine :-)
Thanks.
colin
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin NgamQuoted 6 lines · David S. Miller
My main point is that device attributes belong IN THE DEVICE not in some umpteenth new parameter to the DMA interfaces.
Just put
consistent_dma_maskinto the device struct next to the existingdma_mask, voila problem solved.
Quoted 7 lines
I have two major problems with the generic DMA interfaces and how they are being handled in general:
1) Their use is entirely inappropriate in unary-bus devices. There is no way someone can justify some PCI driver continually asking itself whether it is a PCI device or not on each DMA API call.
Where's that in the API? The PA-RISC implementation copes with about four different buses and nowhere does it have a switch trying to ascertain which bus it's talking about (it uses the dev->platform_data field to cache the relevant bus attributes).
Quoted 8 lines
2) Additions to the parameters in the interfaces is not given enough thought. Each new parameter you add is a discrete failure of the API. It was built to be simple, and new parameters do nothing but add complexity.
I cannot even fathom that someone would suggest to pass device attributes into the DMA interfaces. That is just garbage and unnecessary bloat.
You mean like xadding gfp flags to the allocator?
Quoted 7 lines
Finally, I see nobody tackling the real hard bugs we have in the DMA API. These should be fixed first before adding random DMA behaviorial attributes. For example, pci_dma_sync_*() sucks because it only syncs in one direction, device to cpu, there is nothing available to sync things back from cpu-->device. This prevents MIPS from using the devices that need to do syncs without unmaps (eepro100 is but one example, USB host controller drivers want this too).
OK, I don't get this yet, doesn't DMA_TO_DEVICE do what you want? what am I missing?
James
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin Ngam From: James Bottomley <[email protected]>
Date: 15 May 2003 21:05:12 -0500
Quoted 7 lines
Finally, I see nobody tackling the real hard bugs we have in the DMA API. These should be fixed first before adding random DMA behaviorial attributes. For example, pci_dma_sync_*() sucks because it only syncs in one direction, device to cpu, there is nothing available to sync things back from cpu-->device. This prevents MIPS from using the devices that need to do syncs without unmaps (eepro100 is but one example, USB host controller drivers want this too).
OK, I don't get this yet, doesn't DMA_TO_DEVICE do what you want? what am I missing?
You don't understand, a driver does:
pci_dma_sync_single(... buf);
This transfers ownership of the buffer from device back to the CPU. It guarentees that the CPU will see uptodate contents of the buffer and that it may make modifications to 'buf'.
Then it looks at 'buf', and decides that it wants to give the buffer back to the device. What does it call to transfer ownership back from CPU to device?
Answer: No such API call exists.
This is a big problem for implementations that need to flush CPU caches in order to implement the cpu<-->device buffer ownership transfer properly.
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin NgamQuoted 12 lines
You don't understand, a driver does:
pci_dma_sync_single(... buf);This transfers ownership of the buffer from device back to the CPU. It guarentees that the CPU will see uptodate contents of the buffer and that it may make modifications to 'buf'.
Then it looks at 'buf', and decides that it wants to give the buffer back to the device. What does it call to transfer ownership back from CPU to device?
Quoted 5 lines
Answer: No such API call exists.
This is a big problem for implementations that need to flush CPU caches in order to implement the cpu<-->device buffer ownership transfer properly.
It was my impression that a CPU cache invalidate cycle relinquishes the cache line (and thus leaves the line free to be owned by the device). The DMA_TO_DEVICE of dma_sync.. does this, why isn't that sufficient? I grant that CPU does auto ownership, so if the CPU ever touches the data again the cache line is pulled back in again.
Is the issue that we have no way of preventing the CPU from pulling in the cache line if it is referenced?
James
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin Ngam From: James Bottomley <[email protected]>
Date: 15 May 2003 21:26:14 -0500
It was my impression that a CPU cache invalidate cycle relinquishes the cache line (and thus leaves the line free to be owned by the device). The DMA_TO_DEVICE of dma_sync.. does this, why isn't that sufficient? I grant that CPU does auto ownership, so if the CPU ever touches the data again the cache line is pulled back in again.
What flushes the cpu cache after the cpu writes to 'buf' so that the device sees what the CPU wrote there? Consider copy-back L2 caches like MIPS has.
Is the issue that we have no way of preventing the CPU from pulling in the cache line if it is referenced?
The "issue" is that we need a pci_dma_sync_*_to_device() call added. We should also remane pci_dma_sync_*() to pci_dma_sync_*_to_cpu() to make all of the naming consistent and update all the drivers.
Quoted 10 lines · David S. Miller
From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 08:17:58 -0500Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.
All devices I am aware of do not support 64-bit addresses for their descriptors, only 32-bit.
The Qlogic and LSI Logic SCSI and FC devices support 64 bit addresses for their descriptors. Also the old Adaptec (now JNI) 1160. I do not believe this is unusual. Certainly all PCI-X devices should support 64 bit addresses for descriptors.
The Tigon3 supports 64 bit addresses for descriptors.
Quoted 4 lines
There is a huge difference between using a "64-bit address" and requiring to generate a DAC cycle, the latter of which is probably what you need and that is independant of what kind of address
pci_alloc_consistent()is giving you.
Not independent. Because if pci_alloc_consistent gives an address in which any of bits 32-63 are set, then the adapter will use a DAC to access the descriptor. I guess you're saying that some adapters may support more than 32 bits but fewer than 64. Such devices would not work on our platform. Which PCI-X devices do not support the full 64 bit address space? I don't know of any.
jeremy
From: Jeremy Higdon <[email protected]>
Date: Thu, 15 May 2003 23:47:55 -0700
Please read further into your mailbox before replying, all of the misunderstandings and issues have been resolved already.
Quoted 6 lines · David S. Miller
From: Jeremy Higdon <[email protected]>
Date: Thu, 15 May 2003 23:47:55 -0700Please read further into your mailbox before replying, all of the misunderstandings and issues have been resolved already.
I did. I didn't see that issue addressed. I guess someone did educate you, perhaps in a message I did not see.
Quoted 7 lines · James Bottomley
Quoted 6 lines · Colin Ngam
However, in PCI-X mode, we do not have the capability to generate SAC addresses. And yes, you are also correct that it is very much efficient on our architecture to generate DAC for all(PCI/PCI-X)
So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.
Quoted 7 lines
Quoted 1 line
Hope that is clear ..
Actually, no. This seems to imply that your PCI-X cards can *never* access the first 4GB of memory (because they can't use an IOMMU and they have to generate a DAC cycle with non zero in bits 63--32).
James
I don't know if it was clear or not in the ensuing replies, but we don't present direct physical addresses to PCI-X cards. Instead it's an physical address with various bits ORed in at the top to set routing and attribute information. So if you want to DMA to physical address 0x1000 in the system, the PCI address might be 0x9100000000001000.
If it was already clear, sorry for the duplication.
jeremy
Quoted 9 lines · David S. Miller
From: James Bottomley <[email protected]>
Date: 15 May 2003 21:26:14 -0500It was my impression that a CPU cache invalidate cycle relinquishes the cache line (and thus leaves the line free to be owned by the device). The
DMA_TO_DEVICEofdma_sync.. does this, why isn't that sufficient? I grant that CPU does auto ownership, so if the CPU ever touches the data again the cache line is pulled back in again.
Quoted 3 lines
What flushes the cpu cache after the cpu writes to 'buf' so that the device sees what the CPU wrote there? Consider copy-back L2 caches like MIPS has.
Isn't that what dma_cache_wback_inv does? It flushes cpu cache to memory so that it is visible to devices. Is there some additional semantic desired?
jeremy
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin Ngam From: Jeremy Higdon <[email protected]>
Date: Fri, 16 May 2003 00:48:12 -0700
Isn't that what dma_cache_wback_inv does?
That function is not part of the DMA api, and no portable driver should use this dma_cache_wback_inv() thing.
Quoted 9 lines · James Bottomley
Quoted 5 lines · David S. Miller
Having to support DAC cycles does not mean that it must support full 64-bit DMA addresses in it's descriptors.
A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.
This is certainly a property of the aic79xx driver (only using 32 bit descriptor tables). It has this nice code in the driver for doing it:
This is a driver choice, rather than an adapter limitation, right? Or is the adapter limited to 32 bit addresses for map->bus_addr (the PCI-X flavor)?
Quoted 21 lines
aic79xx.c:/* * Although we can dma data above 4GB, our * "consistent" memory is below 4GB for * space efficiency reasons (only need a 4byte * address). For this reason, we have to reset * our dma mask when doing allocations. */ if (ahd->dev_softc != NULL) ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF); *vaddr = pci_alloc_consistent(ahd->dev_softc, dmat->maxsize, &map->bus_addr); if (ahd->dev_softc != NULL) ahd_pci_set_dma_mask(ahd->dev_softc, ahd->platform_data->hw_dma_mask);And it comes in PCI-X flavours.
James
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin NgamQuoted 3 lines · David S. Miller
The "issue" is that we need a pci_dma_sync_*_to_device() call added. We should also remane pci_dma_sync_*() to pci_dma_sync_*_to_cpu() to make all of the naming consistent and update all the drivers.
OK, I get it now (I was thinking really in terms of the old writeback/invalidate interface where you had to know how to place them).
You mean we need an interface that is called when the CPU hands off the region to the device and again when the CPU reclaims it (so we always get the writeback or invalidate in the correct place).
How about just adding an ownership attribute to dma_sync..? It would either be DMA_OWNER_CPU or DMA_OWNER_DEVICE?
Of course, retrofitting something like this is going to be fun.
James
Quoted 3 lines · James Bottomley
You mean we need an interface that is called when the CPU hands off the region to the device and again when the CPU reclaims it (so we always get the writeback or invalidate in the correct place).
yes
Quoted 2 lines
How about just adding an ownership attribute to
dma_sync..? It would either beDMA_OWNER_CPUorDMA_OWNER_DEVICE?
I prefer davem's suggestion of a different function call (*_to_cpu and *_to_dev) rather than making the same function do completely different things based on parameters. Keeps API documentation and resulting implementations for each smaller.
Quoted 1 line
Of course, retrofitting something like this is going to be fun.
It's not so bad.
It's really only needed for non-coherent machines. Folks who care about those platforms probably already know where those calls belong.
grant
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin NgamQuoted 7 lines · Grant Grundler
Quoted 3 lines · James Bottomley
You mean we need an interface that is called when the CPU hands off the region to the device and again when the CPU reclaims it (so we always get the writeback or invalidate in the correct place).
yes
Quoted 7 lines
Quoted 2 lines
How about just adding an ownership attribute to
dma_sync..? It would either beDMA_OWNER_CPUorDMA_OWNER_DEVICE?I prefer davem's suggestion of a different function call (
*_to_cpuand*_to_dev) rather than making the same function do completely different things based on parameters. Keeps API documentation and resulting implementations for each smaller.
Hmm, another API vs adding properties. I'm easy on this. However, the fact that data direction is an attribute does give me a preference for having owner as an attribute too.
Quoted 6 lines
Quoted 1 line
Of course, retrofitting something like this is going to be fun.
It's not so bad.
It's really only needed for non-coherent machines. Folks who care about those platforms probably already know where those calls belong.
Well, it applies to partially coherent machines too. However, the point of the API is to make it so that the driver writer doesn't need to understand where the calls go, So the API encapsulates all the platform quirks in a simple set of rules of thumb.
Thus:
map region
Use region in driver
dma_sync ... DMA_OWNER_DEVICE .. and direction
device uses region (driver may not touch it)
dma_sync ... DMA_OWNER_CPU .. and direction
driver accesses region again (may repeat back to use region in driver)
unmap region
That's unambiguous and simple to articulate. The platform implementation does the necessary magic.
James
Quoted 13 lines · James Bottomley
Well, it applies to partially coherent machines too. However, the point of the API is to make it so that the driver writer doesn't need to understand where the calls go, So the API encapsulates all the platform quirks in a simple set of rules of thumb.
Thus:
map region
Use region in driver
dma_sync...DMA_OWNER_DEVICE.. and direction
Quoted 8 lines
device uses region (driver may not touch it)
dma_sync...DMA_OWNER_CPU.. and directiondriver accesses region again (may repeat back to use region in driver)
unmap region
Isn't the common use (eg, net devices):
buffer passed from lower levels
map buffer
(device owns buffer)
device uses buffer
unmap buffer
(cpu owns buffer)
free buffer
So, to use your example:
(cpu owns region)
map
(device owns region)
dma_sync...cpu
(cpu owns region)
dma_sync...device
(device owns region)
unmap
(cpu owns region)
Your suggestion seems to add in two extra sync calls (and thus extra code in the common case.)
Signature · 2 lines
Russell King ([email protected]) The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html
Quoted 2 lines · Jeremy Higdon
Certainly all PCI-X devices should support
64 bit addresses for descriptors.
I finally found the bit in the PCI-X spec hat mentions this:
(Section 1.8 "Bus Width")
| The following PCI-X requirements are different from conventional PCI: | 1. All devices that initiate memory transactions must be capable of | generating 64-bit addresses. | 2. Each device includes a configuration status bit that indicates | the width of that device's interface.
PCI-X also has a different command set.
It's definitely NOT just a PCI bus speed up.
thanks,
grant
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin NgamQuoted 2 lines · Russell King
Your suggestion seems to add in two extra sync calls (and thus extra code in the common case.)
Yes, sorry, needlessly complex example.
James
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"David" = David S Miller <[email protected]> writes:
David> Anyways, now that I understand the problem, merely adding the David> consistent_dma_mask attribute to the device struct will allow David> drivers to indicate this capability just fine. The driver David> goes:
David> if (pci_set_consistent_dma_mask(pdev, ~(u64)0)) goto
David> probe_error;
David> and then the platform pci_alloc_consistent() can do whatever is David> appropriate.
Hi Dave,
How does this patch look to you? It adds the generic code for setting the mask, it will then be up to Colin to fix the SGI SN2 specific code to honor it.
I also added a patch to the tg3 driver to set the mask as it seems to work just dandy with 64 bit consistent addresses.
Tested on ia64, but I see no reason why it should fail elsewhere.
Cheers,
Jes
drivers/net/tg3.c +6 -0
@@ -6743,6 +6743,12 @@/* Configure DMA attributes. */ if (!pci_set_dma_mask (pdev, (u64 )0xffffffffffffffff )) { pci_using_dac =1 ; +if (pci_set_consistent_dma_mask (pdev, + (u64 )0xffffffffffffffff )) { + printk(KERN_ERR PFX "Unable to obtain 64 bit DMA " + "for consistent allocations\n"); + goto err_out_free_res; + } }else { err =pci_set_dma_mask (pdev, (u64 )0xffffffff );if (err) {
drivers/pci/pci.c +12 -0
@@ -701,6 +701,17 @@return 0 ; } +int +pci_set_consistent_dma_mask (struct pci_dev *dev,u64 mask) +{ +if (!pci_dma_supported (dev, mask)) +return -EIO ; + + dev->consistent_dma_mask = mask; + +return 0 ; +} +static int __devinit pci_init (void ) {struct pci_dev *dev;
@@ -751,6 +762,7 @@EXPORT_SYMBOL (pci_clear_mwi);EXPORT_SYMBOL (pci_set_dma_mask);EXPORT_SYMBOL (pci_dac_set_dma_mask); +EXPORT_SYMBOL (pci_set_consistent_dma_mask);EXPORT_SYMBOL (pci_assign_resource);EXPORT_SYMBOL (pci_find_parent_resource);
drivers/pci/probe.c +1 -0
@@ -502,6 +502,7 @@/* Assume 32-bit PCI; let 64-bit PCI cards (which are far rarer) set this higher, assuming the system even supports it. */ dev->dma_mask =0xffffffff ; + dev->consistent_dma_mask =0xffffffff ;if (pci_setup_device (dev) <0 ) {kfree (dev);return NULL ;
include/linux/pci.h +6 -0
@@ -390,6 +390,11 @@or supports 64-bit transfers. */ struct list_head pools;/* pci_pools tied to this device */ +u64 consistent_dma_mask;/* Like dma_mask, but for +pci_alloc_consistent mappings as +not all hardware supports +64 bit addresses for consistent +allocations such descriptors. */ u32 current_state;/* Current operating state. In ACPI-speak, this is D0-D3, D0 being fully functional, and D3 being off. */
@@ -614,6 +619,7 @@void pci_clear_mwi (struct pci_dev *dev);int pci_set_dma_mask (struct pci_dev *dev,u64 mask);int pci_dac_set_dma_mask (struct pci_dev *dev,u64 mask); +int pci_set_consistent_dma_mask (struct pci_dev *dev,u64 mask);int pci_assign_resource (struct pci_dev *dev,int i);/* Power management related routines */
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin Ngam From: Jes Sorensen <[email protected]>
Date: 16 May 2003 17:02:36 -0400
How does this patch look to you? It adds the generic code for setting the mask, it will then be up to Colin to fix the SGI SN2 specific code to honor it.
I also added a patch to the tg3 driver to set the mask as it seems to work just dandy with 64 bit consistent addresses.
Tested on ia64, but I see no reason why it should fail elsewhere.
I'd hate to say this but we probably need apci_consistent_dma_supported().
But this can wait and your patch is fine, EXCEPT the missing documentation update :-)
Quoted 3 lines · David S. Miller
But this can wait and your patch is fine, EXCEPT the missing documentation update :-)
From my reading of the Tigon driver, test DMA relies on a 32 bit address from dma_alloc_consistent, whereas other consistent allocations seem to be fine with a 64 bit address.
Do you have any problem with a patch to drivers/net/tg3.c to enable the test DMA to be 64 bit?
drivers/net/tg3.c renamed from usr/tmp/TmpDir.15862-0/linux/drivers/net/tg3.c_1.20 +1 -1
@@ -6401,7 +6401,7 @@tw32 (FTQ_RESET ,0 );/* pci_alloc_consistent gives only non-DAC addresses */ - test_desc.addr_hi =0 ; + test_desc.addr_hi = ((u64 ) buf_dma >>32 ); test_desc.addr_lo = buf_dma &0xffffffff ; test_desc.nic_mbuf =0x00002100 ; test_desc.len = size;
drivers/net/tg3.c line 6407
Quoted 12 lines
drivers/net/tg3.c renamed from usr/tmp/TmpDir.15862-0/linux/drivers/net/tg3.c_1.20 +1
-1@@ -6401,7 +6401,7 @@tw32 (FTQ_RESET ,0 );/* pci_alloc_consistent gives only non-DAC addresses */ - test_desc.addr_hi =0 ; + test_desc.addr_hi = ((u64 ) buf_dma >>32 ); test_desc.addr_lo = buf_dma &0xffffffff ; test_desc.nic_mbuf =0x00002100 ; test_desc.len = size;
I guess this is better:
drivers/net/tg3.c renamed from usr/tmp/TmpDir.15906-0/linux/drivers/net/tg3.c_1.20 +1 -2
@@ -6400,8 +6400,7 @@tw32 (BUFMGR_MODE ,0 );tw32 (FTQ_RESET ,0 ); -/* pci_alloc_consistent gives only non-DAC addresses */ - test_desc.addr_hi =0 ; + test_desc.addr_hi = ((u64 ) buf_dma >>32 ); test_desc.addr_lo = buf_dma &0xffffffff ; test_desc.nic_mbuf =0x00002100 ; test_desc.len = size;
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin Ngam From: Jeremy Higdon <[email protected]>
Date: Fri, 16 May 2003 16:40:25 -0700 (PDT)
I guess this is better:
I have no problems with this, Jes can add it to his diff.
Review
Quoted 4 lines · Jeremy Higdon
(quoted diff) +1
-2- /* pci_alloc_consistent gives only non-DAC addresses */ - test_desc.addr_hi = 0; + test_desc.addr_hi = ((u64) buf_dma >> 32); test_desc.addr_lo = buf_dma & 0xffffffff;
Just curious, is the "& 0xffffffff" really necessary? I was expecting "(u32) buf_dma" to be sufficient.
grant
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin Ngam From: Grant Grundler <[email protected]>
Date: Fri, 16 May 2003 21:26:10 -0700
On Fri, May 16, 2003 at 04:40:25PM -0700, Jeremy Higdon wrote: > - /* pci_alloc_consistent gives only non-DAC addresses */ > - test_desc.addr_hi = 0; > + test_desc.addr_hi = ((u64) buf_dma >> 32); > test_desc.addr_lo = buf_dma & 0xffffffff;
Just curious, is the "& 0xffffffff" really necessary? I was expecting "(u32) buf_dma" to be sufficient.
Not necessary, just a stylistic choice.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"David" = David S Miller <[email protected]> writes:
David> From: Jes Sorensen <[email protected]> Date: 16 May
David> 2003 17:02:36 -0400
Quoted 6 lines
How does this patch look to you? It adds the generic code
for setting the mask, it will then be up to Colin to fix the SGI SN2 specific code to honor it.I also added a patch to the tg3 driver to set the mask as it
seems to work just dandy with 64 bit consistent addresses.
David> I'd hate to say this but we probably need a David> pci_consistent_dma_supported().
Hi Dave,
I thought about this one, but from the system's point of view, I don't see a difference between being able to do 64 bit DMA to consistent allocations and 64 bit DMA to dynamic allocations. I'd say this is a host adapter/driver problem for descriptor limitations as you pointed out.
I am not strongly biased on this one, it just seems redundant to me. I'll be happy to add it to the patch if you prefer that.
David> But this can wait and your patch is fine, EXCEPT the missing David> documentation update :-)
I have an updated patch nearly ready, including fancy documentation updates and Jeremy's fix to tg3.c. I will post this to linux-kernel shortly.
Cheers,
Jes
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin NgamQuoted 15 lines · Jes Sorensen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"David" = David S Miller <[email protected]> writes:
David> From: Jes Sorensen <[email protected]> Date: 16 May
David> 2003 17:02:36 -0400Quoted 6 lines
How does this patch look to you? It adds the generic code
for setting the mask, it will then be up to Colin to fix the SGI SN2 specific code to honor it.I also added a patch to the tg3 driver to set the mask as it
seems to work just dandy with 64 bit consistent addresses.David> I'd hate to say this but we probably need a David>
pci_consistent_dma_supported().
Quoted 7 lines
Hi Dave,
I thought about this one, but from the system's point of view, I don't see a difference between being able to do 64 bit DMA to consistent allocations and 64 bit DMA to dynamic allocations. I'd say this is a host adapter/driver problem for descriptor limitations as you pointed out.
If we define consistent DMA capability as a capability that is restricted and controlled by the driver/adapter, yes, the system DMA capability, which is what pci_dma_supported() returns, is all we need. However, if we define consistent DMA capability as a capability that the system/platform can restrict, and it can be different than what pci_dma_supported(), then , yes we will need pci_consistent_dma_supported().
I am rather bias towards treating this as a device driver/adapter restriction. Unless,
it is also expected by this pci_alloc_consistent() interface to ensure that actual
physical memory allocated is between 0-4G .. which on our platform is not possible as we do not have such memory ranges. I do not understand this to be the case. The requirement for a 32bits DMA handle to perform SAC has nothing to to with the actual location of the Physical memory, when this DMA Handle is presented to the IO Chipset for either a DMA read or a DMA write on the PCI Bus.
I am bias but both ways will work, with pci_consistent_dma_supported() providing
an option for platforms(not driver/adapter) that do restrict consistent DMA capability. Is this a valid system/platform restriction? If so, we will need pci_consistent_dma_supported(). Otherwise, no, we do not need this new interface.
Thanks.
colin
Quoted 14 lines
I am not strongly biased on this one, it just seems redundant to me. I'll be happy to add it to the patch if you prefer that.
David> But this can wait and your patch is fine, EXCEPT the missing David> documentation update :-)
I have an updated patch nearly ready, including fancy documentation updates and Jeremy's fix to
tg3.c. I will post this to linux-kernel shortly.Cheers,
Jes
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin NgamQuoted 28 lines · Russell King
Isn't the common use (eg, net devices):
buffer passed from lower levels
map buffer
(device owns buffer)
device uses buffer
unmap buffer
(cpu owns buffer)
free buffer
So, to use your example:
(cpu owns region)
map
(device owns region)
dma_sync...cpu
(cpu owns region)
Quoted 7 lines
dma_sync...device
(device owns region)
unmap
(cpu owns region)
A grep through the tree implies that you (for arm) and me (for parisc) are the only current implementors of the dma_ API.
The only consumers seem to be: lasi_82596, 53c700 and sym53c8xx_comm.h (all of which I did anyway), and usb/core/usb.c
How would you feel about making this API change? I take it you have a preference for using _to_cpu and _to_device call variants?
James
Quoted 1 line · James Bottomley
How would you feel about making this API change?
No problem.
Quoted 2 lines
I take it you have a preference for using
_to_cpuand_to_devicecall variants?
Thinking about it some, these could be confused with the direction of DMA.
Would "dma_sync_for_device" and "dma_sync_for_cpu" fit better?
Signature · 2 lines
Russell King ([email protected]) The developer of ARM Linux http://www.arm.linux.org.uk/personal/aboutme.html
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin Ngam From: Jes Sorensen <[email protected]>
Date: 17 May 2003 12:44:52 -0400
I thought about this one, but from the system's point of view, I don't see a difference between being able to do 64 bit DMA to consistent allocations and 64 bit DMA to dynamic allocations. I'd say this is a host adapter/driver problem for descriptor limitations as you pointed out.
You're probably right, we'll see ;-)
Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent
In reply to Colin NgamQuoted 5 lines · Russell King
Quoted 1 line · James Bottomley
How would you feel about making this API change?
No problem.
Quoted 7 lines
Quoted 2 lines
I take it you have a preference for using
_to_cpuand_to_devicecall variants?Thinking about it some, these could be confused with the direction of DMA.
Would "
dma_sync_for_device" and "dma_sync_for_cpu" fit better?
That's still a bit ambiguous.
I suppose dma_sync_ownership_to_device etc. would be descriptive but too cumbersome...?
James