Skip to contentSkip to conversation

64 Bits DMA Addresses for Alloc Consistent Interfaces.

Linux-ia64 Colin Ngam 11 participants 83 messages May 15, 2003 – May 18, 2003

Subsystems (3)
lore
Share & tools
lore

Colin Ngam

author
#0 lore raw
Quoted 6 lines · James Bottomley
Quoted 5 lines · Colin Ngam

Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.

Quoted 8 lines
Quoted 4 lines

This extention will not break current usages of these interface, and as driver changes, this extension will enable platforms to support all of it's Physical Memory via the above 2 Interfaces without exhausting limited system mapping resources.

The intent of the dma_alloc_.. was to follow the PCI spec (i.e. lower 32 bits). However, it does take the GFP_ flags, and thus could be adjusted to use the region specifier portion of those.

Hi James,

Do you mean to use the flag parameter to advice that we can return a 64Bit DMA Address? What is a region specifier?

Thanks.

colin

Quoted 6 lines

We're currently debating this very issue, since it came up in the context of the 2.6 must fix list.

James

#1 lore raw
Quoted 9 lines · Colin Ngam

Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.

This extention will not break current usages of these interface, and as driver changes, this extension will enable platforms to support all of it's Physical Memory via the above 2 Interfaces without exhausting limited system mapping resources.

The intent of the dma_alloc_.. was to follow the PCI spec (i.e. lower 32 bits). However, it does take the GFP_ flags, and thus could be adjusted to use the region specifier portion of those.

We're currently debating this very issue, since it came up in the context of the 2.6 must fix list.

James

author
#2 lore raw
In reply to Colin Ngam
Quoted 13 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Colin Ngam

Colin> Anyway, to make a long story short, we like to "extend" these Colin> interfaces to return DMA addresses of 64 bits if and when the Colin> driver has the full capabality to support DAC - via it's DMA Colin> mask capability setting.

Colin> This extention will not break current usages of these Colin> interface, and as driver changes, this extension will enable Colin> platforms to support all of it's Physical Memory via the Colin> above 2 Interfaces without exhausting limited system mapping Colin> resources.

Quoted 4 lines

James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (GFP_DMA32 or some such).

Hi David,

I actually like the flag as a parameter because it allows platform specific extensions to pass in platform specific capabilities that can be ignored by platform that does not support these capabilities.

I do not know whether using the same set of flags i.e.GFP_XXX would overload it's meanings to other MM routines but I really like the a flag parameter, that will provide capability for Platform specific hints.

Thanks.

Colin

Quoted 3 lines

        --david
#3 lore raw
In reply to Colin Ngam
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Colin Ngam

Colin> Anyway, to make a long story short, we like to "extend" these Colin> interfaces to return DMA addresses of 64 bits if and when the Colin> driver has the full capabality to support DAC - via it's DMA Colin> mask capability setting.

Colin> This extention will not break current usages of these Colin> interface, and as driver changes, this extension will enable Colin> platforms to support all of it's Physical Memory via the Colin> above 2 Interfaces without exhausting limited system mapping Colin> resources.

James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (GFP_DMA32 or some such).

	--david
#4 lore raw
In reply to Colin Ngam
Quoted 4 lines · David Mosberger

James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (GFP_DMA32 or some such).

This is still being hashed out. It looks like adding a the _mask variants may still be a viable runner for 2.6. If it turns out not to be, the GFP_DMA32 will almost definitely be there.

James

#5 lore raw
In reply to Colin Ngam
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Colin Ngam

Colin> I do not know whether using the same set of flags i.e.GFP_XXX Colin> would overload it's meanings to other MM routines but I Colin> really like the a flag parameter, that will provide Colin> capability for Platform specific hints.

Overloading the GFP flags with platform-specific DMA stuff is certainly not an option.

If you want to argue for yet-another-flags argument, go ahead and do so. Be prepared for having to make a VERY strong argument...

	--david
#6 lore raw
In reply to Colin Ngam
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · James Bottomley
Quoted 4 lines · David Mosberger
Quoted 4 lines

James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (GFP_DMA32 or some such).

James> This is still being hashed out. It looks like adding a the James> _mask variants may still be a viable runner for 2.6.

That would be good.

	--david
author
#7 lore raw
In reply to Colin Ngam
Quoted 10 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · James Bottomley
Quoted 4 lines · David Mosberger
Quoted 4 lines

James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (GFP_DMA32 or some such).

James> This is still being hashed out. It looks like adding a the James> _mask variants may still be a viable runner for 2.6.

David,

Can you provide a little detail on how the _mask vriant is used and passed into the call?

Thanks.

colin

Quoted 5 lines

That would be good.

        --david
author
#8 lore raw
In reply to Colin Ngam
Quoted 7 lines · David Mosberger
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Colin Ngam

Colin> I do not know whether using the same set of flags i.e.GFP_XXX Colin> would overload it's meanings to other MM routines but I Colin> really like the a flag parameter, that will provide Colin> capability for Platform specific hints.

Quoted 5 lines

Overloading the GFP flags with platform-specific DMA stuff is certainly not an option.

If you want to argue for yet-another-flags argument, go ahead and do so. Be prepared for having to make a VERY strong argument...

David,

I was argueing for using the flag arguement and define it's own set of values versus using the GFP_XXX defines. And not an additional flag arguement.

Thanks.

colin

Quoted 3 lines

        --david
#9 lore raw
In reply to Colin Ngam
Quoted 4 lines · David Mosberger

James and I have been discussing this very point for the last week or so. My preferred solution is to add an explicit dma-mask argument to coherent/non-coherent allocators, but James prefers to do this via a new GFP flag (GFP_DMA32 or some such).

For Streaming DMA, I'd rather stick with the current scheme. The main reason is 64-bit addressing support will be hard coded in the each driver as appropriate. ie it's not a parameter than needs to be passed in with each call to dma_map_single().

I'm assuming there is some performance impact to adding another argument to the existing dma_* calls. (more instructions and another register consumed). That might be a bad assumption for ia64.

Since Consistent DMA allocations aren't in the performance path normally, implementation details here don't matter as much (to me).

One of the original goals for my talk at OLS was to propose extensions to 2.4.x to provide "DMA Hints". This would help chipset specific DMA support pick "optimal" settings for DMA operations. I was thinking of redefining "direction" parameter to be "hints". It would continue to support Read or Write "hint" but then use upper bits as flags for other things (don't want to completely spoil my talk). But I can't justify the change based on the performance differences I've measured to date and probably will not offer a proposal. I haven't gotten to more complex testing (only single card tests so far) and HPUX certainly saw performance benefits in a similar scheme on PARISC (using PCI, not PCI-X).

hth,
grant

#10 lore raw
In reply to Colin Ngam
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Grant Grundler

Grant> For Streaming DMA, I'd rather stick with the current scheme.

Nobody said anything about changing streaming DMA. The discussion was purely about coherent/non-coherent allocations.

	--david
#11 lore raw
In reply to Colin Ngam
Quoted 6 lines · Colin Ngam

Hi,

The Altix PCI-X Bridge chipset expects devices that runs in PCI-X mode to always be 64Bit DMA Address capable(DAC). This requirement is directly in conflict with both the pci_alloc_consistent() and dma_alloc_coherent() interfaces.

I don't (yet) see the conflict.
Can't the 32-bit address can be extended to 64-bits as described in the dma_alloc_coherent() interface? Are you running out of 32-bit memory?

Quoted 4 lines

Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.

[email protected] proposed this before. On April 23, 2002, davem wrote:

| %99 of PCI chips out there do not support DAC addressing for things
| like descriptor tables etc.  So it's not a matter of "well" it's
| a matter of "at all".

If you want the complete thread, check ia64-linux 2002 mail archives.

hth,
grant

#12 lore raw
In reply to Colin Ngam
Quoted 3 lines · Grant Grundler

Can't the 32-bit address can be extended to 64-bits as described in the dma_alloc_coherent() interface? Are you running out of 32-bit memory?

I found the answer to my question here:
https://lists.linuxia64.org/archives//linux-ia64/2002-April/003418.html

Quoted 1 line

If you want the complete thread, check ia64-linux 2002 mail archives.

/me happened to take his own advice this time...

grant

ps. "cygnus.com" is a dead domain. IIRC, rth is at redhat now. trying that.

#13 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 08:17:58 -0500

Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.

All devices I am aware of do not support 64-bit addresses for their descriptors, only 32-bit.

There is a huge difference between using a "64-bit address" and requiring to generate a DAC cycle, the latter of which is probably what you need and that is independant of what kind of address pci_alloc_consistent() is giving you.

#14 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 10:05:28 -0500

I actually like the flag as a parameter because it allows platform specific extensions to pass in platform specific capabilities that can be ignored by platform that does not support these capabilities.

I hate this concept. I can't think of any attribute that all platforms can't take advantage of in some way and we'd actually want drivers to use.

Here be the way to private platform-specific interfaces, which is totally against the reason we have these interfaces in the first place.

#15 lore raw
In reply to Colin Ngam

From: Grant Grundler <[email protected]>
Date: Thu, 15 May 2003 12:06:22 -0700

This would help chipset specific DMA support pick "optimal" settings for DMA operations.

I vehemently will fight any such rediculious ideas.

The whole concept behind these portable DMA apis is to avoid platforms specific junk, it all must be generic and apply to everyone.

I was thinking of redefining "direction" parameter to be "hints". It would continue to support Read or Write "hint" but then use upper bits as flags for other things (don't want to completely spoil my talk).

Do not mix "must" parameters such as read/write protections with "it would be nice if" flag bits.

#16 lore raw
In reply to Colin Ngam
Quoted 10 lines · David S. Miller

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 08:17:58 -0500

Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.

All devices I am aware of do not support 64-bit addresses for their descriptors, only 32-bit.

PCI-X devices are required to support 64 bit addrs. They'll only generate SAC if the top 32 bits are 0, but our platform is incapable of handling this, since the IOMMU can't be used in PCI-X mode (i.e. we need high order bits for PCI-X devices).

Quoted 4 lines

There is a huge difference between using a "64-bit address" and requiring to generate a DAC cycle, the latter of which is probably what you need and that is independant of what kind of address pci_alloc_consistent() is giving you.

Unfortunately, we need both for PCI-X on Altix. How about this:
drivers that support PCI-X cards are required to program the full 64 bits of a pci_alloc_consistent DMA address, the top 32 bits of which can be zero for most platforms. That means that the device will only generate a SAC for most platforms, but on platforms like ours, the top 32 bits will contain useful information, so a DAC will be generated.

Thanks,
Jesse

#17 lore raw
In reply to Colin Ngam
Quoted 2 lines · David S. Miller

Do not mix "must" parameters such as read/write protections with "it would be nice if" flag bits.

R/W "hints" is not required for ia64/parisc IOMMUs. AFAIK, only swiotlb requires it and ZX1 IOMMU support has a one performance optimization which makes use of R/W hint.

In short, "must" only applies to some subset of working platforms.

But in general I agree, hints should be useful to most architectures. Having too many platform specific hints becomes hell for driver writers.

grant

#18 lore raw
In reply to Colin Ngam
Quoted 7 lines · Jesse Barnes
Quoted 2 lines

All devices I am aware of do not support 64-bit addresses for their descriptors, only 32-bit.

PCI-X devices are required to support 64 bit addrs. They'll only generate SAC if the top 32 bits are 0, but our platform is incapable of handling this, since the IOMMU can't be used in PCI-X mode (i.e. we need high order bits for PCI-X devices).

I think you're missing Dave's point. While the card is required to support DAC that doesn't mean the device on the card is necessarily capable of being handed a full 64-bit address to DMA into.

From the way he's talking, I'd surmise tg3 is a good example -- I suspect the ring buffers only support 32-bit addresses.

Signature · 3 lines
"It's not Hollywood.  War is real, war is primarily not about defeat or
victory, it is about death.  I've seen thousands and thousands of dead bodies.
Do you think I want to have an academic debate on this subject?" -- Robert Fisk
author
#19 lore raw
In reply to Colin Ngam
Quoted 8 lines · David S. Miller

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 08:17:58 -0500

Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.

Quoted 7 lines

All devices I am aware of do not support 64-bit addresses for their descriptors, only 32-bit.

There is a huge difference between using a "64-bit address" and requiring to generate a DAC cycle, the latter of which is probably what you need and that is independant of what kind of address pci_alloc_consistent() is giving you.

David,

I am quite confuse with what you are saying.

pci_alloc_consistent(), as documented must return a 32bits DMA in the dma mapped address in "dma_addr_t".

Right now, on our platform, the Upper 32 Bits of this address is always 0s and the Lower 32 Bits has the IOMMU mapped DMA address. When this address is given to the card, SAC is generated because the Upper 32 Bits are 0s - if the whole 64bits are given to the card.

Now, for PCI-X devices, we need to generate an IOMMU mapped DMA address that has the Upper 32 bits non 0s. When the driver gives this address to the card, it will generate a DAC for the DMA.

From what I understand from the pci_alloc_consistant() interface, we cannot return a "dma_addr_t" that has the Upper 32bits NON 0s. Is that TRUE? If it is, our platform cannot deal with this restriction .. therefore we are requesting
for this extension.

Thanks.

colin

author
#20 lore raw
In reply to Colin Ngam
Quoted 11 lines · James Bottomley
Quoted 10 lines · Colin Ngam

Right now, on our platform, the Upper 32 Bits of this address is always 0s and the Lower 32 Bits has the IOMMU mapped DMA address. When this address is given to the card, SAC is generated because the Upper 32 Bits are 0s - if the whole 64bits are given to the card.

Now, for PCI-X devices, we need to generate an IOMMU mapped DMA address that has the Upper 32 bits non 0s. When the driver gives this address to the card, it will generate a DAC for the DMA.

Quoted 1 line

From what I understand from the pci_alloc_consistant() interface, we cannot

Quoted 8 lines
Quoted 4 lines

return a "dma_addr_t" that has the Upper 32bits NON 0s. Is that TRUE? If it is, our platform cannot deal with this restriction .. therefore we are requesting
for this extension.

OK, now you've confused me. I thought you were saying that your PCI hardware generates two type of addresses: one for DAC that *doesn't* go through the IOMMU and one for SAC that does?

Yes, you are correct .. we have the capability to generate a SAC or DAC address in PCI Mode only.

Quoted 9 lines

In which case, SAC should be able to go through the IOMMU to reach anwhere in memory, and thus pci_alloc_consistent should work for you now.

Or are you saying it's more efficient to go via the DAC non-iommu cycle and you'd like to do that preferentially if you could (which is a feature request, not a hard requirement)?

However, in PCI-X mode, we do not have the capability to generate SAC addresses. And yes, you are also correct that it is very much efficient on our architecture to generate DAC for all(PCI/PCI-X)

So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.

Hope that is clear ..

Thanks.

colin

Quoted 3 lines

James

#21 lore raw
In reply to Colin Ngam
Quoted 9 lines · Colin Ngam

Right now, on our platform, the Upper 32 Bits of this address is always 0s and the Lower 32 Bits has the IOMMU mapped DMA address. When this address is given to the card, SAC is generated because the Upper 32 Bits are 0s - if the whole 64bits are given to the card.

Now, for PCI-X devices, we need to generate an IOMMU mapped DMA address that has the Upper 32 bits non 0s. When the driver gives this address to the card, it will generate a DAC for the DMA.

Quoted 5 lines
Quoted 1 line

From what I understand from the pci_alloc_consistant() interface, we cannot

return a "dma_addr_t" that has the Upper 32bits NON 0s. Is that TRUE? If it is, our platform cannot deal with this restriction .. therefore we are requesting
for this extension.

OK, now you've confused me. I thought you were saying that your PCI hardware generates two type of addresses: one for DAC that *doesn't* go through the IOMMU and one for SAC that does?

In which case, SAC should be able to go through the IOMMU to reach anwhere in memory, and thus pci_alloc_consistent should work for you now.

Or are you saying it's more efficient to go via the DAC non-iommu cycle and you'd like to do that preferentially if you could (which is a feature request, not a hard requirement)?

James

#22 lore raw
In reply to Colin Ngam
Quoted 7 lines · Colin Ngam

However, in PCI-X mode, we do not have the capability to generate SAC addresses. And yes, you are also correct that it is very much efficient on our architecture to generate DAC for all(PCI/PCI-X)

So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.

Hope that is clear ..

Actually, no. This seems to imply that your PCI-X cards can *never* access the first 4GB of memory (because they can't use an IOMMU and they have to generate a DAC cycle with non zero in bits 63--32).

James

author
#23 lore raw
In reply to Colin Ngam
Quoted 5 lines · James Bottomley
Quoted 4 lines · Colin Ngam

However, in PCI-X mode, we do not have the capability to generate SAC addresses. And yes, you are also correct that it is very much efficient on our architecture to generate DAC for all(PCI/PCI-X)

Quoted 7 lines
Quoted 3 lines

So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.

Hope that is clear ..

Actually, no. This seems to imply that your PCI-X cards can *never* access the first 4GB of memory (because they can't use an IOMMU and they have to generate a DAC cycle with non zero in bits 63--32).

Actually, our system does not have Physical Memory Address smaller than 48Bits - give or take couple bits ... Hence, 32Bits Mapped DMA address maps to 48Bits System Physical Address.

64Bits Mapped DMA address maps directly to System Physical Memory with Bits 63-49 containing transfer attributes like hardware support for byte swapping .. if needed(e.g. Big Endian Cards).

Thanks.

colin

Quoted 3 lines

James

#24 lore raw
In reply to Colin Ngam

From: Jesse Barnes <[email protected]>
Date: Thu, 15 May 2003 13:24:12 -0700

How about this: drivers that support PCI-X cards are required to program the full 64 bits of a pci_alloc_consistent DMA address, the top 32 bits of which can be zero for most platforms.

How about this: you put the device attributes where they belong, in the device struct.

author
#25 lore raw
In reply to Colin Ngam
Quoted 8 lines · David S. Miller

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 17:14:11 -0500

Now, for PCI-X devices, we need to generate an IOMMU mapped DMA address that has the Upper 32 bits non 0s.

Then add a consistent_dma_mask attribute to the device struct, problem solved.

Hi David,

I do not understand what you are saying .. how would a consistent_dma_mask attribute in the device struct helps pci_alloc_consistent() to return a dma hanlde
that can have the Upper 32 Bits Non 0's, and, moreover, these Upper 32 Bits are not constants, but can vary greatly, and is part of the 64 bit Physical System
Memory Address?

Thanks.

colin

#26 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam

From: Matthew Wilcox <[email protected]>
Date: Thu, 15 May 2003 22:29:18 +0100

From the way he's talking,

My main point is that device attributes belong IN THE DEVICE not in some umpteenth new parameter to the DMA interfaces.

Just put consistent_dma_mask into the device struct next to the existing dma_mask, voila problem solved.

I have two major problems with the generic DMA interfaces and how they are being handled in general:

1) Their use is entirely inappropriate in unary-bus devices. There is no way someone can justify some PCI driver continually asking itself whether it is a PCI device or not on each DMA API call.

2) Additions to the parameters in the interfaces is not given enough thought. Each new parameter you add is a discrete failure of the API. It was built to be simple, and new parameters do nothing but add complexity.

I cannot even fathom that someone would suggest to pass device attributes into the DMA interfaces. That is just garbage and unnecessary bloat.

Finally, I see nobody tackling the real hard bugs we have in the DMA API. These should be fixed first before adding random DMA behaviorial attributes. For example, pci_dma_sync_*() sucks because it only syncs in one direction, device to cpu, there is nothing available to sync things back from cpu-->device. This prevents MIPS from using the devices that need to do syncs without unmaps (eepro100 is but one example, USB host controller drivers want this too).

author
#27 lore raw
In reply to Colin Ngam
Quoted 11 lines · David S. Miller

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:25:54 -0500

Actually, our system does not have Physical Memory Address smaller than 48Bits - give or take couple bits ... Hence, 32Bits Mapped DMA address maps to 48Bits System Physical Address.

If you support mapping 32-bit DMA bus spaces to the full 48-bit physical memory spaces, then why do you even need >4GB addressing?

Saying that PCI-X devices can only generate DAC cycles is bogus,

I did not say that. it is very obvious that PCI-X devices can generate either SAC or DAC.

What I did say is that, on our platform, in PCI-X mode, our PCI-X Bridge Chipset do NOT support 32-Bit DMA Bus Space.

This does not break any PCI-X specs.

However, the pci_alloc_consistent() interface is documented to ONLY return a 32-Bit DMA Bus Space .. we like this to be extended so that PCI-X device drivers or PCI drivers can consumed 64-Bit DMA Bus Space, if they have the capability.

Thanks.

colin

Quoted 10 lines

I have PCI-X tg3 cards in my systems and they function just fine. Or does this rule only apply when the device is on a PCI-X bus?

This all sounds seriously broken to me, I thought the issue was that
a) the addresses could not be translated, it was direct phys-->virt and b) you can't guarentee that the physical address of memory allocated on any particular node is <4GB.

#28 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 17:14:11 -0500

Now, for PCI-X devices, we need to generate an IOMMU mapped DMA address that has the Upper 32 bits non 0s.

Then add a consistent_dma_mask attribute to the device struct, problem solved.

#29 lore raw
In reply to Colin Ngam

From: James Bottomley <[email protected]>
Date: 15 May 2003 17:47:04 -0500

OK, now you've confused me.

Their hardware platform cannot guarentee low-32bit memory allocations to devices because each node can have physical memory starting at

Quoted 1 line

4GB etc etc.

#30 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 17:44:57 -0500

However, in PCI-X mode, we do not have the capability to generate SAC addresses.

The device can generate addresses that have the top-32bits 0 though.

You problem is rather that you cannot guarentee to be able to allocate physical memory which is <4GB on each and every node.

That is very different from a "PCI-X issue". It is a "this NUMA platform" issue.

author
#31 lore raw
In reply to Colin Ngam
Quoted 8 lines · David S. Miller

From: James Bottomley <[email protected]>
Date: 15 May 2003 17:47:04 -0500

OK, now you've confused me.

Their hardware platform cannot guarentee low-32bit memory allocations to devices because each node can have physical memory starting at

Quoted 1 line

4GB etc etc.

Hi,

Something else I do not understand .. in the PCI world, what does Physical Memory Address got to do with 32Bit Mapped DMA Bus Address?

Not only do we not guarantee that we cannot generate 32 Bit Memory Allocations .. we plain old do not have Memory Addresses smaller below 32Bits .. well, below 48 Bits for that matter .. I have not seen any of the drivers broken yet - e.g. qlXXX, acenic, tg3 etc.

Thanks.

colin

#32 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:25:54 -0500

Actually, our system does not have Physical Memory Address smaller than 48Bits - give or take couple bits ... Hence, 32Bits Mapped DMA address maps to 48Bits System Physical Address.

If you support mapping 32-bit DMA bus spaces to the full 48-bit physical memory spaces, then why do you even need >4GB addressing?

Saying that PCI-X devices can only generate DAC cycles is bogus, I have PCI-X tg3 cards in my systems and they function just fine. Or does this rule only apply when the device is on a PCI-X bus?

This all sounds seriously broken to me, I thought the issue was that
a) the addresses could not be translated, it was direct phys-->virt and b) you can't guarentee that the physical address of memory allocated on any particular node is <4GB.

author
#33 lore raw
In reply to Colin Ngam
Quoted 9 lines · David S. Miller

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:32:31 -0500

I do not understand what you are saying .. how would a consistent_dma_mask attribute in the device struct helps pci_alloc_consistent() to return a dma hanlde that can have the Upper 32 Bits Non 0's, and, moreover, these Upper 32 Bits are not constants, but can vary greatly, and is part of the 64 bit Physical System Memory Address?

Quoted 6 lines

The mask says that the consistent address "can" have the upper 32-bits non-zero. Then you define a consistent_dma_attrs bitmask that can define things like "requires >4GB address" etc.

Again, all of this is device attribute stuff and not an issue concerning the DMA api calls.

Hi David,

Implementation details aside, you are agreeing that the pci_alloc_consistent() interface can return an address > 4GB and we can document as such?

Thanks.

colin

Quoted 4 lines

But I am ever further confused, how do things like tg3 cards even function TODAY on any PCI-X systems.

#34 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:32:31 -0500

I do not understand what you are saying .. how would a consistent_dma_mask attribute in the device struct helps pci_alloc_consistent() to return a dma hanlde that can have the Upper 32 Bits Non 0's, and, moreover, these Upper 32 Bits are not constants, but can vary greatly, and is part of the 64 bit Physical System Memory Address?

The mask says that the consistent address "can" have the upper 32-bits non-zero. Then you define a consistent_dma_attrs bitmask that can define things like "requires >4GB address" etc.

Again, all of this is device attribute stuff and not an issue concerning the DMA api calls.

But I am ever further confused, how do things like tg3 cards even function TODAY on any PCI-X systems.

author
#35 lore raw
In reply to Colin Ngam
Quoted 10 lines · David S. Miller

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:45:42 -0500

Something else I do not understand .. in the PCI world, what does Physical Memory Address got to do with 32Bit Mapped DMA Bus Address?

Many platforms direct map, that is what I assumed was happening here and causing the problems.

Quoted 7 lines

I have not seen any of the drivers broken yet - e.g. qlXXX, acenic, tg3 etc.

It only works because these devices happen to allow >32-bit DMA addresses for descriptors. You will find eventually that many comodity pieces of PCI-X hardware will opt to only support 32-bit DMA addressing for descriptor tables and whatnot.

Hi David,

It is in the PCI-X Spec that all PCI-X devices in PCI-X Mode have to support DAC.
It did not say that it cannot support SAC or should not support SAC, but it has to support DAC.

Thanks.

colin

#36 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:40:22 -0500

What I did say is that, on our platform, in PCI-X mode, our PCI-X Bridge Chipset do NOT support 32-Bit DMA Bus Space.

Sounds like you can't use a whole swath of devices then. What an entirely unfortunate design decision.

Anyways, now that I understand the problem, merely adding the consistent_dma_mask attribute to the device struct will allow drivers to indicate this capability just fine. The driver goes:

	if (pci_set_consistent_dma_mask(pdev, ~(u64)0))
		goto probe_error;

and then the platform pci_alloc_consistent() can do whatever is appropriate.

#37 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:45:42 -0500

Something else I do not understand .. in the PCI world, what does Physical Memory Address got to do with 32Bit Mapped DMA Bus Address?

Many platforms direct map, that is what I assumed was happening here and causing the problems.

I have not seen any of the drivers broken yet - e.g. qlXXX, acenic, tg3 etc.

It only works because these devices happen to allow >32-bit DMA addresses for descriptors. You will find eventually that many comodity pieces of PCI-X hardware will opt to only support 32-bit DMA addressing for descriptor tables and whatnot.

#38 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:51:35 -0500

Implementation details aside, you are agreeing that the pci_alloc_consistent() interface can return an address > 4GB and we can document as such?

I can return an address that matches the requirements specified by dev->consistent_dma_mask which is set by default to 0xffffffff and must be changed by a device driver if it wants to use a non-default value.

#39 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:57:09 -0500

It is in the PCI-X Spec that all PCI-X devices in PCI-X Mode have to support DAC.

Having to support DAC cycles does not mean that it must support full 64-bit DMA addresses in it's descriptors.

A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.

#40 lore raw
In reply to Colin Ngam
Quoted 9 lines · David S. Miller

From: Jesse Barnes <[email protected]>
Date: Thu, 15 May 2003 13:24:12 -0700

How about this: drivers that support PCI-X cards are required to program the full 64 bits of a pci_alloc_consistent DMA address, the top 32 bits of which can be zero for most platforms.

How about this: you put the device attributes where they belong, in the device struct.

That's fine, it'll work for us. I just wanted to make sure that dma_alloc_consistent could return a 64 bit DMA addr. Having a consistent_dma_mask will allow us to do that for devices that support it. We'll have to switch devices that can't into PCI mode for them to operate correctly on our hw.

Also, are you saying that tg3 doesn't support all 64 bits of the DMA address? If so, I'm not sure how it's working on our platform either, unless we're forcing it into PCI mode; I'll have to check.

Thanks,
Jesse

#41 lore raw
In reply to Colin Ngam
Quoted 5 lines · David S. Miller

Having to support DAC cycles does not mean that it must support full 64-bit DMA addresses in it's descriptors.

A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.

This is certainly a property of the aic79xx driver (only using 32 bit descriptor tables). It has this nice code in the driver for doing it:

aic79xx.c:

	/*
	 * Although we can dma data above 4GB, our
	 * "consistent" memory is below 4GB for
	 * space efficiency reasons (only need a 4byte
	 * address).  For this reason, we have to reset
	 * our dma mask when doing allocations.
	 */
	if (ahd->dev_softc != NULL)
		ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF);
	*vaddr = pci_alloc_consistent(ahd->dev_softc,
				      dmat->maxsize, &map->bus_addr);
	if (ahd->dev_softc != NULL)
		ahd_pci_set_dma_mask(ahd->dev_softc,
				     ahd->platform_data->hw_dma_mask);

And it comes in PCI-X flavours.

James

#42 lore raw
In reply to Colin Ngam

From: Jesse Barnes <[email protected]>
Date: Thu, 15 May 2003 17:29:05 -0700

Also, are you saying that tg3 doesn't support all 64 bits of the DMA address? If so, I'm not sure how it's working on our platform either, unless we're forcing it into PCI mode; I'll have to check.

I believe tg3 does, but some cost-sensitive PCI-X devices might decide not to.

author
#43 lore raw
In reply to Colin Ngam
Quoted 27 lines · James Bottomley
Quoted 5 lines · David S. Miller

Having to support DAC cycles does not mean that it must support full 64-bit DMA addresses in it's descriptors.

A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.

This is certainly a property of the aic79xx driver (only using 32 bit descriptor tables). It has this nice code in the driver for doing it:

aic79xx.c:

        /*
         * Although we can dma data above 4GB, our
         * "consistent" memory is below 4GB for
         * space efficiency reasons (only need a 4byte
         * address).  For this reason, we have to reset
         * our dma mask when doing allocations.
         */
        if (ahd->dev_softc != NULL)
                ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF);
        *vaddr = pci_alloc_consistent(ahd->dev_softc,
                                      dmat->maxsize, &map->bus_addr);
        if (ahd->dev_softc != NULL)
                ahd_pci_set_dma_mask(ahd->dev_softc,
                                     ahd->platform_data->hw_dma_mask);
Quoted 1 line

And it comes in PCI-X flavours.

Yes agreed. And I think this is fine. But let's not have the interface prevents 64bits drivers that wants to do DAC. that's all I am asking ..

Thanks.

colin

Quoted 3 lines

James

#44 lore raw
In reply to Colin Ngam

From: James Bottomley <[email protected]>
Date: 15 May 2003 19:31:20 -0500

   	if (ahd->dev_softc != NULL)
   		ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF);
   	*vaddr = pci_alloc_consistent(ahd->dev_softc,
   				      dmat->maxsize, &map->bus_addr);

Funny how whoever wrote this didn't understand that pci_set_dma_mask() has zero effect on pci_alloc_consistent() behavior.

Someone please tell Justin to fix this broken bit of code.

author
#45 lore raw
In reply to Colin Ngam
Quoted 8 lines · David S. Miller

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 18:57:09 -0500

It is in the PCI-X Spec that all PCI-X devices in PCI-X Mode have to support DAC.

Having to support DAC cycles does not mean that it must support full 64-bit DMA addresses in it's descriptors.

Hi David,

I do not believe it is within spec to perform DAC when the Upper 32bits are 0s. Current usage of pci_alloc_consistent(), even if you want to do DAC, you cannot, because the interface is defined to return only 32bits worth of addresses.

However, I think we are beyond this point now.

Thanks.

colin

Quoted 4 lines

A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.

#46 lore raw
In reply to Colin Ngam
Quoted 23 lines · James Bottomley
Quoted 2 lines

A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.

This is certainly a property of the aic79xx driver (only using 32 bit descriptor tables). It has this nice code in the driver for doing it:

aic79xx.c:

	/*
	 * Although we can dma data above 4GB, our
	 * "consistent" memory is below 4GB for
	 * space efficiency reasons (only need a 4byte
	 * address).  For this reason, we have to reset
	 * our dma mask when doing allocations.
	 */
	if (ahd->dev_softc != NULL)
		ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF);
	*vaddr = pci_alloc_consistent(ahd->dev_softc,
				      dmat->maxsize, &map->bus_addr);
	if (ahd->dev_softc != NULL)
		ahd_pci_set_dma_mask(ahd->dev_softc,
				     ahd->platform_data->hw_dma_mask);
Quoted 1 line

And it comes in PCI-X flavours.

Ah, ok (set_dma_mask vs. alloc_consistent bugs aside). I think there was some confusion about the fact that the PCI-X spec. says that cards must support DAC vs. them actually supporting 64 bit DMA addresses. They all have to do the former, but not the latter. Also, using an IOMMU is different than using direct mapping. Our hw can do both for PCI cards that only handle 32 bit DMA addresses. IOMMU mapping allows you to map pages anywhere in memory, while direct mapping gives you a static 2GB window into system memory.

Hope this clears things up a little.

Thanks,
Jesse

author
#47 lore raw
In reply to Colin Ngam
Quoted 10 lines · David S. Miller

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 19:36:55 -0500

I do not believe it is within spec to perform DAC when the Upper 32bits are 0s.

That is an illogical requirement in that it means PCI-X mode is illegal in the presence of such addresses. It means that PCI-X devices cannot work in x86 systems with <4GB of memory which is obviously not true.

No that is not the case.

If you have an address that only has the Lower 32bits set, your card do a SAC and not a DAC.

SAC is not prohibited in PCI-X mode. But DAC has to be supported by the card.

Someone please correct me if I misread.

Thanks.

colin

#48 lore raw
In reply to Colin Ngam

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 19:36:55 -0500

I do not believe it is within spec to perform DAC when the Upper 32bits are 0s.

That is an illogical requirement in that it means PCI-X mode is illegal in the presence of such addresses. It means that PCI-X devices cannot work in x86 systems with <4GB of memory which is obviously not true.

#49 lore raw
In reply to Colin Ngam
Quoted 2 lines · David S. Miller

But I am ever further confused, how do things like tg3 cards even function TODAY on any PCI-X systems.

They work just fine on most platforms with 32-bit addresses. The issue is with what the SGI platform can support, not what the PCI-X devices can support.

BTW, adding consistent_dma_mask to struct pci_dev and a corresponding pci_set_consistent_dma_mask() is a good idea. I think it would simplify our SBA code a bit.

thanks,
grant

#50 lore raw
In reply to Colin Ngam

From: Grant Grundler <[email protected]>
Date: Thu, 15 May 2003 18:20:51 -0700

BTW, adding consistent_dma_mask to struct pci_dev and a corresponding pci_set_consistent_dma_mask() is a good idea. I think it would simplify our SBA code a bit.

Great, I'm glad this is resolved. :-)

#51 lore raw
In reply to Colin Ngam
Quoted 3 lines · Colin Ngam

I do not understand what you are saying .. how would a consistent_dma_mask attribute in the device struct helps pci_alloc_consistent() to return a dma hanlde that can have the Upper 32 Bits Non 0's...

Colin,
davem has offered a counter proposal that makes way more sense that adding parameters to the dma_*() API. It won't change existing code or existing behaviors. And yes, Documentation/DMA_mapping.txt (API documentation) has to change to reflect that.

Device Drivers SGI cares about will have to be modified to call pci_set_consistent_dma_mask() before calling pci_alloc_consistent().

hth,
grant

author
#52 lore raw
In reply to Colin Ngam
Quoted 11 lines · Grant Grundler
Quoted 3 lines · Colin Ngam

I do not understand what you are saying .. how would a consistent_dma_mask attribute in the device struct helps pci_alloc_consistent() to return a dma hanlde that can have the Upper 32 Bits Non 0's...

Colin,
davem has offered a counter proposal that makes way more sense that adding parameters to the dma_*() API. It won't change existing code or existing behaviors. And yes, Documentation/DMA_mapping.txt (API documentation) has to change to reflect that.

Quoted 5 lines

Device Drivers SGI cares about will have to be modified to call pci_set_consistent_dma_mask() before calling pci_alloc_consistent().

hth,
grant

Hi Grant,

That would be very fine :-)

Thanks.

colin

#53 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam
Quoted 6 lines · David S. Miller

My main point is that device attributes belong IN THE DEVICE not in some umpteenth new parameter to the DMA interfaces.

Just put consistent_dma_mask into the device struct next to the existing dma_mask, voila problem solved.

Quoted 7 lines

I have two major problems with the generic DMA interfaces and how they are being handled in general:

1) Their use is entirely inappropriate in unary-bus devices. There is no way someone can justify some PCI driver continually asking itself whether it is a PCI device or not on each DMA API call.

Where's that in the API? The PA-RISC implementation copes with about four different buses and nowhere does it have a switch trying to ascertain which bus it's talking about (it uses the dev->platform_data field to cache the relevant bus attributes).

Quoted 8 lines

2) Additions to the parameters in the interfaces is not given enough thought. Each new parameter you add is a discrete failure of the API. It was built to be simple, and new parameters do nothing but add complexity.

I cannot even fathom that someone would suggest to pass device attributes into the DMA interfaces. That is just garbage and unnecessary bloat.

You mean like xadding gfp flags to the allocator?

Quoted 7 lines

Finally, I see nobody tackling the real hard bugs we have in the DMA API. These should be fixed first before adding random DMA behaviorial attributes. For example, pci_dma_sync_*() sucks because it only syncs in one direction, device to cpu, there is nothing available to sync things back from cpu-->device. This prevents MIPS from using the devices that need to do syncs without unmaps (eepro100 is but one example, USB host controller drivers want this too).

OK, I don't get this yet, doesn't DMA_TO_DEVICE do what you want? what am I missing?

James

#54 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam

From: James Bottomley <[email protected]>
Date: 15 May 2003 21:05:12 -0500

Quoted 7 lines

Finally, I see nobody tackling the real hard bugs we have in the DMA API. These should be fixed first before adding random DMA behaviorial attributes. For example, pci_dma_sync_*() sucks because it only syncs in one direction, device to cpu, there is nothing available to sync things back from cpu-->device. This prevents MIPS from using the devices that need to do syncs without unmaps (eepro100 is but one example, USB host controller drivers want this too).

OK, I don't get this yet, doesn't DMA_TO_DEVICE do what you want? what am I missing?

You don't understand, a driver does:

	pci_dma_sync_single(... buf);

This transfers ownership of the buffer from device back to the CPU. It guarentees that the CPU will see uptodate contents of the buffer and that it may make modifications to 'buf'.

Then it looks at 'buf', and decides that it wants to give the buffer back to the device. What does it call to transfer ownership back from CPU to device?

Answer: No such API call exists.

This is a big problem for implementations that need to flush CPU caches in order to implement the cpu<-->device buffer ownership transfer properly.

#55 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam
Quoted 12 lines

You don't understand, a driver does:

	pci_dma_sync_single(... buf);

This transfers ownership of the buffer from device back to the CPU. It guarentees that the CPU will see uptodate contents of the buffer and that it may make modifications to 'buf'.

Then it looks at 'buf', and decides that it wants to give the buffer back to the device. What does it call to transfer ownership back from CPU to device?

Quoted 5 lines

Answer: No such API call exists.

This is a big problem for implementations that need to flush CPU caches in order to implement the cpu<-->device buffer ownership transfer properly.

It was my impression that a CPU cache invalidate cycle relinquishes the cache line (and thus leaves the line free to be owned by the device). The DMA_TO_DEVICE of dma_sync.. does this, why isn't that sufficient? I grant that CPU does auto ownership, so if the CPU ever touches the data again the cache line is pulled back in again.

Is the issue that we have no way of preventing the CPU from pulling in the cache line if it is referenced?

James

#56 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam

From: James Bottomley <[email protected]>
Date: 15 May 2003 21:26:14 -0500

It was my impression that a CPU cache invalidate cycle relinquishes the cache line (and thus leaves the line free to be owned by the device). The DMA_TO_DEVICE of dma_sync.. does this, why isn't that sufficient? I grant that CPU does auto ownership, so if the CPU ever touches the data again the cache line is pulled back in again.

What flushes the cpu cache after the cpu writes to 'buf' so that the device sees what the CPU wrote there? Consider copy-back L2 caches like MIPS has.

Is the issue that we have no way of preventing the CPU from pulling in the cache line if it is referenced?

The "issue" is that we need a pci_dma_sync_*_to_device() call added. We should also remane pci_dma_sync_*() to pci_dma_sync_*_to_cpu() to make all of the naming consistent and update all the drivers.

#57 lore raw
In reply to Colin Ngam
Quoted 10 lines · David S. Miller

From: Colin Ngam <[email protected]>
Date: Thu, 15 May 2003 08:17:58 -0500

Anyway, to make a long story short, we like to "extend" these interfaces to return DMA addresses of 64 bits if and when the driver has the full capabality to support DAC - via it's DMA mask capability setting.

All devices I am aware of do not support 64-bit addresses for their descriptors, only 32-bit.

The Qlogic and LSI Logic SCSI and FC devices support 64 bit addresses for their descriptors. Also the old Adaptec (now JNI) 1160. I do not believe this is unusual. Certainly all PCI-X devices should support 64 bit addresses for descriptors.

The Tigon3 supports 64 bit addresses for descriptors.

Quoted 4 lines

There is a huge difference between using a "64-bit address" and requiring to generate a DAC cycle, the latter of which is probably what you need and that is independant of what kind of address pci_alloc_consistent() is giving you.

Not independent. Because if pci_alloc_consistent gives an address in which any of bits 32-63 are set, then the adapter will use a DAC to access the descriptor. I guess you're saying that some adapters may support more than 32 bits but fewer than 64. Such devices would not work on our platform. Which PCI-X devices do not support the full 64 bit address space? I don't know of any.

jeremy

#59 lore raw
In reply to Colin Ngam
Quoted 6 lines · David S. Miller

From: Jeremy Higdon <[email protected]>
Date: Thu, 15 May 2003 23:47:55 -0700

Please read further into your mailbox before replying, all of the misunderstandings and issues have been resolved already.

I did. I didn't see that issue addressed. I guess someone did educate you, perhaps in a message I did not see.

#60 lore raw
In reply to Colin Ngam
Quoted 7 lines · James Bottomley
Quoted 6 lines · Colin Ngam

However, in PCI-X mode, we do not have the capability to generate SAC addresses. And yes, you are also correct that it is very much efficient on our architecture to generate DAC for all(PCI/PCI-X)

So, it is a hard requirement to have DAC for PCI-X devices and not PCI devices.

Quoted 7 lines
Quoted 1 line

Hope that is clear ..

Actually, no. This seems to imply that your PCI-X cards can *never* access the first 4GB of memory (because they can't use an IOMMU and they have to generate a DAC cycle with non zero in bits 63--32).

James

I don't know if it was clear or not in the ensuing replies, but we don't present direct physical addresses to PCI-X cards. Instead it's an physical address with various bits ORed in at the top to set routing and attribute information. So if you want to DMA to physical address 0x1000 in the system, the PCI address might be 0x9100000000001000.

If it was already clear, sorry for the duplication.

jeremy

#61 lore raw
In reply to Colin Ngam
Quoted 9 lines · David S. Miller

From: James Bottomley <[email protected]>
Date: 15 May 2003 21:26:14 -0500

It was my impression that a CPU cache invalidate cycle relinquishes the cache line (and thus leaves the line free to be owned by the device). The DMA_TO_DEVICE of dma_sync.. does this, why isn't that sufficient? I grant that CPU does auto ownership, so if the CPU ever touches the data again the cache line is pulled back in again.

Quoted 3 lines

What flushes the cpu cache after the cpu writes to 'buf' so that the device sees what the CPU wrote there? Consider copy-back L2 caches like MIPS has.

Isn't that what dma_cache_wback_inv does? It flushes cpu cache to memory so that it is visible to devices. Is there some additional semantic desired?

jeremy

#62 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam

From: Jeremy Higdon <[email protected]>
Date: Fri, 16 May 2003 00:48:12 -0700

Isn't that what dma_cache_wback_inv does?

That function is not part of the DMA api, and no portable driver should use this dma_cache_wback_inv() thing.

#63 lore raw
In reply to Colin Ngam
Quoted 9 lines · James Bottomley
Quoted 5 lines · David S. Miller

Having to support DAC cycles does not mean that it must support full 64-bit DMA addresses in it's descriptors.

A cost sensitive PCI-X device may wish to use 32-bit addressing in it's descriptors in order to keep pin counts down etc.

This is certainly a property of the aic79xx driver (only using 32 bit descriptor tables). It has this nice code in the driver for doing it:

This is a driver choice, rather than an adapter limitation, right? Or is the adapter limited to 32 bit addresses for map->bus_addr (the PCI-X flavor)?

Quoted 21 lines

aic79xx.c:

	/*
	 * Although we can dma data above 4GB, our
	 * "consistent" memory is below 4GB for
	 * space efficiency reasons (only need a 4byte
	 * address).  For this reason, we have to reset
	 * our dma mask when doing allocations.
	 */
	if (ahd->dev_softc != NULL)
		ahd_pci_set_dma_mask(ahd->dev_softc, 0xFFFFFFFF);
	*vaddr = pci_alloc_consistent(ahd->dev_softc,
				      dmat->maxsize, &map->bus_addr);
	if (ahd->dev_softc != NULL)
		ahd_pci_set_dma_mask(ahd->dev_softc,
				     ahd->platform_data->hw_dma_mask);

And it comes in PCI-X flavours.

James

#64 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam
Quoted 3 lines · David S. Miller

The "issue" is that we need a pci_dma_sync_*_to_device() call added. We should also remane pci_dma_sync_*() to pci_dma_sync_*_to_cpu() to make all of the naming consistent and update all the drivers.

OK, I get it now (I was thinking really in terms of the old writeback/invalidate interface where you had to know how to place them).

You mean we need an interface that is called when the CPU hands off the region to the device and again when the CPU reclaims it (so we always get the writeback or invalidate in the correct place).

How about just adding an ownership attribute to dma_sync..? It would either be DMA_OWNER_CPU or DMA_OWNER_DEVICE?

Of course, retrofitting something like this is going to be fun.

James

#65 lore raw
In reply to Colin Ngam
Quoted 3 lines · James Bottomley

You mean we need an interface that is called when the CPU hands off the region to the device and again when the CPU reclaims it (so we always get the writeback or invalidate in the correct place).

yes

Quoted 2 lines

How about just adding an ownership attribute to dma_sync..? It would either be DMA_OWNER_CPU or DMA_OWNER_DEVICE?

I prefer davem's suggestion of a different function call (*_to_cpu and *_to_dev) rather than making the same function do completely different things based on parameters. Keeps API documentation and resulting implementations for each smaller.

Quoted 1 line

Of course, retrofitting something like this is going to be fun.

It's not so bad.
It's really only needed for non-coherent machines. Folks who care about those platforms probably already know where those calls belong.

grant

#66 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam
Quoted 7 lines · Grant Grundler
Quoted 3 lines · James Bottomley

You mean we need an interface that is called when the CPU hands off the region to the device and again when the CPU reclaims it (so we always get the writeback or invalidate in the correct place).

yes

Quoted 7 lines
Quoted 2 lines

How about just adding an ownership attribute to dma_sync..? It would either be DMA_OWNER_CPU or DMA_OWNER_DEVICE?

I prefer davem's suggestion of a different function call (*_to_cpu and *_to_dev) rather than making the same function do completely different things based on parameters. Keeps API documentation and resulting implementations for each smaller.

Hmm, another API vs adding properties. I'm easy on this. However, the fact that data direction is an attribute does give me a preference for having owner as an attribute too.

Quoted 6 lines
Quoted 1 line

Of course, retrofitting something like this is going to be fun.

It's not so bad.
It's really only needed for non-coherent machines. Folks who care about those platforms probably already know where those calls belong.

Well, it applies to partially coherent machines too. However, the point of the API is to make it so that the driver writer doesn't need to understand where the calls go, So the API encapsulates all the platform quirks in a simple set of rules of thumb.

Thus:

map region

Use region in driver

dma_sync ... DMA_OWNER_DEVICE .. and direction

device uses region (driver may not touch it)

dma_sync ... DMA_OWNER_CPU .. and direction

driver accesses region again (may repeat back to use region in driver)

unmap region

That's unambiguous and simple to articulate. The platform implementation does the necessary magic.

James

#67 lore raw
In reply to Colin Ngam
Quoted 13 lines · James Bottomley

Well, it applies to partially coherent machines too. However, the point of the API is to make it so that the driver writer doesn't need to understand where the calls go, So the API encapsulates all the platform quirks in a simple set of rules of thumb.

Thus:

map region

Use region in driver

dma_sync ... DMA_OWNER_DEVICE .. and direction

Quoted 8 lines

device uses region (driver may not touch it)

dma_sync ... DMA_OWNER_CPU .. and direction

driver accesses region again (may repeat back to use region in driver)

unmap region

Isn't the common use (eg, net devices):

buffer passed from lower levels

map buffer

(device owns buffer)

device uses buffer

unmap buffer

(cpu owns buffer)

free buffer

So, to use your example:

(cpu owns region)

map

(device owns region)

dma_sync...cpu

(cpu owns region)

dma_sync...device

(device owns region)

unmap

(cpu owns region)

Your suggestion seems to add in two extra sync calls (and thus extra code in the common case.)

Signature · 2 lines
Russell King ([email protected])                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html
#68 lore raw
In reply to Colin Ngam
Quoted 2 lines · Jeremy Higdon

Certainly all PCI-X devices should support
64 bit addresses for descriptors.

I finally found the bit in the PCI-X spec hat mentions this:
(Section 1.8 "Bus Width")

| The following PCI-X requirements are different from conventional PCI:
| 1. All devices that initiate memory transactions must be capable of
|    generating 64-bit addresses.
| 2. Each device includes a configuration status bit that indicates
|    the width of that device's interface.

PCI-X also has a different command set.
It's definitely NOT just a PCI bus speed up.

thanks,
grant

#69 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam
Quoted 2 lines · Russell King

Your suggestion seems to add in two extra sync calls (and thus extra code in the common case.)

Yes, sorry, needlessly complex example.

James

#70 lore raw
In reply to Colin Ngam
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"David" = David S Miller <[email protected]> writes:

David> Anyways, now that I understand the problem, merely adding the David> consistent_dma_mask attribute to the device struct will allow David> drivers to indicate this capability just fine. The driver David> goes:

David> if (pci_set_consistent_dma_mask(pdev, ~(u64)0)) goto
David> probe_error;

David> and then the platform pci_alloc_consistent() can do whatever is David> appropriate.

Hi Dave,

How does this patch look to you? It adds the generic code for setting the mask, it will then be up to Colin to fix the SGI SN2 specific code to honor it.

I also added a patch to the tg3 driver to set the mask as it seems to work just dandy with 64 bit consistent addresses.

Tested on ia64, but I see no reason why it should fail elsewhere.

Cheers,
Jes

drivers/net/tg3.c +6 -0
diff -urN -X /home/jes/exclude-linux linux-2.5.69-030509-vanilla/drivers/net/tg3.c linux-2.5.69-pci/drivers/net/tg3.c
@@ -6743,6 +6743,12 @@
 	/* Configure DMA attributes. */
 	if (!pci_set_dma_mask(pdev, (u64) 0xffffffffffffffff)) {
 		pci_using_dac = 1;
+		if (pci_set_consistent_dma_mask(pdev,
+						(u64) 0xffffffffffffffff)) {
+			printk(KERN_ERR PFX "Unable to obtain 64 bit DMA "
+			       "for consistent allocations\n");
+			goto err_out_free_res;
+		}
 	} else {
 		err = pci_set_dma_mask(pdev, (u64) 0xffffffff);
 		if (err) {
drivers/pci/pci.c +12 -0
diff -urN -X /home/jes/exclude-linux linux-2.5.69-030509-vanilla/drivers/pci/pci.c linux-2.5.69-pci/drivers/pci/pci.c
@@ -701,6 +701,17 @@
 	return 0;
 }
 
+int
+pci_set_consistent_dma_mask(struct pci_dev *dev, u64 mask)
+{
+	if (!pci_dma_supported(dev, mask))
+		return -EIO;
+
+	dev->consistent_dma_mask = mask;
+
+	return 0;
+}
+
 static int __devinit pci_init(void)
 {
 	struct pci_dev *dev;
@@ -751,6 +762,7 @@
 EXPORT_SYMBOL(pci_clear_mwi);
 EXPORT_SYMBOL(pci_set_dma_mask);
 EXPORT_SYMBOL(pci_dac_set_dma_mask);
+EXPORT_SYMBOL(pci_set_consistent_dma_mask);
 EXPORT_SYMBOL(pci_assign_resource);
 EXPORT_SYMBOL(pci_find_parent_resource);
 
drivers/pci/probe.c +1 -0
diff -urN -X /home/jes/exclude-linux linux-2.5.69-030509-vanilla/drivers/pci/probe.c linux-2.5.69-pci/drivers/pci/probe.c
@@ -502,6 +502,7 @@
 	/* Assume 32-bit PCI; let 64-bit PCI cards (which are far rarer)
 	   set this higher, assuming the system even supports it.  */
 	dev->dma_mask = 0xffffffff;
+	dev->consistent_dma_mask = 0xffffffff;
 	if (pci_setup_device(dev) < 0) {
 		kfree(dev);
 		return NULL;
include/linux/pci.h +6 -0
diff -urN -X /home/jes/exclude-linux linux-2.5.69-030509-vanilla/include/linux/pci.h linux-2.5.69-pci/include/linux/pci.h
@@ -390,6 +390,11 @@
 					   or supports 64-bit transfers.  */
 	struct list_head pools;		/* pci_pools tied to this device */
 
+	u64		consistent_dma_mask;/* Like dma_mask, but for
+					       pci_alloc_consistent mappings as
+					       not all hardware supports
+					       64 bit addresses for consistent
+					       allocations such descriptors. */
 	u32             current_state;  /* Current operating state. In ACPI-speak,
 					   this is D0-D3, D0 being fully functional,
 					   and D3 being off. */
@@ -614,6 +619,7 @@
 void pci_clear_mwi(struct pci_dev *dev);
 int pci_set_dma_mask(struct pci_dev *dev, u64 mask);
 int pci_dac_set_dma_mask(struct pci_dev *dev, u64 mask);
+int pci_set_consistent_dma_mask(struct pci_dev *dev, u64 mask);
 int pci_assign_resource(struct pci_dev *dev, int i);
 
 /* Power management related routines */
#71 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam

From: Jes Sorensen <[email protected]>
Date: 16 May 2003 17:02:36 -0400

How does this patch look to you? It adds the generic code for setting the mask, it will then be up to Colin to fix the SGI SN2 specific code to honor it.

I also added a patch to the tg3 driver to set the mask as it seems to work just dandy with 64 bit consistent addresses.

Tested on ia64, but I see no reason why it should fail elsewhere.

I'd hate to say this but we probably need a
pci_consistent_dma_supported().

But this can wait and your patch is fine, EXCEPT the missing documentation update :-)

#72 lore raw
In reply to Colin Ngam
Quoted 3 lines · David S. Miller

But this can wait and your patch is fine, EXCEPT the missing documentation update :-)

From my reading of the Tigon driver, test DMA relies on a 32 bit address from dma_alloc_consistent, whereas other consistent allocations seem to be fine with a 64 bit address.

Do you have any problem with a patch to drivers/net/tg3.c to enable the test DMA to be 64 bit?

drivers/net/tg3.c renamed from usr/tmp/TmpDir.15862-0/linux/drivers/net/tg3.c_1.20 +1 -1
@@ -6401,7 +6401,7 @@
 	tw32(FTQ_RESET, 0);
 
 	/* pci_alloc_consistent gives only non-DAC addresses */
-	test_desc.addr_hi = 0;
+	test_desc.addr_hi = ((u64) buf_dma >> 32);
 	test_desc.addr_lo = buf_dma & 0xffffffff;
 	test_desc.nic_mbuf = 0x00002100;
 	test_desc.len = size;
#73 lore raw
In reply to Colin Ngam

drivers/net/tg3.c line 6407

Quoted 12 lines
drivers/net/tg3.c renamed from usr/tmp/TmpDir.15862-0/linux/drivers/net/tg3.c_1.20 +1 -1
@@ -6401,7 +6401,7 @@
 	tw32(FTQ_RESET, 0);
 
 	/* pci_alloc_consistent gives only non-DAC addresses */
-	test_desc.addr_hi = 0;
+	test_desc.addr_hi = ((u64) buf_dma >> 32);
 	test_desc.addr_lo = buf_dma & 0xffffffff;
 	test_desc.nic_mbuf = 0x00002100;
 	test_desc.len = size;

I guess this is better:

drivers/net/tg3.c renamed from usr/tmp/TmpDir.15906-0/linux/drivers/net/tg3.c_1.20 +1 -2
@@ -6400,8 +6400,7 @@
 	tw32(BUFMGR_MODE, 0);
 	tw32(FTQ_RESET, 0);
 
-	/* pci_alloc_consistent gives only non-DAC addresses */
-	test_desc.addr_hi = 0;
+	test_desc.addr_hi = ((u64) buf_dma >> 32);
 	test_desc.addr_lo = buf_dma & 0xffffffff;
 	test_desc.nic_mbuf = 0x00002100;
 	test_desc.len = size;
#75 lore raw
In reply to Colin Ngam

Review

Quoted 4 lines · Jeremy Higdon
(quoted diff) +1 -2
-	/* pci_alloc_consistent gives only non-DAC addresses */
-	test_desc.addr_hi = 0;
+	test_desc.addr_hi = ((u64) buf_dma >> 32);
 	test_desc.addr_lo = buf_dma & 0xffffffff;

Just curious, is the "& 0xffffffff" really necessary? I was expecting "(u32) buf_dma" to be sufficient.

grant

#76 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam

From: Grant Grundler <[email protected]>
Date: Fri, 16 May 2003 21:26:10 -0700

   On Fri, May 16, 2003 at 04:40:25PM -0700, Jeremy Higdon wrote:
   > -	/* pci_alloc_consistent gives only non-DAC addresses */
   > -	test_desc.addr_hi = 0;
   > +	test_desc.addr_hi = ((u64) buf_dma >> 32);
   >  	test_desc.addr_lo = buf_dma & 0xffffffff;

Just curious, is the "& 0xffffffff" really necessary? I was expecting "(u32) buf_dma" to be sufficient.

Not necessary, just a stylistic choice.

#77 lore raw
In reply to Colin Ngam
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"David" = David S Miller <[email protected]> writes:

David> From: Jes Sorensen <[email protected]> Date: 16 May
David> 2003 17:02:36 -0400

Quoted 6 lines

How does this patch look to you? It adds the generic code
for setting the mask, it will then be up to Colin to fix the SGI SN2 specific code to honor it.

I also added a patch to the tg3 driver to set the mask as it
seems to work just dandy with 64 bit consistent addresses.

David> I'd hate to say this but we probably need a David> pci_consistent_dma_supported().

Hi Dave,

I thought about this one, but from the system's point of view, I don't see a difference between being able to do 64 bit DMA to consistent allocations and 64 bit DMA to dynamic allocations. I'd say this is a host adapter/driver problem for descriptor limitations as you pointed out.

I am not strongly biased on this one, it just seems redundant to me. I'll be happy to add it to the patch if you prefer that.

David> But this can wait and your patch is fine, EXCEPT the missing David> documentation update :-)

I have an updated patch nearly ready, including fancy documentation updates and Jeremy's fix to tg3.c. I will post this to linux-kernel shortly.

Cheers,
Jes

author
#78 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam
Quoted 15 lines · Jes Sorensen
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"David" = David S Miller <[email protected]> writes:

David> From: Jes Sorensen <[email protected]> Date: 16 May
David> 2003 17:02:36 -0400

Quoted 6 lines

How does this patch look to you? It adds the generic code
for setting the mask, it will then be up to Colin to fix the SGI SN2 specific code to honor it.

I also added a patch to the tg3 driver to set the mask as it
seems to work just dandy with 64 bit consistent addresses.

David> I'd hate to say this but we probably need a David> pci_consistent_dma_supported().

Quoted 7 lines

Hi Dave,

I thought about this one, but from the system's point of view, I don't see a difference between being able to do 64 bit DMA to consistent allocations and 64 bit DMA to dynamic allocations. I'd say this is a host adapter/driver problem for descriptor limitations as you pointed out.

If we define consistent DMA capability as a capability that is restricted and controlled by the driver/adapter, yes, the system DMA capability, which is what pci_dma_supported() returns, is all we need. However, if we define consistent DMA capability as a capability that the system/platform can restrict, and it can be different than what pci_dma_supported(), then , yes we will need pci_consistent_dma_supported().

I am rather bias towards treating this as a device driver/adapter restriction. Unless,
it is also expected by this pci_alloc_consistent() interface to ensure that actual
physical memory allocated is between 0-4G .. which on our platform is not possible as we do not have such memory ranges. I do not understand this to be the case. The requirement for a 32bits DMA handle to perform SAC has nothing to to with the actual location of the Physical memory, when this DMA Handle is presented to the IO Chipset for either a DMA read or a DMA write on the PCI Bus.

I am bias but both ways will work, with pci_consistent_dma_supported() providing

an option for platforms(not driver/adapter) that do restrict consistent DMA capability. Is this a valid system/platform restriction? If so, we will need pci_consistent_dma_supported(). Otherwise, no, we do not need this new interface.

Thanks.

colin

Quoted 14 lines

I am not strongly biased on this one, it just seems redundant to me. I'll be happy to add it to the patch if you prefer that.

David> But this can wait and your patch is fine, EXCEPT the missing David> documentation update :-)

I have an updated patch nearly ready, including fancy documentation updates and Jeremy's fix to tg3.c. I will post this to linux-kernel shortly.

Cheers,
Jes

#79 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam
Quoted 28 lines · Russell King

Isn't the common use (eg, net devices):

buffer passed from lower levels

map buffer

(device owns buffer)

device uses buffer

unmap buffer

(cpu owns buffer)

free buffer

So, to use your example:

(cpu owns region)

map

(device owns region)

dma_sync...cpu

(cpu owns region)

Quoted 7 lines

dma_sync...device

(device owns region)

unmap

(cpu owns region)

A grep through the tree implies that you (for arm) and me (for parisc) are the only current implementors of the dma_ API.

The only consumers seem to be: lasi_82596, 53c700 and sym53c8xx_comm.h (all of which I did anyway), and usb/core/usb.c

How would you feel about making this API change? I take it you have a preference for using _to_cpu and _to_device call variants?

James

#80 lore raw
In reply to Colin Ngam
Quoted 1 line · James Bottomley

How would you feel about making this API change?

No problem.

Quoted 2 lines

I take it you have a preference for using _to_cpu and _to_device call variants?

Thinking about it some, these could be confused with the direction of DMA.

Would "dma_sync_for_device" and "dma_sync_for_cpu" fit better?

Signature · 2 lines
Russell King ([email protected])                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html
#81 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam

From: Jes Sorensen <[email protected]>
Date: 17 May 2003 12:44:52 -0400

I thought about this one, but from the system's point of view, I don't see a difference between being able to do 64 bit DMA to consistent allocations and 64 bit DMA to dynamic allocations. I'd say this is a host adapter/driver problem for descriptor limitations as you pointed out.

You're probably right, we'll see ;-)

#82 lore raw

Re: [Linux-ia64] Re: 64 Bits DMA Addresses for Alloc Consistent

In reply to Colin Ngam
Quoted 5 lines · Russell King
Quoted 1 line · James Bottomley

How would you feel about making this API change?

No problem.

Quoted 7 lines
Quoted 2 lines

I take it you have a preference for using _to_cpu and _to_device call variants?

Thinking about it some, these could be confused with the direction of DMA.

Would "dma_sync_for_device" and "dma_sync_for_cpu" fit better?

That's still a bit ambiguous.

I suppose dma_sync_ownership_to_device etc. would be descriptive but too cumbersome...?

James