PCI DAC routines for SN

Jesse> Here's an implementation of the PCI DAC API for SN, along Jesse> with some fixes to the existing implementation. If this Jesse> looks ok, I'd like it to get integrated into the next Jesse> release, otherwise I'll fix it up. Please take a look.

You do realize that there it is never necessary to use this API for ia64 linux? dma_addr_t is already 64 bits.

	--david

Except that pci_alloc_consistent is supposed to return a 32 bit address...

Jesse

Jesse> Except that pci_alloc_consistent is supposed to return a 32 Jesse> bit address...

Oh man, what a mess! Have you checked with Dave Miller? I suspect he might not like this. I'm not terribly fond of it either as it makes it even more confusing as to when to use the DAC interface. I'd rather see a separate "alloc_consistent" routine, which doesn't isn't on <4GB addresses.

	--david

He was the one that implemented it I thought. His assertion was that since most chipsets don't handle 64 bit coherent allocations well, the consistent interface should be forced to return a 32 bit address (is that right Dave?). I don't mind that as long as we add a DAC consistent call, otherwise I'd like to leave it up to the platform (i.e. ia64/sn could return a 64 bit address and sparc64 could do 32).

I agree, Cc'ing Dave Miller.

Thanks,
Jesse

And why is that a problem?
Doesn't (or couldn't) swiotlb allocate memory from a "DMA ZONE"? (I've only worked on real IOMMU support for parisc-linux)

or do you have a driver/system config that might exhaust available 32-bit address space?

BTW, I agree the DAC support and SAC for pci_alloc_consistent() is, uhm, not very clear.

thanks,
grant

Right - HP recently changed ZX1 support based on the same assertion. And for the drivers I'm aware of (eg tg3, sym53c8xx), Dave is right.

...

This is not feasible. The driver/device will use/support the same number of address bits regardless of architecture. If the architecture provides fewer bits than the driver/device supports, no problem obviously.

I'm curious to see what happens with the pci_dac_alloc_consistent() idea.

grant

From: Jesse Barnes <[email protected]>
Date: Mon, 22 Apr 2002 16:40:14 -0700

He was the one that implemented it I thought. His assertion was that since most chipsets don't handle 64 bit coherent allocations well, the consistent interface should be forced to return a 32 bit address (is that right Dave?). I don't mind that as long as we add a DAC consistent call, otherwise I'd like to leave it up to the platform (i.e. ia64/sn could return a 64 bit address and sparc64 could do 32).

%99 of PCI chips out there do not support DAC addressing for things like descriptor tables etc. So it's not a matter of "well" it's a matter of "at all".

Therefore pci_alloc_consistent MUST provide SAC only addressing.

I was seeing patches where people would set the DMA mask for the pci_dev around pci_alloc_consistent calls in order to accomplish getting SAC addresses. That is exactly the kind of crap I was trying to avoid.

Therefore, as per the API specification (Documentation/DMA-mapping.txt) and reality, it's unacceptable for pci_alloc_consistent() to return anything other than SAC addresses (or something more constrained, if the DMA mask indicates this, for example for devices with ISA addressing limitations).

I think it is unreasonable to add a special DAC alloc consistent call.

Is this needed because you bozos don't have any physical memory below 4GB on some weird ia64 system ___AND___ you lack a PCI IOMMU in the controllers again? This is getting rediculious if so, and I really want to avoid crapping up the PCI DMA interfaces just because the ia64 PCI hardware folks keep making stupid design decisions.

Right. But some platforms _can_ do DAC addressing for regions that need to be coherent. Seems like we shouldn't penalize good hardware just because there's a lot of bad hardware out there.

Why would they need to do that? If the driver sets its dma_mask to 64 bits, why can't the platform choose to return a 64 bit DMA address? Obviously, if their device only supports 32 bits, then they'd set their dma_mask accordingly and only get SAC addresses back from alloc_consistent. I'm obviously not understaning something...

Why? It seems like a simple addition, even if not all platforms can handle it. Our platform won't work optimally if we don't either change pci_alloc_consistent, or provide some other call to get 64 bit coherent regions.

Not at all. SGI Origin hardware has PCI bridges that can coherently access 64 bit DMA regions. It can also map a limited number of 32 bit addresses into arbitrary 4 GB memory windows of system memory. The number of mappings is somewhat limited however, since most devices are expected to use 64 bit addresses directly.

Thanks,
Jesse

From: Jesse Barnes <[email protected]>
Date: Tue, 23 Apr 2002 22:49:48 -0700

Why would they need to do that? If the driver sets its dma_mask to 64 bits, why can't the platform choose to return a 64 bit DMA address? Obviously, if their device only supports 32 bits, then they'd set their dma_mask accordingly and only get SAC addresses back from alloc_consistent. I'm obviously not understaning something...

NO! They were doing this to workaround the IA64 broken behavior.

They wanted "dma_mask = DAC, but give me SAC for consistent allocations" (ie. what the code should do automatically for them as per Documentation/DMA-mapping.txt) and they were messing with the dma_mask around the pci_alloc_consistent() calls because of IA64 not following the API.

Not at all. SGI Origin hardware has PCI bridges that can coherently access 64 bit DMA regions. It can also map a limited number of 32 bit addresses into arbitrary 4 GB memory windows of system memory. The number of mappings is somewhat limited however, since most devices are expected to use 64 bit addresses directly.

Bad expectation, most devices cannot do this. Like I suspected, bad hardware design decisions are behind this.

I keep my position that there is no legitimate need to add this interface besides perhaps broken hardware.

4GB of SAC addressible RAM is enough to satisfy even the most convoluted use of consisten allocations.

Furthermore, and the important part you are ignoring, using DAC for consistent allocations will incur a performance penalty because every address cycle emitted by the device for a transaction will require more cycles than if SAC addresses were only used for consistent memory.

You have totally failed to consider this issue. All you can think about is "my system can do it, why can't I have the interface" without once considering if that interface makes any sense to use at all even if you can do it.

Can you be more specific about which HW you are talking about? (ie which PCI device can do 64-bit DAC to coherent regions)

thanks,
grant

From: Grant Grundler <[email protected]>
Date: Wed, 24 Apr 2002 09:13:15 -0700

Can you be more specific about which HW you are talking about? (ie which PCI device can do 64-bit DAC to coherent regions)

I remember that the qla2200's can...

From: Grant Grundler <[email protected]>
Date: Tue, 23 Apr 2002 14:11:37 -0700

I'm curious to see what happens with the pci_dac_alloc_consistent() idea.

I'm deep space 9'ing it currently. It makes no sense especially from a performance standpoint.

After an enlightening discussion with Mr. Miller, my patch stands as is. Does it look good enough to integrate into the next release? If not, what should I clean up?

Thanks a lot,
Jesse

Ok, there are _many_ cards that can handle 64 bit coherent pointers, including qlogic 1040, 1240, 1080, 1280, 12160, 2100, 2200, 2300, 2310, 2342, adaptec/jni 1160 (and followons, presumably). The extra address cycle is *rarely* an issue under most loads (e.g. on a 64k scsi i/o, using SAC saves you 2 or 3 cycles out of 8000). The advantage of using 64 bit mappings is that you save 32 bit mappings for cards that really need them.

So the penalty for using DAC comes down to ~.0375% for a 64k I/O, and the benefit is that you don't have to use 32 bit mappings (which can be scarce on some platforms).

In short, please reconsider your opposition to adding a pci_dac_page_to_consistent type call. It's an easy thing to add...

Thanks,
Jesse

From: Jesse Barnes <[email protected]>
Date: Wed, 24 Apr 2002 16:13:08 -0700

So the penalty for using DAC comes down to ~.0375% for a 64k I/O, and the benefit is that you don't have to use 32 bit mappings (which can be scarce on some platforms).

No some, _your_ platform. It is the only one with this problem.

You only need this 4GB memory for consistent memory, not for anything else! %99 of the kernel is going to use memory for pages and data, not descriptors.

You can use zones and whatnot to make sure this isn't a problem at all.

From: Jesse Barnes <[email protected]>
Date: Wed, 24 Apr 2002 17:11:19 -0700

So if performance isn't an obstacle to implementing a new call, and I've shown that there are platforms where 64 bit consistent mappings are better to use than 32 bit mappings when possible, why can't we add it? Or maybe just a seperate consistent_mask that drivers could set to indicate they support the operation and let the platform take care of it?

Where is the platform where it is a problem to keep around enough of the _4_ __GIGABYTES__ of consistent memory available at bootup?

If you can't keep around enough consistent memory within the lower 4GB, you have other problems.

Devices use, EACH, something like a few kilobytes of consistent memory for descriptors. If you are going to tell me this is a big deal, you have to be kidding me.

Why don't you tell me what the real problem is?

Right, the platform that I'd like to see work well.

I think you're confusing our platform with generic ia64 platforms. I'm not worried about allocating in the low 4GB or whatever, I'm concerned about using 32 bit mappings needlessly. Of course, generic ia64 boxes would also benefit...

So if performance isn't an obstacle to implementing a new call, and I've shown that there are platforms where 64 bit consistent mappings are better to use than 32 bit mappings when possible, why can't we add it? Or maybe just a seperate consistent_mask that drivers could set to indicate they support the operation and let the platform take care of it?

Thanks,
Jesse

From: Jesse Barnes <[email protected]>
Date: Wed, 24 Apr 2002 17:21:19 -0700

I thought I did. The issue isn't keeping 4gb around, it's using up the 32 bit mappings in the bridge for 64 bit capable cards.

Reread what I've quoted, these cards use almost nothing.

Consistent allocations account for maybe %2 to %6 of the total IOMMU mappings on a fully loaded sparc64 bus. It doesn't even show up on the radar.

I've rolled the numbers before, I know it isn't a problem.

I thought I did. The issue isn't keeping 4gb around, it's using up the 32 bit mappings in the bridge for 64 bit capable cards.

Jesse

Ok, so there's no performance argument against adding the call. You're just saying that since there are enough 32 bit mappings to go around, using 64 bit mappings is silly. Right?

Jesse

From: Jesse Barnes <[email protected]>
Date: Wed, 24 Apr 2002 17:36:52 -0700

Ok, so there's no performance argument against adding the call. You're just saying that since there are enough 32 bit mappings to go around, using 64 bit mappings is silly. Right?

We don't add things to Linux that we "might need" we add things we actually need. It is a slight performance problem too.

I don't think the hardware folks would be happy to hear that you're willing to sacrifice real hardware cycles just for some DAC nirvana or whatever it is you have for this issue.

From: Jesse Barnes <[email protected]>
Date: Wed, 24 Apr 2002 18:01:59 -0700

I fully understand and agree with that, and I don't propose adding the call just for the fun of it. It's just that most of the time, it's easier software-wise to use the 64 bit consistent mappings if your device can support it (e.g. generic ia64, ia64/sn).

It isn't easier, in fact it's a lot harder.

The DMA api is confusing enough, and you want to add yet another interface for device driver author Joe Blow to have to learn?

I don't think the performance impact of those extra cycles would be measurable, but I'd be happy to see any numbers you might have to the contrary.

You already showed me the numbers.

Hardware folks don't make hardware so you can waste cycles, every one counts. I guess this is a fundamental disagreement in approach between you and me. So let's just agree to disagree.

You have to show me a significant advantage to add a completely new API to the mix. It has to be important enough that it is worth making every driver author learn the new interface.

This basically boils down to the fact that it must do one of two things:

1) It must make performance phenominally better.

In the DAC consistent case, it actaully will decrease bus utilization slightly, we agree on this.

2) It must make some DMA performance optimization possible which currently is not possible at all.

I do not believe this is the case. You've tried to argue that SAC consistent memory is precious for mappings, and I've tried to show you that the consistent part of the mappings used by devices won't even show up on the radar.

We are at an impasse' and I really doubt this is going to change. All you can do by arguing further is just make me type the same things a few more times, and I'd like to avoid that, I type enough already. :-)

I fully understand and agree with that, and I don't propose adding the call just for the fun of it. It's just that most of the time, it's easier software-wise to use the 64 bit consistent mappings if your device can support it (e.g. generic ia64, ia64/sn).

I don't think the performance impact of those extra cycles would be measurable, but I'd be happy to see any numbers you might have to the contrary.

Jesse

What if it's just a simple flag (or mask like dma_mask) that the driver optionally sets to true (or 64 bits)? That way, the drivers wouldn't have to change at all and the default behaviour would be identical to the current API?

Of course, this only make sense if you acknowledge that (2) is at least of some small concern (small, simple fix for a small, simple problem).

I know the feeling. At any rate, thanks for the lively discussion. It's been educational.

Jesse