howto use ioremap_wc?

Hello,

We're looking at using ioremap_wc() in myri10ge. No drivers seem to be using it yet, so I'd like to get some clarification regarding ioremap_wc failures, MTRR and so on.

What we currently do is mtrr_add() and then ioremap. Depending on the mtrr_add() success, we use the "wc_fifo" or regular PIO with fences to submit requests to the NIC. How are we supposed to switch this to ioremap_wc?

Are we sure that if the arch supports _wc, it does not return success when the underlying plain ioremap worked but setting up _wc() failed? If so, why does it revert to ioremap_nocache when PAT isn't enabled?

Can we keep the mtrr_add() in the regular path? Or are we supposed to drop it when the arch provides ioremap_wc and it did not fail?

thanks,
Brice

Ok this leads to a question: since write combining is effectively an extension (eg relaxation) to uncached, how much do you care if you actually get uncached? Eg can you just use the "WC" function even for the case where you get an uncached mapping ?

why would it not return success? You asked for a mapping with a set of guarantees, and you got something that adheres to those guarantees (and then some ;(...

Would you expect ioremap_cache() to fail if you couldn't get a cachable mapping? That would suck (hint: on PC's it will then fail 99.9% of the time)....

Actually it would make sense for the ioremap_wc() implementation to try to add an mtrr I suppose... it's better to be done there than trying to do it in some driver....

If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

WC is strictly required for our "wcfifo" path, but this path is actually not so important nowadays. It is disabled by default and might even be removed in the future. So, no, myri10ge itself does not really need to know whether the mapping is actually WC.

The only case I see where it would be helpful is to avoid calling mtrr_add when WC got enabled through PAT. See below.

Agreed, that would be nice!

replace
ioremap+mtrr_add
with
ioremap_wc+mtrr_add

If mtrr_add() is moved into ioremap_wc(), we'll remove it from myri10ge. For now, when PAT is enabled, we may have PAT + MTRR both doing WC, but I don't think it can break anything, right?

thanks,
Brice

To be more accurate about the above statement, that codepath will still work correctly even if the mapping ends-up being uncached, but that could lead to a 16X slowdown on some machines, since that path was really designed for the WC case.

Anyway that codepath does not really matter as Brice mentioned afterwards :-)

Loic

which doesn't work as ioremap() may/will force uncached ;)

Both having WC is nicely consistent and the right thing happens...

If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

This thread reminds me of another question about interfaces for using PAT. Are there any plans for making pgprot_writecombine() a generic interface that portable drivers can use to map PCI memory into userspace with WC turned on? (Falling back to uncached when WC is not available for whatever reason is fine for the case I have in mind)

Maybe it's better to ask at a higher level: what's the right thing for a driver to do in an mmap method when it wants to expose PCI memory to userspace with WC turned on if possible? I see that the PCI sysfs code on x86 does it using non-exported (hence not available to a driver buildable as a module) interfaces.

Thanks,
Roland