MMIO accessors & barriers documentation
Ok, here's a formal documentation of the proposed accessor semantics. It still contains a couple of questions (see [* Question]) that need answering before we can start implementing anything, so I'm waiting for feedback here. The Questions are grouped at the end of the document to avoid cluttering it.
I've on purpose not included Segher proposal of having the "writel/readl" type accessors behave differently based on an ioremap flag. There are pros and cons to this approach, but this is almost a separate debate as we shall first define the semantics we need and that's what that document attempt to do.
*** Definitions of MMIO accessors and IO related barriers semantics ***
* I * Ordering requirements:
============================
First, let's define 4 types of ordering requirements that can be provided by MMIO accessors:
1- MMIO + MMIO: This type of ordering means that two consecutive MMIO
accesses performed by one processors are issued in program order on the bus. Reads can't cross writes. Writes can't be re-oredered vs. each other. There is no implication on MMIOs issued by different CPUs nor non-MMIO accesses.
2- memory W + MMIO W: This type of ordering means that an store to main
memory that is performed in program order before an MMIO store to a device, must be visible to that device before the MMIO store reaches it. For example: updating of a DMA descriptor in memory is visible to the chip before the MMIO write that cause that chip to go fetch it. This is purely a store ordering, there is no assumption made about reads
3- MMIO R + memory R: This type of ordering means that an MMIO read will
be effectively performed (the result returned by the device to the processor) before a following read for memory. That is, the value returned by that following read is what was present in the coherency domain after the MMIO read is complete. For example: reading a DMA "pointer" from a device with an MMIO read, and then fetching the data in memory up to that pointer.
4- MMIO W + spin_unlock: This type of ordering means that MMIO stores
followed by a spin unlock will have all reached the host PCI bridge before the unlocking is visible to other CPUs. For example, two CPUs have a locked section (same spinlock) issuing some MMIO stores to the same device. Such ordering means that both sets of MMIO stores will not be interleaved when reaching the host PCI controller (and thus the device). All MMIO stores from one locked section will be performed before all MMIO stores from the other.
[Note] Rule #4 is strictly specific to MMIO stores followed by a spin_unlock(). There is no ordering requirement provided by Linux to ensure ordering of an MMIO store followed by a generic memory store unless an explicit barrier is used:
[ -> Question 1]
* II * Accessors:
=================
We provide 3 classes of accessors:
Class 1: Ordered accessors
--------------------------
[Note] None of these accessors will provide write combining
1- {read,write}{b,w,l,q} : Those accessors provide all MMIO ordering
requirements. They are thus called "fully ordered". That is #1, #2 and #4 for writes and #1 and #3 for reads.
[ -> Question 2]
2- PIO accessors (all of them, that is inb...inl, ins*, out
equivalents,...): Those are fully ordered, all ordering rules apply. They are slow anyways :)
3- memcpy_to_io, memcpy_from_io: #1 semantics apply (all MMIO loads or
stores are performed in order to each other). #2+#4 (stores) or #3 (loads) semantics apply to the operation as a whole. That is #2: all previous memory stores are globally visible before the first MMIO store of memcpy_to_io, #3: The last MMIO read (and thus all previous ones too due to rule #1) have been fully performed before a subsequent memory read is performed by memcpy_from_io. And #4: all MMIO stores performed by memcpy_to_io will have reached the host bridge before the effect of a subsequent spin_unlock are visible.
4- io{read,write}{8,16,32}[be]: Those have the same semantics as 1 for
MMIO and the same semantics as 2 for PIO. As for the "repeat" versions of those, they follow the semantics of memcpy_to_io and memcpy_from_io (the only difference being the lack of increment of the MMIO address).
Class 2: Partially relaxed accessors
------------------------------------
[Note] Stores using those accessors will provide write combining on MMIO (not PIO) which have been mapped with the appropriate <insert call name here, TBD. possible ioremap_wc>
1- __{read,write}{b,w,l,q} : Those accessors provide only ordering rule
#1. That is, MMIOs are ordered vs. each other as issued by one CPU. Barriers are required to ensure ordering vs. memory and vs. locks (see "Barriers" section).
2- __io{read,write}{8,16,32}[be] (optional ?) : Those have the same
semantic as 1 for MMIO, and provide full ordering requirements as defined in Classe 1 for PIO.
3- __memcpy_to_io, __memcpy_from_io. Those provide only requirement #1,
that is the MMIOs within the copy are performed in order and are in order vs. preceeding and subsequent MMIOs executed on the same CPU.
Class 3: Fully relaxed accessors
--------------------------------
[Note] Stores using those accessors will provide write combining the same way as Class 2 accessors.
1- __raw_{read,write}{b,w,l,q} : Those accessors provide no ordering
rule whatsoever. They also provide no endian swapping. They are essentially the equivalent of a direct load/store instruction to/from the MMIO space. Access is done in platform native endian.
* III * IO related barriers
===========================
Some of the above accessors do not provide all ordering rules define in
* I *, thus explicit barriers are provided to enforce those ordering rules:
1- io_to_io_barrier() : This barrier provides ordering requirement #1
between two MMIO accesses. It's to be used in conjuction with fully relaxed accessors of Class 3.
2- memory_to_io_wb() : This barrier provides ordering requirement #2
between a memory store and an MMIO store. It can be used in conjunction with write accessors of Class 2 and 3.
3- io_to_memory_rb(value) : This barrier provides ordering requirement
#3 between an MMIO read and a subsequent read from memory. For implementation purposes on some architectures, the value actually read by the MMIO read shall be passed as an argument to this barrier. (This allows to generate the appropriate CPU instruction magic to force the CPU to consider the value as being "used" and thus force the read to be performed immediately). It can be used in conjunction with read accessors of Class 2 and 3
4- io_to_lock_wb() : This barrier provides ordering requirement #4
between an MMIO store and a subsequent spin_unlock(). It can be used in conjunction with write accessors of Class 2 and 3.
[ -> Question 3]
[ -> Question 4]
[Note] A barrier commonly used by drivers and not described here are the memory-to-memory read and write barriers (rmb, wmb, mb). Those are necessary when manipulating data structures in memory that are accessed at the same time via DMA. The rules here are identical to the usual SMP data ordering rules and are beyond the scope of this document.
[ -> Question 5]
* IV * Mixing of accessors
==========================
There are few rules concerning the mixing of accessors of the different ordering Classes. Basically, when accessors of different classes share an ordering rule, then that rule apply. If not, it doesn't apply. For example:
writel followed by __writel : both accesors provide rule #1, thus it applies and stores are visible in order. Since the previous writel will have ordered previous stores to memory, the second __writel naturally benefits from this despite the fact that __writel doesn't normally provide that semantic.
Ben.
Questions:
==========
[* Question 1] Should Rule #4 be generalized to MMIO store followed by a memory store ? (as spin_unlock are essentially a wmb followed by a memory store) or do we need to keep a rule specific for locks to avoid arch specific pitfalls on some architecture ? In that case, do we need a specific barrier to provide MMIO store followed by a memory store ? That sort of ordering is not generally useful and is generally expensive as it requires to access the PCI host bridge to enforce that the previous MMIO stores have reached the bus. Drivers generally don't need such a rule or a barrier, as they have to deal with write posting anyway, and thus use an MMIO read to provide the necessary synchronisation when it makes sense.
[* Question 2] : Do we actually want the "ordered" accessors to also provide ordering rule #4 in the general case ? This can be very expensive on some architectures like ia64 where, I think, it has to actually access the PCI host bridge to provide the guarantee that the previous MMIO stores have reached it before the unlock is made visible to the coherency domain. If we decide not to, then an explicit barrier will still be needed in most drivers before spin_unlock(). This is the current mmiowb() barrier that I'm proposing to rename (section * III *). A way to provide that ordering requirement with less performance impact is to instead set a per-cpu flag in writeX(), and test it in spin_unlock() which would then do the barrier only if the flag is set. It's to be measured whether the impact on unrelated spin_unlock() is low enough to make that solution realstic.
If we decide to not enforce rule #4 for ordered accessors, and thus require the barrier before spin_unlock, the above trick, could still be implemented as a debug option to "detect" the lack of appropriate barriers.
[* Question 3] If we decide that accessors of Class 1 do not provide rule #4, then this barrier is to be used for all classes of accessors, except maybe PIO which should always be fully ordered.
[* Question 4] Would it be a useful optimisation on archs like ia64 to require this accessor to take the struct device of the device as an argument (with can NULL for a "generic" barrier) or it doesn't matter ?
[* Question 5] Should we document the rules for memory-memory barriers here as well ? (and give examples, like live updating of a network driver ring descriptor entry)
Ar Llu, 2006-09-11 am 14:03 +1000, ysgrifennodd Benjamin Herrenschmidt:
Quoted 1 line
be interleaved when reaching the host PCI controller (and thus the
"a host PCI controller". The semantics with multiple independant PCI busses are otherwise evil.
Quoted 3 lines
1- {read,write}{b,w,l,q} : Those accessors provide all MMIO ordering
requirements. They are thus called "fully ordered". That is #1, #2 and #4 for writes and #1 and #3 for reads.
#4 may be incredibly expensive on NUMA boxes.
Quoted 2 lines
3-
memcpy_to_io,memcpy_from_io: #1 semantics apply (all MMIO loads or
stores are performed in order to each other). #2+#4 (stores) or #3
What is "in order" here. "In ascending order of address" would be tighter.
Quoted 4 lines
1- __{read,write}{b,w,l,q} : Those accessors provide only ordering rule
#1. That is, MMIOs are ordered vs. each other as issued by one CPU. Barriers are required to ensure ordering vs. memory and vs. locks (see "Barriers" section).
"Except where the underlying device is marked as cachable or prefetchable"
Q2:
Quoted 3 lines
coherency domain. If we decide not to, then an explicit barrier will still be needed in most drivers before
spin_unlock(). This is the currentmmiowb()barrier that I'm proposing to rename (section * III *).
I think we need mmiowb() still anyway (for __writel etc)
Quoted 4 lines
If we decide to not enforce rule #4 for ordered accessors, and thus require the barrier before
spin_unlock, the above trick, could still be implemented as a debug option to "detect" the lack of appropriate barriers.
This I think is an excellent idea.
Quoted 3 lines
[* Question 3] If we decide that accessors of Class 1 do not provide rule #4, then this barrier is to be used for all classes of accessors, except maybe PIO which should always be fully ordered.
On x86 PIO (outb/inb) etc are always ordered and always stall until the cycle completes on the device.
Quoted 4 lines
[* Question 5] Should we document the rules for memory-memory barriers here as well ? (and give examples, like live updating of a network driver ring descriptor entry)
Update the existing docs
Quoted 5 lines · Alan Cox
Ar Llu, 2006-09-11 am 14:03 +1000, ysgrifennodd Benjamin Herrenschmidt:
Quoted 1 line
be interleaved when reaching the host PCI controller (and thus the
"a host PCI controller". The semantics with multiple independant PCI busses are otherwise evil.
Ok.
Quoted 5 lines
Quoted 3 lines
1- {read,write}{b,w,l,q} : Those accessors provide all MMIO ordering
requirements. They are thus called "fully ordered". That is #1, #2 and #4 for writes and #1 and #3 for reads.#4 may be incredibly expensive on NUMA boxes.
Yes, and that's why there is Question #2 :)
I don't care either way for PowerPC at this point, but it's an open question and I'd like folks like you to tell me what you prefer.
Quoted 5 lines
Quoted 2 lines
3-
memcpy_to_io,memcpy_from_io: #1 semantics apply (all MMIO loads or
stores are performed in order to each other). #2+#4 (stores) or #3What is "in order" here. "In ascending order of address" would be tighter.
In program order. Every time I say "in order", I mean "in program order". I agree that this is not enough precision as it's not obvious that memcpy will copy in ascending order of addresses (it doesn't have to), I'll add that precision... or not. THat could be another question. What do we want here ? I would rather have those strongly ordered for Class 1.
Quoted 7 lines
Quoted 4 lines
1- __{read,write}{b,w,l,q} : Those accessors provide only ordering rule
#1. That is, MMIOs are ordered vs. each other as issued by one CPU. Barriers are required to ensure ordering vs. memory and vs. locks (see "Barriers" section)."Except where the underlying device is marked as cachable or prefetchable"
You aren't supposed to use MMIO accessors on cacheable memory, are you ? On PowerPC, even if using cacheable mappings, they would still be visible in order to coherency domain, though being cacheable, there is indeed no saying in what order they'll end up hitting the PCI host bridge. In fact, I know of platforms (like Apple G5s) who cannot cope with cacheable mappings of anything behind HT... I'd keep use of cacheable mapping as an arch specific special case for now, and that definitely doesn't allow for MMIO accessors ...
Quoted 6 lines
Q2:
Quoted 3 lines
coherency domain. If we decide not to, then an explicit barrier will still be needed in most drivers before
spin_unlock(). This is the currentmmiowb()barrier that I'm proposing to rename (section * III *).I think we need
mmiowb()still anyway (for__writeletc)
Oh, we surely have a barrier providing that semantic (I call it io_to_lock_wb() in my proposal, and it can be #defined to mmiowb to ease driver migration). The question is wether we want rule #4 to be enforced by accessors of Class 1 or not ..
Quoted 6 lines
Quoted 4 lines
If we decide to not enforce rule #4 for ordered accessors, and thus require the barrier before
spin_unlock, the above trick, could still be implemented as a debug option to "detect" the lack of appropriate barriers.This I think is an excellent idea.
Thanks :)
Quoted 6 lines
Quoted 3 lines
[* Question 3] If we decide that accessors of Class 1 do not provide rule #4, then this barrier is to be used for all classes of accessors, except maybe PIO which should always be fully ordered.
On x86 PIO (outb/inb) etc are always ordered and always stall until the cycle completes on the device.
Yes and I think that as far as PIO is concerned, we shall remain as close as possible to x86. PIO is mostly used by "old stuff", that is drivers that are likely not to have been adapted/audited to undestand ordering issues, and is generally slow anyway. Thus even if we decide to relax rule #4 for Class 1 MMIO accessors, I'd be tempted to keep it for PIO (and config space too btw)
Quoted 6 lines
Quoted 4 lines
[* Question 5] Should we document the rules for memory-memory barriers here as well ? (and give examples, like live updating of a network driver ring descriptor entry)
Update the existing docs
Ok.
Thanks for your comments. I'll wait for more of these and post an updated version tomorrow. I'm still waiting for your preference regarding including or not rule #4 for Class 1 (ordered) MMIO accessors.
Cheers,
Ben.
Ar Llu, 2006-09-11 am 19:17 +1000, ysgrifennodd Benjamin Herrenschmidt:
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
3-
memcpy_to_io,memcpy_from_io: #1 semantics apply (all MMIO loads or
stores are performed in order to each other). #2+#4 (stores) or #3What is "in order" here. "In ascending order of address" would be tighter.
Quoted 6 lines
In program order. Every time I say "in order", I mean "in program order". I agree that this is not enough precision as it's not obvious that memcpy will copy in ascending order of addresses (it doesn't have to), I'll add that precision... or not. THat could be another question. What do we want here ? I would rather have those strongly ordered for Class 1.
I'd rather memcpy_to/from_io only made guarantees about the start/end of the transfer and not order of read/writes or size of read/writes. The reason being that a more restrictive sequence can be efficiently expressed using read/writefoo but the reverse is not true.
Quoted 4 lines
Quoted 2 lines
"Except where the underlying device is marked as cachable or prefetchable"
You aren't supposed to use MMIO accessors on cacheable memory, are you ?
Why not. Providing it is in MMIO space, consider ROMs for example or write path consider frame buffers.
Quoted 3 lines
with cacheable mappings of anything behind HT... I'd keep use of cacheable mapping as an arch specific special case for now, and that definitely doesn't allow for MMIO accessors ...
I'm describing existing semantics 8)
Quoted 4 lines
I'd rather memcpy_to/from_io only made guarantees about the start/end of the transfer and not order of read/writes or size of read/writes. The reason being that a more restrictive sequence can be efficiently expressed using read/writefoo but the reverse is not true.
Ok, so we would define ordering on the first and last accesses (being the first and last in ascending addresses order) and leave it free to the implementation to do what it wants in between. Is that ok ?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
"Except where the underlying device is marked as cachable or prefetchable"
You aren't supposed to use MMIO accessors on cacheable memory, are you ?
Why not. Providing it is in MMIO space, consider ROMs for example or write path consider frame buffers.
If we consider cacheable accesses, we need to also provide cache flushing primitives as MMIO devices are generally not coherent. Take for example the case of the frame buffer: you may want to upload a texture, and later use it with the engine. You need a way in between to make sure all the cached dirty lines have been pushed to the device before you start the engine. Since we provide no generically useable functions for doing such cache coherency on MMIO space, I'd rather keep usage of MMIO accessors on cacheable storage non-defined. That is add a simple note at the top of the file that the rules defined here only apply to non-cacheable mappings. Is that ok ?
Quoted 5 lines
Quoted 3 lines
with cacheable mappings of anything behind HT... I'd keep use of cacheable mapping as an arch specific special case for now, and that definitely doesn't allow for MMIO accessors ...
I'm describing existing semantics 8)
Well, there is no clear existing semantics, at least not global to all archs for cacheable access to MMIO so yeah, let's say that ordering on cacheable storage is left undefined :)
Ben.
Ar Llu, 2006-09-11 am 19:59 +1000, ysgrifennodd Benjamin Herrenschmidt:
Quoted 3 lines
Ok, so we would define ordering on the first and last accesses (being the first and last in ascending addresses order) and leave it free to the implementation to do what it wants in between. Is that ok ?
Not sure you can go that far. I'd stick to "_fromio/_toio" transfer blocks of data efficiently between host and bus addresses. The guarantees are the same as readl/writel respectively with respect to the start and end of the transfer.
[How do you define start and end addresses with memcpy_fromio(foo, bar,
4) for example ]
Quoted 5 lines · Alan Cox
Ar Llu, 2006-09-11 am 19:59 +1000, ysgrifennodd Benjamin Herrenschmidt:
Quoted 3 lines
Ok, so we would define ordering on the first and last accesses (being the first and last in ascending addresses order) and leave it free to the implementation to do what it wants in between. Is that ok ?
Quoted 7 lines
Not sure you can go that far. I'd stick to "_fromio/_toio" transfer blocks of data efficiently between host and bus addresses. The guarantees are the same as readl/writel respectively with respect to the start and end of the transfer.
[How do you define start and end addresses with memcpy_fromio(foo, bar,
4) for example ]
Ok. So they behave like a writel or a readl globally respective to other accesses but there is no guarantee in order or size of the individual transfers making them up.
Ben
Quoted 7 lines · Alan Cox
Quoted 4 lines
Quoted 2 lines
"Except where the underlying device is marked as cachable or prefetchable"
You aren't supposed to use MMIO accessors on cacheable memory, are you ?
Why not. Providing it is in MMIO space, consider ROMs for example or write path consider frame buffers.
Frame buffers are rarely cachable as such, on x86 they are usually
write-combining. Which means that the writes can be merged and
possibly reordered while they are being written but they can't be cached. Most arches I believe have something that roughly corresponds to write combining.
Ensuring we can still use this optimization to mmio space is moderately important.
Eric
Quoted 9 lines
Frame buffers are rarely cachable as such, on x86 they are usually
write-combining. Which means that the writes can be merged and
possibly reordered while they are being written but they can't be cached. Most arches I believe have something that roughly corresponds to write combining.Ensuring we can still use this optimization to mmio space is moderately important.
I've not gone too much in details about write combining (we need to do something about it but I don't want to mix problems) but I did define that the ordered accessors aren't guaranteed to provide write combining on storage mapped with WC enabled while the relaxed or non ordered ones are. That should be enough at this point.
Later, we should look into providing an ioremap_wc() and possibly page table flags for write combining userland mappings. Time to get rid of MTRRs for graphics :) And infiniband-style stuff seems to want that too.
Ben.
Quoted 5 lines · Benjamin Herrenschmidt
I've not gone too much in details about write combining (we need to do something about it but I don't want to mix problems) but I did define that the ordered accessors aren't guaranteed to provide write combining on storage mapped with WC enabled while the relaxed or non ordered ones are. That should be enough at this point.
Sounds good.
Quoted 3 lines
Later, we should look into providing an
ioremap_wc()and possibly page table flags for write combining userland mappings. Time to get rid of MTRRs for graphics :) And infiniband-style stuff seems to want that too.
ioremap_wc is actually the easy half. I have an old patch that handles that. The trick is to make certain multiple people don't map the same thing with different attributes. Unfortunately I haven't had time to work through that one yet.
Eric
Quoted 5 lines
ioremap_wcis actually the easy half. I have an old patch that handles that. The trick is to make certain multiple people don't map the same thing with different attributes. Unfortunately I haven't had time to work through that one yet.
Actually, that's interesting because I need the exactly oposite on PowerPC I think.... That is people will -need- to do both a wc and a non-wc mapping if they want to be able to issue stores that are guaranteed not to be combined.
The problem I've seen is that at least one processor (the Cell) and maybe more seem to be combining between threads on the same CPU (unless the stores are issues to a guarded mapping which prevents combining completely, that is the sort of mapping we currently do with ioremap).
That means that it's impossible to prevent combining with explicit barriers. For example:
Thread 0 Thread 1 store to A store to A+1 barrier barrier
\ /
\ /
\ /
Store unit might sees:
store to A
store to A+1
barrier
barrier
That is the stores aren't tagged with their source thread and thus the non cacheable store unit will not prevent combining between them.
Again, it might just be a Cell CPU bug in which case we may have to just disable use of WC on that processor, period. But it might be a more generic problem too, we need to investigate.
If the problem ends up being widespread, the only ways I see to prevent the combining from happening are to do a dual mapping as I explained earlier, or maybe to have drivers always do the stores that must not be combine as part of spinlocks, with appropriate use of io_to_lock_barrier() (mmiowb()).
Anyway, let's not pollute this discussion with that too much now :)
Ben.
Quoted 4 lines
Actually, that's interesting because I need the exactly oposite on PowerPC I think.... That is people will -need- to do both a wc and a non-wc mapping if they want to be able to issue stores that are guaranteed not to be combined.
Or you do the sane thing and just not allow two threads of execution access to the same I/O device at the same time.
Quoted 7 lines
The problem I've seen is that at least one processor (the Cell) and maybe more seem to be combining between threads on the same CPU (unless the stores are issues to a guarded mapping which prevents combining completely, that is the sort of mapping we currently do with ioremap).
That means that it's impossible to prevent combining with explicit barriers. For example:
Now compare this with the similar scenario for "normal" MMIO, where we do store;sync (or sync;store or even sync;store;sync) for every writel() -- exactly the same problem.
Quoted 3 lines
Again, it might just be a Cell CPU bug in which case we may have to just disable use of WC on that processor, period. But it might be a more generic problem too, we need to investigate.
It's a bit like why IA64 has mmiowb(). Not quite the same, but similar.
Quoted 5 lines
If the problem ends up being widespread, the only ways I see to prevent the combining from happening are to do a dual mapping as I explained earlier, or maybe to have drivers always do the stores that must not be combine as part of spinlocks, with appropriate use of
io_to_lock_barrier()(mmiowb()).
Better lock at a higher level than just per instruction.
Some devices that want to support multiple clients at the same time have multiple identical "register files", one for each client, to prevent this and other problems (and it's useful anyway).
Quoted 1 line
Anyway, let's not pollute this discussion with that too much now :)
Au contraire -- if you're proposing to hugely invasively change some core interface, and add millions of little barriers(*), you better explain how this is going to help us tackle the problems (like WC) that we are starting to see already, and that will be a big deal in the near future.
Now I'm saying there's no way to make the barriers needed for write- combining efficient, unless those barriers can take advantage of the ordering rules of the path all the way from the CPU to the device; i.e. make those barriers bus-specific. The MMIO and memory-like-space read/write accessors will have to follow suit. Non-WC stuff can take advantage of bus-specific rules as well, e.g. the things you are proposing, which, face it, are really just designed for PCI.
And even today, only looking at PCI, we already have two different kinds of drivers: the ones that use the PCI ordering rules, with wmb() and mmiowb() [your #2 and #4; #1 is implicit on PCI (everything pushes posted writes); and #3 is covered by the twi;isync we have in readX()], which work correctly on PowerPC today; and on the other hand, the drivers that pretend PCI is a bus arch where everything is strongly ordered (even vs. main memory), which do not all work for PowerPC in today's kernel [devices not doing DMA might seem to work fine, since #4 is hard to break and even if you do it's not often fatal or bad at all; heck we have this one device now where breaking #2 "almost" works, it took almost two full kernel release cycles for anyone to notice].
If you change the rules you'll have to audit *all* existing device drivers.
So, again: unless we make the I/O accessors and barriers bus-specific, we'll end up with millions(**) of slightly different barriers and whatnot, in an attempt to get decent performance out of our devices; and we will never reach that goal. Also, no device driver author will ever know what barrier to use where and when.
Now if we _do_ make it all bus-specific, we still might have quite a few barriers in total, but only a few per bus type -- and they can have descriptive names that explain where to use them. Maybe, just maybe, we'll for the first time see a device driver that gets it right ;-)
I still like the idea of overloading the semantics of readX()/writeX() to do whatever is needed for the region that is mapped for their arguments, but you can introduce pci_readl() and friends for all I care, it's a separate issue... If you want to keep the nice short names with different semantics though, well, have fun fixing device drivers for the next twenty(***) years.
Segher
(*) Yes I know I'm exaggerating.
(**) It's a habit :-)
(***) Did it again... It's more like fifteen years really.
Quoted 3 lines
Or you do the sane thing and just not allow two threads of execution access to the same I/O device at the same time.
Why ? Some devices are designed to be able to handle that...
Quoted 3 lines
Now compare this with the similar scenario for "normal" MMIO, where we do store;sync (or sync;store or even sync;store;sync) for every
writel()-- exactly the same problem.
What problem ? "Normal" MMIO doesn't get combined, thus there is no problem. Of course there is no guarantee of ordering of the stores from the 2 CPUs unless there is a spinlock etc etc... but we are talking about a case where that is acceptable here. Howver, combining is not.
Quoted 5 lines
Better lock at a higher level than just per instruction.
Some devices that want to support multiple clients at the same time have multiple identical "register files", one for each client, to prevent this and other problems (and it's useful anyway).
Yes, they do, and what happen if those register "files" happen to be consecutive in the address space and the CPU suddenly combines a store to the last register of one "file" and an unrelated store from another thread to the first register of the other ?
This is a very specific problem that has nothing to do with your "grand general case" which means that at least on Cell, you cannot use explicit barriers to guarantee the absence of write combining. That's as simple as that. All I need to figure out now is if that problem is specific to one CPU implementation or more general, in which case, we'll have to figure out some way to provide an interface.
Quoted 7 lines
Quoted 1 line
Anyway, let's not pollute this discussion with that too much now :)
Au contraire -- if you're proposing to hugely invasively change some core interface, and add millions of little
barriers(*), you better explain how this is going to help us tackle the problems (like WC) that we are starting to see already, and that will be a big deal in the near future.
No, this is totally irrelevant. I'm proposing a simple change (nothing invasive there) to the MMIO accessors of weakly ordered platforms only, to make them guarantee ordering like x86 etc... and I'm proposing the -addition- (which is not something I would cause invasive) of -one- class of partially relaxed accessors and the -few- (damn, there are only 4 of them) barriers that precisely match the semantics that drivers need. Oh, and make sure those semantics are well defined or they are useless.
This has strictly nothing to do with WC and mixing things up will only confuse the discussion and guarantee that we'll never get anything done.
<snip useless digression>
Ben.
Quoted 4 lines
Quoted 2 lines
Or you do the sane thing and just not allow two threads of execution access to the same I/O device at the same time.
Why ? Some devices are designed to be able to handle that...
Sure, but not many -- and even then, you normally get a separate MMIO area to write to for each thread. Not really differrnt.
Quoted 8 lines
Quoted 3 lines
Now compare this with the similar scenario for "normal" MMIO, where we do store;sync (or sync;store or even sync;store;sync) for every
writel()-- exactly the same problem.What problem ? "Normal" MMIO doesn't get combined, thus there is no problem. Of course there is no guarantee of ordering of the stores from the 2 CPUs unless there is a spinlock etc etc... but we are talking about a case where that is acceptable here. Howver, combining is not.
As an example, the first access might set off a DMA, and the 2nd MMIO interferes. That's not necessarily acceptable. Now you might point me to the spinlock again, but I'll just point you right back to your original example, because that's my whole point.
Quoted 10 lines
Quoted 5 lines
Better lock at a higher level than just per instruction.
Some devices that want to support multiple clients at the same time have multiple identical "register files", one for each client, to prevent this and other problems (and it's useful anyway).
Yes, they do, and what happen if those register "files" happen to be consecutive in the address space and the CPU suddenly combines a store to the last register of one "file" and an unrelated store from another thread to the first register of the other ?
That's why those devices rely on the CPU's not combining over the edges of (typically) 4kB pages.
Quoted 2 lines
This is a very specific problem that has nothing to do with your "grand general case"
Oh I have no "grand general case", my main argument still is to have accessors _per bus_ (per bus type really, archs can make it more specific if they want).
In the "grand general case", you have to do lowest-common-denominator for everything, and you're increasingly forcing yourself into that corner.
Quoted 9 lines
Quoted 7 lines
Quoted 1 line
Anyway, let's not pollute this discussion with that too much now :)
Au contraire -- if you're proposing to hugely invasively change some core interface, and add millions of little
barriers(*), you better explain how this is going to help us tackle the problems (like WC) that we are starting to see already, and that will be a big deal in the near future.No, this is totally irrelevant.
"The (near) future [and it's only not right now because Linux is dragging behind] is totally irrelevant, only my current this-second itch is?"
Quoted 3 lines
I'm proposing a simple change (nothing
invasive there) to the MMIO accessors of weakly ordered platforms only, to make them guarantee ordering like x86 etc...
Please explain what drivers will need changes because of this. Not just the few you really care about, but _all_ that could be plugged into PowerPC machines' PCI busses, and might need changes because of changing the ordering semantics of readX()/writeX() from the supposed standard Linux semantics (i.e., the x86 semantics).
Quoted 6 lines
and I'm proposing the
-addition- (which is not something I would cause invasive) of -one- class of partially relaxed accessors and the -few- (damn, there are only 4 of them) barriers that precisely match the semantics that drivers need. Oh, and make sure those semantics are well defined or they are useless.
Erm, wait a minute, I might start to understand now... You want all drivers that you care about to be converted to use __readX()/__writeX() instead? How is this going to help, exactly?
Quoted 2 lines
This has strictly nothing to do with WC and mixing things up will only confuse the discussion and guarantee that we'll never get anything done.
No, it _has_ to do with WC. If the Linux I/O API is going to be changed/ amended/expanded/mot du jour, we better do it in such a way that we will get a positive outlook on the problems that we will have to face next (or in this case, that we should be handling already really).
Quoted 1 line
<snip useless digression>
Very constructive.
Segher
Quoted 7 lines
Please explain what drivers will need changes because of this. Not just the few you really care about, but _all_ that could be plugged into PowerPC
machines' PCI busses, and might need changes because of changing the ordering semantics of readX()/writeX() from the supposed standard Linux semantics (i.e., the x86 semantics).
They won't. They will still work, and in some (many ?) case better due to the removal of a potential bug since lots of driver don't have a barrier where they should be with relaxed semantics. So the net effect is positive here.
Now, it also means that we -can- start improving drivers we care about to use the relaxed semantics and benefit from there. And since the semantics are well defined, all archs with some sort of relaxed ordering will be able to benefit in a way or another.
In addition, it will allow us to get a small optimisation on PowerPC vs. the current situation by slightly relaxing wmb() which currently has to do a full sync because it might be used to order memory vs. MMIO, which it will no longer do (it will go back to a pure memory store barrier).
Anyway, Paul has a patch we are testing that makes our writel/readl's synchronous (by moving the sync to before writel, adding an eieio before readl, and doing the percpu trick so spin_unlock magically does a sync when a writel occurred). With that, we'll get full correctness with no more sync's in writel than we had before. We are running some benchs here now to see what kind of performance impact it has overall, and if we are happy, that can make it into 2.6.18 and close the problem of drivers assuming ordered MMIO vs. memory at least.
Then, in a -separate- step, we can provide a set of relaxed accessors that will allow for additional performance improvements on the hot path of selected drivers.
I'm tired of arguing over and over again the same thing here anyway, I'll post a new version of the document including some of the feedback we got already and will submit it for inclusion along with a __writel/__readl implementation for powerpc (and a generic one that defaults to readl/writel) for the 2.6.19 timeframe.
We'll see from there if there are more constructive comments.
Ben.
Quoted 3 lines · Benjamin Herrenschmidt
1- {read,write}{b,w,l,q} : Those accessors provide all MMIO ordering
requirements. They are thus called "fully ordered". That is #1, #2 and #4 for writes and #1 and #3 for reads.
Fine.
Quoted 3 lines
2- PIO accessors (all of them, that is inb...inl, ins*, out
equivalents,...): Those are fully ordered, all ordering rules apply. They are slow anyways :)
Yeah, I think these are already defined to operate this way. Not sure if that fact is documented clearly though (haven't checked).
Quoted 9 lines
3-
memcpy_to_io,memcpy_from_io: #1 semantics apply (all MMIO loads or
stores are performed in order to each other). #2+#4 (stores) or #3 (loads) semantics apply to the operation as a whole. That is #2: all previous memory stores are globally visible before the first MMIO store ofmemcpy_to_io, #3: The last MMIO read (and thus all previous ones too due to rule #1) have been fully performed before a subsequent memory read is performed bymemcpy_from_io. And #4: all MMIO stores performed bymemcpy_to_iowill have reached the host bridge before the effect of a subsequentspin_unlockare visible.
See Alan's comments here. I don't think the intra-memcpy semantics have to be defined as strongly as you say here... it should be enough to treat the whole memcpy as a unit, not specifying what happens inside but rather defining it to be strongly ordered wrt to previous and subsequent code.
Quoted 4 lines
4- io{read,write}{8,16,32}[be]: Those have the same semantics as 1 for
MMIO and the same semantics as 2 for PIO. As for the "repeat" versions of those, they follow the semantics ofmemcpy_to_ioandmemcpy_from_io(the only difference being the lack of increment of the MMIO address).
This reminds me... when these routines were added I asked that they be defined as having weak ordering wrt DMA (does linux-arch have archives?), but then I think Linus changed his mind?
Quoted 4 lines
1- __{read,write}{b,w,l,q} : Those accessors provide only ordering rule
#1. That is, MMIOs are ordered vs. each other as issued by one CPU. Barriers are required to ensure ordering vs. memory and vs. locks (see "Barriers" section).
Ok, but I still don't like the naming. __ implies some sort of implementation detail and doesn't communicate meaning very clearly. But I'm not going to argue too much about it.
Quoted 7 lines
Some of the above accessors do not provide all ordering rules define in
* I *, thus explicit barriers are provided to enforce those ordering rules:1-
io_to_io_barrier(): This barrier provides ordering requirement #1
between two MMIO accesses. It's to be used in conjuction with fully relaxed accessors of Class 3.
Ok, basically mb() but for I/O space.
Quoted 4 lines
2-
memory_to_io_wb(): This barrier provides ordering requirement #2
between a memory store and an MMIO store. It can be used in conjunction with write accessors of Class 2 and 3.
Quoted 8 lines
3-
io_to_memory_rb(value): This barrier provides ordering requirement
#3 between an MMIO read and a subsequent read from memory. For implementation purposes on some architectures, the value actually read by the MMIO read shall be passed as an argument to this barrier. (This allows to generate the appropriate CPU instruction magic to force the CPU to consider the value as being "used" and thus force the read to be performed immediately). It can be used in conjunction with read accessors of Class 2 and 3
These sound fine. I think PPC64 is the only platform that will need them?
Quoted 3 lines
4-
io_to_lock_wb(): This barrier provides ordering requirement #4
between an MMIO store and a subsequentspin_unlock(). It can be used in conjunction with write accessors of Class 2 and 3.
Ok.
Quoted 5 lines
[Note] A barrier commonly used by drivers and not described here are the memory-to-memory read and write barriers (rmb, wmb, mb). Those are necessary when manipulating data structures in memory that are accessed at the same time via DMA. The rules here are identical to the usual SMP data ordering rules and are beyond the scope of this document.
Unless as Alan suggests these barriers are also documented in memory-barriers.txt (probably a good place).
Quoted 11 lines
[* Question 1] Should Rule #4 be generalized to MMIO store followed by a memory store ? (as
spin_unlockare essentially a wmb followed by a memory store) or do we need to keep a rule specific for locks to avoid arch specific pitfalls on some architecture ? In that case, do we need a specific barrier to provide MMIO store followed by a memory store ? That sort of ordering is not generally useful and is generally expensive as it requires to access the PCI host bridge to enforce that the previous MMIO stores have reached the bus. Drivers generally don't need such a rule or a barrier, as they have to deal with write posting anyway, and thus use an MMIO read to provide the necessary synchronisation when it makes sense.
But isn't this how you'll implement io_to_lock_wb() on PPC anyway? If so, might be best to name it and document it that way (though keeping the idea of barriering before unlocking prominent in the documentation).
Quoted 2 lines
[* Question 2] : Do we actually want the "ordered" accessors to also provide ordering rule #4 in the general case ?
Isn't that the whole point of making the regular readX/writeX strongly ordered? To get rid of the need for mmiowb() in the general case and make it into a performance optimization to be used in conjunction with __writeX?
Quoted 4 lines
If we decide to not enforce rule #4 for ordered accessors, and thus require the barrier before
spin_unlock, the above trick, could still be implemented as a debug option to "detect" the lack of appropriate barriers.
I think this should be done in any case, and I think it can be done in generic code (using per-cpu counters in the spinlock and mmiowb() routines); it's a good idea.
Quoted 3 lines
[* Question 3] If we decide that accessors of Class 1 do not provide rule #4, then this barrier is to be used for all classes of accessors, except maybe PIO which should always be fully ordered.
Right, though see above about my understanding of the genesis of this discussion. :)
Quoted 3 lines
[* Question 4] Would it be a useful optimisation on archs like ia64 to require this accessor to take the struct device of the device as an argument (with can NULL for a "generic" barrier) or it doesn't matter ?
For ia64 in particular it doesn't matter, though there was speculation several years that it might be necessary. No actual examples stepped forward though, so the current implementation doesn't take an argument.
Quoted 3 lines
[* Question 5] Should we document the rules for memory-memory barriers here as well ? (and give examples, like live updating of a network driver ring descriptor entry)
Should probably be added to memory-barriers.txt.
Thanks,
Jesse
Quoted 14 lines
Quoted 5 lines
2-
memory_to_io_wb(): This barrier provides ordering requirement #2
between a memory store and an MMIO store. It can be used in conjunction with write accessors of Class 2 and 3.Quoted 8 lines
3-
io_to_memory_rb(value): This barrier provides ordering requirement
#3 between an MMIO read and a subsequent read from memory. For implementation purposes on some architectures, the value actually read by the MMIO read shall be passed as an argument to this barrier. (This allows to generate the appropriate CPU instruction magic to force the CPU to consider the value as being "used" and thus force the read to be performed immediately). It can be used in conjunction with read accessors of Class 2 and 3
Quoted 1 line
These sound fine. I think PPC64 is the only platform that will need them?
Ah ? What about the comment in e1000 saying that it needs a wmb() between descriptor updates in memory and the mmio to kick them ? That would typically be a memory_to_io_wb(). Or are your MMIOs ordered cs. your cacheable stores ?
Quoted 6 lines
Quoted 3 lines
4-
io_to_lock_wb(): This barrier provides ordering requirement #4
between an MMIO store and a subsequentspin_unlock(). It can be used in conjunction with write accessors of Class 2 and 3.Ok.
Quoted 8 lines
Quoted 5 lines
[Note] A barrier commonly used by drivers and not described here are the memory-to-memory read and write barriers (rmb, wmb, mb). Those are necessary when manipulating data structures in memory that are accessed at the same time via DMA. The rules here are identical to the usual SMP data ordering rules and are beyond the scope of this document.
Unless as Alan suggests these barriers are also documented in memory-barriers.txt (probably a good place).
They are, but I was thinking about providing more IO-like examples. I suppose I could refer to memory-barriers.txt from here and update it with IO-like examples.
Quoted 12 lines
Quoted 11 lines
[* Question 1] Should Rule #4 be generalized to MMIO store followed by a memory store ? (as
spin_unlockare essentially a wmb followed by a memory store) or do we need to keep a rule specific for locks to avoid arch specific pitfalls on some architecture ? In that case, do we need a specific barrier to provide MMIO store followed by a memory store ? That sort of ordering is not generally useful and is generally expensive as it requires to access the PCI host bridge to enforce that the previous MMIO stores have reached the bus. Drivers generally don't need such a rule or a barrier, as they have to deal with write posting anyway, and thus use an MMIO read to provide the necessary synchronisation when it makes sense.
Quoted 3 lines
But isn't this how you'll implement
io_to_lock_wb()on PPC anyway? If so, might be best to name it and document it that way (though keeping the idea of barriering before unlocking prominent in the documentation).
Well, the whole question is what does the linux semantics guarantee to driver writers (accross archs), not what PowerPC implements :) I'd rather not add guarantees that aren't useful to drivers even if all current implementations happen to provide them. I'm trying to find a case where ordering MMIO W + memory W is useful and I can't see any since the MMIO W will take any time to go to the device anyway. The lock rule seems to be the only useful, thus the only I think I'll guarantee.
Quoted 6 lines
Quoted 2 lines
[* Question 2] : Do we actually want the "ordered" accessors to also provide ordering rule #4 in the general case ?
Isn't that the whole point of making the regular readX/writeX strongly ordered? To get rid of the need for
mmiowb()in the general case and make it into a performance optimization to be used in conjunction with__writeX?
Well, as far as I'm concerned, the whole point is rule #2 and #3 :) Those are the ones biting us on PowerPC (we haven't seen the lock problem but then it can't happen the way our current accessors are written. However, if we change our accessors to provide rule #2 more specifically, we'll end up with 2 sync instructions in writel, one for rule #2 before the store and one for rule #4, thus we go from expensive to very expensive). It's also my understanding that mmiowb is very expensive on ia64 and gets worse as the box grows bigger.
Hence the question: do we provide -fully- ordered accessors in class 1, or do we provide -mostly- ordered accessors, ordered in all means except rule #4 vs locks. ia64 is afaik by far the platform taking the biggest hit if you have to provide #4, so I'm interesting in your point of view here.
Quoted 8 lines
Quoted 4 lines
If we decide to not enforce rule #4 for ordered accessors, and thus require the barrier before
spin_unlock, the above trick, could still be implemented as a debug option to "detect" the lack of appropriate barriers.I think this should be done in any case, and I think it can be done in generic code (using per-cpu counters in the spinlock and
mmiowb()routines); it's a good idea.
We don't need counters, just a flag. We did a test implementation, seems to work. We also clear the flag in spin_lock. That means that MMIOs issued before a lock aren't ordered vs. the locked section. But because of rule #1, they should be ordered vs. other MMIOs inside the locked section and thus implicitely get ordered anyway.
Quoted 6 lines
Quoted 3 lines
[* Question 3] If we decide that accessors of Class 1 do not provide rule #4, then this barrier is to be used for all classes of accessors, except maybe PIO which should always be fully ordered.
Right, though see above about my understanding of the genesis of this discussion. :)
As far as I'm concerned, genesis of this discussion is rules #2 and #3, not #4 :) Though the later quickly came in of course.
Quoted 7 lines
Quoted 3 lines
[* Question 4] Would it be a useful optimisation on archs like ia64 to require this accessor to take the struct device of the device as an argument (with can NULL for a "generic" barrier) or it doesn't matter ?
For ia64 in particular it doesn't matter, though there was speculation several years that it might be necessary. No actual examples stepped forward though, so the current implementation doesn't take an argument.
Ok. My question is wether it would improve the implementation to take it. If we define a new macro with a new name, we can do it....
Quoted 5 lines
Quoted 3 lines
[* Question 5] Should we document the rules for memory-memory barriers here as well ? (and give examples, like live updating of a network driver ring descriptor entry)
Should probably be added to memory-barriers.txt.
Yup, agreed.
Cheers,
Ben.
Quoted 4 lines · Benjamin Herrenschmidt
Ah ? What about the comment in e1000 saying that it needs a
wmb()between descriptor updates in memory and the mmio to kick them ? That would typically be amemory_to_io_wb(). Or are your MMIOs ordered cs. your cacheable stores ?
That's likely just following existing practice found in many network drivers. The following two design patterns have been copied across a great many network drivers:
1) When in a loop, reading through a DMA ring, put an "rmb()" at the top of the loop, to ensure that the compiler does not optimize out all memory loads after the first.
2) Use "wmb()" to ensure that just-written-to memory is visible to a PCI device that will be reading said memory region via DMA.
I don't claim that either of these is correct, just that's existing practice, perhaps in some case perpetuated by my own arch ignorance.
So, in a perfect world where I was designing my own API, I would create two new API functions:
prepare_to_read_dma_memory()
andmake_memory_writes_visible_to_dmaing_devices()
and leave the existing APIs untouched. Those are the two fundamental operations that are needed.
Jeff
Quoted 9 lines · Jeff Garzik
Quoted 4 lines · Benjamin Herrenschmidt
Ah ? What about the comment in e1000 saying that it needs a
wmb()between descriptor updates in memory and the mmio to kick them ? That would typically be amemory_to_io_wb(). Or are your MMIOs ordered cs. your cacheable stores ?That's likely just following existing practice found in many network drivers. The following two design patterns have been copied across a great many network drivers:
Well, I was mentioning that one specifically because this comment:
/* Force memory writes to complete before letting h/w
* know there are new descriptors to fetch. (Only
* applicable for weak-ordered memory model archs,
* such as IA-64). */
Which made me ask wether, ia64 was or was not ordering memory store followed by MMIO store, that is does ia64 -current- accessors provide rule #2 (memory W + MMIO W) currently or not and would it benefit from not having to provide it with my new partially relaxed accessors ?
Quoted 3 lines
1) When in a loop, reading through a DMA ring, put an "
rmb()" at the top of the loop, to ensure that the compiler does not optimize out all memory loads after the first.
and rmb is heavy handed for a compiler barrier :) what you might need on some platforms is an rmb between the MMIO read of whatever status/index register and the following memory reads of descriptors, and you may want an rmb in case where it matters if the chip has been changing a value behind your back (which it generally doesn't) but that's pretty much it....
Quoted 2 lines
2) Use "
wmb()" to ensure that just-written-to memory is visible to a PCI device that will be reading said memory region via DMA.
That will definitely help on PowerPC with our current accessors which are mostly ordered except for that rule #2 I mentioned above.
Quoted 2 lines
I don't claim that either of these is correct, just that's existing practice, perhaps in some case perpetuated by my own arch ignorance.
No worries :) That's also why I'm trying to describe precisely what semantics are provided by the MMIO accessors with real world examples in a way that is not arch dependant. The 4 "rules" I've listed in the first part are precisely what should be needed for drivers, then I list the accessors and what rules they are guaranteed to comply with, then I list the barriers allowing to implement those ordering rules when the accessors don't.
Quoted 9 lines
So, in a perfect world where I was designing my own API, I would create two new API functions:
prepare_to_read_dma_memory()
andmake_memory_writes_visible_to_dmaing_devices()and leave the existing APIs untouched. Those are the two fundamental operations that are needed.
Well, the argument currently is to make writel and readl imply the above barriers by making them fully ordered (and slow on some platforms) and so also provide more weakly ordered routines along with barriers for people who know what they do. The above 2 barriers are what I've called io_to_memory_rb() and memory_to_io_wb() (actually, prepare_to_read_dma_memory() by itself doesn't really make much sense. It does in conjunction with an MMIO read to flush DMA buffers, in which case the barrier provides an ordering guarantee that the memory reads will only be performed after the MMIO read has fully completed).
Ben.
Benjamin> and rmb is heavy handed for a compiler barrier :) what Benjamin> you might need on some platforms is an rmb between the Benjamin> MMIO read of whatever status/index register and the Benjamin> following memory reads of descriptors, and you may want Benjamin> an rmb in case where it matters if the chip has been Benjamin> changing a value behind your back (which it generally Benjamin> doesn't) but that's pretty much it....
In drivers/infiniband/hw/mthca/mthca_eq.c, there is:
while ((eqe = next_eqe_sw(eq))) {
/*
* Make sure we read EQ entry contents after we've
* checked the ownership bit.
*/
rmb();
switch (eqe->type) {where next_eqe_sw() checks a "valid" bit of a 32-byte event queue entry that is DMA-ed into memory by the device. The device is careful to write the valid bit (byte actually) last, but on PowerPC 970 without the rmb(), we actually saw the CPU reordering the read of eqe->type (which is another field of the EQ entry written by the device) so it happened before the entry was valid, but then executing the check of the valid bit far enough into the future so that the entry tested as valid.
This isn't that surprising: if you had two CPUs, with one CPU writing into a queue and the other CPU polling the queue, you would obviously need smp_rmb() on the CPU doing the reading. But somehow it's not quite as obvious when a device plays the role of one of the CPUs.
Of course there's no MMIO anywhere in sight here, so this isn't directly applicable I guess.
- R.
Quoted 9 lines
where
next_eqe_sw()checks a "valid" bit of a 32-byte event queue entry that is DMA-ed into memory by the device. The device is careful to write the valid bit (byte actually) last, but on PowerPC 970 without thermb(), we actually saw the CPU reordering the read ofeqe->type(which is another field of the EQ entry written by the device) so it happened before the entry was valid, but then executing the check of the valid bit far enough into the future so that the entry tested as valid.
Yes, the CPU can perfectly load it before the previous load, indeed. I'm sure that wouldn't be powerpc specific. In this case, it would be a speculative load (since there is a data dependency, thus you would think it's ok, but it's not on CPUs that do speculative execution).
Quoted 7 lines
This isn't that surprising: if you had two CPUs, with one CPU writing into a queue and the other CPU polling the queue, you would obviously need
smp_rmb()on the CPU doing the reading. But somehow it's not quite as obvious when a device plays the role of one of the CPUs.Of course there's no MMIO anywhere in sight here, so this isn't directly applicable I guess.
It's a "normal" case memory barrier in this case. Same as for SMP. Yup.
Ben.
Quoted 9 lines · Benjamin Herrenschmidt
Well, the argument currently is to make writel and readl imply the above barriers by making them fully ordered (and slow on some platforms) and so also provide more weakly ordered routines along with barriers for people who know what they do. The above 2 barriers are what I've called
io_to_memory_rb()andmemory_to_io_wb()(actually,prepare_to_read_dma_memory()by itself doesn't really make much sense. It does in conjunction with an MMIO read to flush DMA buffers, in which case the barrier provides an ordering guarantee that the memory reads will only be performed after the MMIO read has fully completed).
<jgarzik throws a monkey wrench into the works>
I think focusing on MMIO just confuses the issue.
wmb() is often used to make sure a memory store is visible to a busmastering PCI device... before the code proceeds with some more transactions in the memory space shared by the host and PCI device.
prepare_to_read_dma_memory() is the operation that an ethernet driver's RX code wants. And this is _completely_ unrelated to MMIO. It just wants to make sure that the device and host are looking at the same data. Often this involves polling a DMA descriptor (or index, stored inside DMA-able memory) looking for changes.
flush_my_writes_to_dma_memory() is the operation that an ethernet driver's TX code wants, to precede either an MMIO "poke" or any other non-MMIO operation where the driver needs to be certain that the write is visible to the PCI device, should the PCI device desire to read that area of memory.
Jeff
Quoted 4 lines
wmb()is often used to make sure a memory store is visible to a busmastering PCI device... before the code proceeds with some more transactions in the memory space shared by the host and PCI device.
Yes and that's a different issue. It's purely memory-to-memory barriers and we already have these well defined.
The problem _is_ with MMIO :) There, you have some ordering issues happening with some processors that we need to handle, hence the whole discussion. See below my discussion of your example
Quoted 5 lines
prepare_to_read_dma_memory()is the operation that an ethernet driver's RX code wants. And this is _completely_ unrelated to MMIO. It just wants to make sure that the device and host are looking at the same data. Often this involves polling a DMA descriptor (or index, stored inside DMA-able memory) looking for changes.
Why would you need a barrier other than a compiler barrier() for that ?
All you need for such operations that do not involve MMIOs are the standard wmb(), rmb() and mb() with their usual semantics and polling for something to change isn't something that requires any of these. Only a compiler barrier (or an ugly volatile maybe). Though having a subsequent read from memory that must be done after that change happened is indeed the job of rmb().
This has nothing to do with MMIO and is not what I'm describing in the document. MMIO has it's own issues especially when it comes to MMIO vs. memmory coherency. I though I described them well enough, looks like not.
Quoted 5 lines
flush_my_writes_to_dma_memory()is the operation that an ethernet driver's TX code wants, to precede either an MMIO "poke" or any other non-MMIO operation where the driver needs to be certain that the write is visible to the PCI device, should the PCI device desire to read that area of memory.
That's the problem. You need -different- type of barriers wether the subsequent operation to "poke" the device is an MMIO or an update in memory. Again, the whole problem is that on some out of order architectures, non-cacheable storage is on a completely different domain than cachaeable storage and ordering between them requires specific barriers unless you want to ditch performances.
Thus in your 2 above examples, we have:
1- Descriptor update followed by MMIO poke. That needs ordering rule #2
in my list (memory W + MMIO W), which is today not provided by the PowerPC writel(), but should be according to the discussions we had and which would be provided by the barrier memory_to_io_wb() in my list if you chose to use relaxed ordering __writel() version instead for performances.
2- Descriptor update followed by update of an index in memory (so no
MMIO involved). This is a standard memory ordering issue and thus a simple wmb() is needed there.
Currently, the PowerPC writel(), as I just said, doesn't provide ordering for your example #1, but the PowerPC wmb() does provide the semantics of both memory/memory coherency and memory/MMIO coherency (thus making it more expensive than necessary in the memory/memory case).
My goal here, is to:
- remove the problem for people who don't understand the issues by
making writel() etc... fully ordered vs. memory. for the cases that matter to drivers. Thus the -only- case that drivers writers would have to care about if using those accessors is the memory-memory case in your second example.
- provide relaxed __writel etc... for people who -do- understand those
issues and want to improve preformances of the hot path of their driver. In order to make this actually optimal and safe, I need to precisely define in what way it is relaxed, what precise ordering semantics are provided, and provide specific barriers for each of these.
That's what I documented. If you think my document is not clear enough, I would be happy to have your input on how to make it clearer. Maybe some introduction explaining the difference above ? (re-using your examples).
There are still a few questions that I listed about what we want to provide. The main one is the ordering of MMIO vs. spin_unlock. Do we want to provide that in the default writel or do we accept that we still require a barrier in that case even when using "ordered" versions of the accessors because the performance cost would be too high.
So far, I tend to prefer being fully ordered (and thus not require the barrier) but I wanted some feedback there. So far, everybody have carefuly avoided to voice an firm opinion on that one though :)
Ben.
Quoted 3 lines
prepare_to_read_dma_memory()is the operation that an ethernet driver's RX code wants. And this is _completely_ unrelated to MMIO. It just wants to make sure that the device and host are looking at the same data. Often this involves polling a DMA descriptor (or index, stored inside DMA-able memory) looking for changes.
flush_my_writes_to_dma_memory()is the operation that an ethernet driver's TX code wants, to precede either an MMIO "poke" or any other non-MMIO operation where the driver needs to be certain that the write is visible to the PCI device, should the PCI device desire to read that area of memory.
Because those are the operations, those should be the actual function names, too (well, prefixed with pci_). Architectures can implement them whatever way is appropriate, or perhaps default to some ultra-strong semantics if they prefer; driver writers should not have to know about the underlying mechanics (like why we need which barriers).
Segher
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 2 lines
These sound fine. I think PPC64 is the only platform that will need them?
Ah ? What about the comment in e1000 saying that it needs a
wmb()between descriptor updates in memory and the mmio to kick them ? That would typically be amemory_to_io_wb(). Or are your MMIOs ordered cs. your cacheable stores ?
I think that's a separate issue? As Jeff points out, those macros are intended to provide memory vs. I/O ordering, but isn't PPC the only platform that will reorder accesses so aggressively and independently? I don't think ia64 for example will reorder them separately, so a regular memory barrier *should* be enough to ensure ordering in both domains.
Quoted 3 lines
They are, but I was thinking about providing more IO-like examples. I suppose I could refer to memory-barriers.txt from here and update it with IO-like examples.
Yeah, either way. Not sure if adding more I/O examples to the existing doc is better or worse than an I/O specific document.
Quoted 11 lines
Quoted 3 lines
But isn't this how you'll implement
io_to_lock_wb()on PPC anyway? If so, might be best to name it and document it that way (though keeping the idea of barriering before unlocking prominent in the documentation).Well, the whole question is what does the linux semantics guarantee to driver writers (accross archs), not what PowerPC implements :) I'd rather not add guarantees that aren't useful to drivers even if all current implementations happen to provide them. I'm trying to find a case where ordering MMIO W + memory W is useful and I can't see any since the MMIO W will take any time to go to the device anyway. The lock rule seems to be the only useful, thus the only I think I'll guarantee.
Sure, that's fair. If any potential application of the more precise semantics is just theoretical, we may as well limit our guarantees to locks only.
Quoted 8 lines
Well, as far as I'm concerned, the whole point is rule #2 and #3 :) Those are the ones biting us on PowerPC (we haven't seen the lock problem but then it can't happen the way our current accessors are written. However, if we change our accessors to provide rule #2 more specifically, we'll end up with 2 sync instructions in writel, one for rule #2 before the store and one for rule #4, thus we go from expensive to very expensive). It's also my understanding that mmiowb is very expensive on ia64 and gets worse as the box grows bigger.
Yeah, that's true (I see your point about being more worried about other things on PPC as well ;).
Quoted 5 lines
Hence the question: do we provide -fully- ordered accessors in class 1, or do we provide -mostly- ordered accessors, ordered in all means except rule #4 vs locks. ia64 is afaik by far the platform taking the biggest hit if you have to provide #4, so I'm interesting in your point of view here.
Either way is fine with me as long as we have a way to get at the fast and loose stuff (and required barriers of course) in a portable way. And that we don't regress the existing users of mmiowb().
Quoted 5 lines
We don't need counters, just a flag. We did a test implementation, seems to work. We also clear the flag in
spin_lock. That means that MMIOs issued before a lock aren't ordered vs. the locked section. But because of rule #1, they should be ordered vs. other MMIOs inside the locked section and thus implicitely get ordered anyway.
Oh right, a flag would be enough. Is it good enough for -mm yet? Might be fun to run on an Altix machine with a bunch of supported devices (not that I work with them anymore...).
Quoted 6 lines
Quoted 3 lines
For ia64 in particular it doesn't matter, though there was speculation several years that it might be necessary. No actual examples stepped forward though, so the current implementation doesn't take an argument.
Ok. My question is wether it would improve the implementation to take it. If we define a new macro with a new name, we can do it....
Right, but unless there's a real need at this point, we probably shouldn't bother. Let the poor sucker with the future machine needing the device argument do the work. :)
Thanks,
Jesse
Quoted 6 lines
I think that's a separate issue? As Jeff points out, those macros are intended to provide memory vs. I/O ordering, but isn't PPC the only platform that will reorder accesses so aggressively and independently? I don't think ia64 for example will reorder them separately, so a regular memory barrier *should* be enough to ensure ordering in both domains.
Well, I don't know, that's what I'm asking since the comment in the driver specifically mentions IA64 :)
Quoted 9 lines
Quoted 5 lines
Hence the question: do we provide -fully- ordered accessors in class 1, or do we provide -mostly- ordered accessors, ordered in all means except rule #4 vs locks. ia64 is afaik by far the platform taking the biggest hit if you have to provide #4, so I'm interesting in your point of view here.
Either way is fine with me as long as we have a way to get at the fast and loose stuff (and required barriers of course) in a portable way. And that we don't regress the existing users of
mmiowb().
Well, existing users of mmiowb() will regress in performances if we decide that class 1 (ordered) accessors do imply rule #4 (ordering with locks) since they'll end up doing redundant mmiowb's ;) but then, they'll be affected anyway to to the sheer amount of mmiowb's (one per IO) unless you implement the trick I described, which would bring down the cost to nothing except maybe the test in spin_unlock (which I still need to measure on PowerPC).
Quoted 9 lines
Quoted 5 lines
We don't need counters, just a flag. We did a test implementation, seems to work. We also clear the flag in
spin_lock. That means that MMIOs issued before a lock aren't ordered vs. the locked section. But because of rule #1, they should be ordered vs. other MMIOs inside the locked section and thus implicitely get ordered anyway.Oh right, a flag would be enough. Is it good enough for -mm yet? Might be fun to run on an Altix machine with a bunch of supported devices (not that I work with them anymore...).
The PowerPC patch is probably good enough for 2.6.18 in fact :) I'll let Paulus post what he has. It's fairly ppc specific in the actual implementation though.
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
For ia64 in particular it doesn't matter, though there was speculation several years that it might be necessary. No actual examples stepped forward though, so the current implementation doesn't take an argument.
Ok. My question is wether it would improve the implementation to take it. If we define a new macro with a new name, we can do it....
Right, but unless there's a real need at this point, we probably shouldn't bother. Let the poor sucker with the future machine needing the device argument do the work. :)
Ok :)
Ben.