ARM: Change the mandatory barriers implementation
RFC PATCH 1/1 ARM: Change the mandatory barriers implementation
(I cc'ed LKML as well just in case I got the wrong semantics of the mandatory barriers)
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses dmb() for mb() and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory. If a DMA transfer is started by a write to Device memory, the data to be transfered may not reach the main memory (even if mapped as Normal Non-cacheable) before the device receives the notification to begin the transfer. The only barrier that would help in this situation is DSB which would completely drain the write buffers.
The patch also adds support for platform-defined barriers that can be defined in mach/barriers.h. This is required by at least two platforms - MSM and RealView (possible OMAP as well). On RealView with an outer cache (PL310 for example) stores to Normal Non-cacheable memory are buffered by the outer cache but the DSB doesn't go as far as this. A separate L2x0 sync command is required (a store to Strongly Ordered memory would do as well, similar to the MSM requirements and maybe faster).
Note that the SMP barriers are not affected as they only deal with ordering in Normal memory. There is however a situation with the use of IPIs. A DMB is not enough to ensure that a write to Normal memory is strictly ordered with respect to the IPI generation (and interrupt handling). A solution is for the users of smp_call_function() to use a mandatory barrier.
--- arch/arm/include/asm/system.h | 18 ++++++++---------- arch/arm/mm/Kconfig | 6 ++++++ 2 files changed, 14 insertions(+), 10 deletions(-)
arch/arm/include/asm/system.h +8 -10
@@ -138,14 +138,12 @@ extern unsigned int user_debug;#define dmb () __asm__ __volatile__ ("" : : : "memory")#endif -#if __LINUX_ARM_ARCH__ >=7 || defined(CONFIG_SMP ) -#define mb () dmb() -#define rmb () dmb() -#define wmb () dmb() +#ifdef CONFIG_ARCH_HAS_BARRIERS +#include <mach/barriers.h> #else -#define mb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define rmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define wmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) +#define mb () dsb() +#define rmb () dmb() +#define wmb () dsb()#endif #ifndef CONFIG_SMP
@@ -153,9 +151,9 @@ extern unsigned int user_debug;#define smp_rmb () barrier()#define smp_wmb () barrier()#else -#define smp_mb () mb() -#define smp_rmb () rmb() -#define smp_wmb () wmb() +#define smp_mb () dmb() +#define smp_rmb () dmb() +#define smp_wmb () dmb()#endif #define read_barrier_depends () do { } while(0)
arch/arm/mm/Kconfig +6 -0
@@ -802,3 +802,9 @@ config ARM_L1_CACHE_SHIFT int default 6 if ARCH_OMAP3 || ARCH_S5PC1XX default 5 + +config ARCH_HAS_BARRIERS + bool + help + This option allows the use of custom mandatory barriers + included via the mach/barriers.h file.
Quoted 10 lines
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses
dmb()formb()and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory. If a DMA transfer is started by a write to Device memory, the data to be transfered may not reach the main memory (even if mapped as Normal Non-cacheable) before the device receives the notification to begin the transfer. The only barrier that would help in this situation is DSB which would completely drain the write buffers.
On ARMv7, DMB guarantees that all accesses prior to DMB are observed by an observer if that observer sees any accesses _after_ the DMB. In this case, since DMA engine observes a write to itself( It is being written to and hence must observe the write) it should also see the writes to the buffers. A dmb() after the writes to buffer and before write to DMA engine should suffice.
Moreover an mb() could be in places where accesses to ARM's Device type memory need ordering and are 1kb apart. Such usages of mb() would result in a dsb() and could cause performance problems.
Since you mention the write buffers this probably applies only to ARMv6. Correct me here, I think that dmb on ARMv6 should suffice too.
Quoted 20 lines
-----Original Message-----
From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel- bounces at lists.infradead.org] On Behalf Of Abhijeet Dharmapurikar
Sent: Thursday, February 04, 2010 5:52 AM
To: Catalin Marinas
Cc: Daniel Walker; Russell King; Tony Lindgren; linux-arm-msm at vger.kernel.org; linux- kernel at vger.kernel.org; Larry Bassel; linux-arm-kernel at lists.infradead.org
Subject: Re: [RFC PATCH] ARM: Change the mandatory barriers implementationQuoted 10 lines
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses
dmb()formb()and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory. If a DMA transfer is started by a write to Device memory, the data to be transfered may not reach the main memory (even if mapped as Normal Non-cacheable) before the device receives the notification to begin the transfer. The only barrier that would help in this situation is DSB which would completely drain the write buffers.
Quoted 6 lines
On ARMv7, DMB guarantees that all accesses prior to DMB are observed by an observer if that observer sees any accesses _after_ the DMB. In this case, since DMA engine observes a write to itself( It is being written to and hence must observe the write) it should also see the writes to the buffers. A
dmb()after the writes to buffer and before write to DMA engine should suffice.
This may be true if the DMA engine is an observer in the cluster. Without coherence hardware, DMA won't see the correct data without draining the write buffer. So DSB is necessary in DMA case at least.
Regards,
Santosh
Quoted 11 lines · Abhijeet Dharmapurikar
Quoted 10 lines
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses
dmb()formb()and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory. If a DMA transfer is started by a write to Device memory, the data to be transfered may not reach the main memory (even if mapped as Normal Non-cacheable) before the device receives the notification to begin the transfer. The only barrier that would help in this situation is DSB which would completely drain the write buffers.
Quoted 6 lines
On ARMv7, DMB guarantees that all accesses prior to DMB are observed by an observer if that observer sees any accesses _after_ the DMB. In this case, since DMA engine observes a write to itself( It is being written to and hence must observe the write) it should also see the writes to the buffers. A
dmb()after the writes to buffer and before write to DMA engine should suffice.
I asked our processor architect for a clarification on the wording of the DMB definition but the "all accesses" part most likely refer to accesses to the same peripheral or memory block (but not together). Intuitively, you can have a hardware configuration as below:
CPU Device
| | |
+-----+ | (1)
| |
Buffer |
| |
+---+---+ (2)
|
RAMThe peripheral register write and memory write go on different paths. A DMB may ensure the ordering at level (1) but there could be delays before a write reaches the RAM and the peripheral would get the DMA start notification before that. Only DSB would ensure the draining of the buffer.
Quoted 3 lines
Moreover an
mb()could be in places where accesses to ARM's Device type memory need ordering and are 1kb apart. Such usages ofmb()would result in adsb()and could cause performance problems.
Note that accesses to Device memory are ordered relative to each-other without any barrier. If you have weakly ordered I/O (not the ARM case), there's mmiowb() for this.
If you need ordering between accesses to Normal memory and Device memory, a DSB is needed, hence the definition of mb() to be a DSB (some processors like Cortex-A8 implement DMB so that it drains the write buffer but this is not always the case on other implementations).
Of course, there are situations when you only need ordering of Normal memory accesses without any peripheral access and a DMB would be fine in this situation. But so far Linux uses mb() for both situations, hence I'm taking the less optimal approach for Normal-Normal ordering.
Quoted 2 lines
Since you mention the write buffers this probably applies only to ARMv6. Correct me here, I think that dmb on ARMv6 should suffice too.
I can't guarantee. It depends on the processor implementation (ARM11MPCore may have a different behaviour). Linux on ARM should pretty much be architecturally generic.
Signature · 1 line
Catalin
Quoted 34 lines · Catalin Marinas
Quoted 11 lines · Abhijeet Dharmapurikar
Quoted 10 lines
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses
dmb()formb()and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory. If a DMA transfer is started by a write to Device memory, the data to be transfered may not reach the main memory (even if mapped as Normal Non-cacheable) before the device receives the notification to begin the transfer. The only barrier that would help in this situation is DSB which would completely drain the write buffers.Quoted 6 lines
On ARMv7, DMB guarantees that all accesses prior to DMB are observed by an observer if that observer sees any accesses _after_ the DMB. In this case, since DMA engine observes a write to itself( It is being written to and hence must observe the write) it should also see the writes to the buffers. A
dmb()after the writes to buffer and before write to DMA engine should suffice.I asked our processor architect for a clarification on the wording of the DMB definition but the "all accesses" part most likely refer to accesses to the same peripheral or memory block (but not together). Intuitively, you can have a hardware configuration as below:
CPU Device | | | +-----+ | (1) | | Buffer | | | +---+---+ (2) | RAM
Quoted 5 lines
The peripheral register write and memory write go on different paths. A DMB may ensure the ordering at level (1) but there could be delays before a write reaches the RAM and the peripheral would get the DMA start notification before that. Only DSB would ensure the draining of the buffer.
Additionally if there's an external bus at (1), the ARM ordering may does not guarantee that bus is done.
For example, on omap L3/L4 buses we need to do a readback from the same device on the bus to ensure the write got to the device. Otherwise things like spurious interrupts can happen as the device has not yet acked the interrupt while ARM thinks the handler is done.
Quoted 13 lines
Quoted 3 lines
Moreover an
mb()could be in places where accesses to ARM's Device type memory need ordering and are 1kb apart. Such usages ofmb()would result in adsb()and could cause performance problems.Note that accesses to Device memory are ordered relative to each-other without any barrier. If you have weakly ordered I/O (not the ARM case), there's
mmiowb()for this.If you need ordering between accesses to Normal memory and Device memory, a DSB is needed, hence the definition of
mb()to be a DSB (some processors like Cortex-A8 implement DMB so that it drains the write buffer but this is not always the case on other implementations).
Quoted 4 lines
Of course, there are situations when you only need ordering of Normal memory accesses without any peripheral access and a DMB would be fine in this situation. But so far Linux uses
mb()for both situations, hence I'm taking the less optimal approach for Normal-Normal ordering.
Yeah. The device access is ordered relative to each-other, but since the device writes may directly affect ARM (for example IRQ status), a readback from the device is the only way to guarantee ordering for an external bus.
In most cases only the ordering of instructions matters and there the barriers work just fine. Just FYI, in case others are experiencing similar issues.
Quoted 6 lines
Quoted 2 lines
Since you mention the write buffers this probably applies only to ARMv6. Correct me here, I think that dmb on ARMv6 should suffice too.
I can't guarantee. It depends on the processor implementation (ARM11MPCore may have a different behaviour). Linux on ARM should pretty much be architecturally generic.
Also, my experience is that what I describe above was rare on v6 omaps (or some different problem), but was happening often on v7 omaps.
Regards,
Tony
Quoted 19 lines · Catalin Marinas
Quoted 11 lines · Abhijeet Dharmapurikar
Quoted 10 lines
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses
dmb()formb()and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory. If a DMA transfer is started by a write to Device memory, the data to be transfered may not reach the main memory (even if mapped as Normal Non-cacheable) before the device receives the notification to begin the transfer. The only barrier that would help in this situation is DSB which would completely drain the write buffers.Quoted 6 lines
On ARMv7, DMB guarantees that all accesses prior to DMB are observed by an observer if that observer sees any accesses _after_ the DMB. In this case, since DMA engine observes a write to itself( It is being written to and hence must observe the write) it should also see the writes to the buffers. A
dmb()after the writes to buffer and before write to DMA engine should suffice.
Quoted 3 lines
I asked our processor architect for a clarification on the wording of the DMB definition but the "all accesses" part most likely refer to accesses to the same peripheral or memory block (but not together).
I got some clarification and there is nothing wrong with the definition of the DMB. The catch here is that "observe" (as per the ARM ARM) is defined only for master accesses. The DMA engine in this case above does not "observe" the write to itself as this is a slave access to one of its memory-mapped ports.
So, the code below:
STR [Normal non-cacheable]
DMB
STR [Device]
puts the first store in Group A (according to the DMB definition) and the second store in Group B but since the DMA device does not "observe" Group B in this case, there is no requirement for the ordering between the observability of the store to normal memory and the observability of the side-effects of the store to the DMA device.
Signature · 1 line
Catalin
Quoted 34 lines · Catalin Marinas
Quoted 17 lines · Catalin Marinas
Quoted 10 lines · Abhijeet Dharmapurikar
Quoted 10 lines
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses
dmb()formb()and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory. If a DMA transfer is started by a write to Device memory, the data to be transfered may not reach the main memory (even if mapped as Normal Non-cacheable) before the device receives the notification to begin the transfer. The only barrier that would help in this situation is DSB which would completely drain the write buffers.Quoted 6 lines
On ARMv7, DMB guarantees that all accesses prior to DMB are observed by an observer if that observer sees any accesses _after_ the DMB. In this case, since DMA engine observes a write to itself( It is being written to and hence must observe the write) it should also see the writes to the buffers. A
dmb()after the writes to buffer and before write to DMA engine should suffice.Quoted 3 lines
I asked our processor architect for a clarification on the wording of the DMB definition but the "all accesses" part most likely refer to accesses to the same peripheral or memory block (but not together).
I got some clarification and there is nothing wrong with the definition of the DMB. The catch here is that "observe" (as per the ARM ARM) is defined only for master accesses. The DMA engine in this case above does not "observe" the write to itself as this is a slave access to one of its memory-mapped ports.
So, the code below:
STR [Normal non-cacheable]
DMB
STR [Device]
Quoted 5 lines
puts the first store in Group A (according to the DMB definition) and the second store in Group B but since the DMA device does not "observe" Group B in this case, there is no requirement for the ordering between the observability of the store to normal memory and the observability of the side-effects of the store to the DMA device.
I agree now. DSB would be the right thing to do. Moreover the option to have a barrier.h for each platform gives them a chance to fine tune these barriers further.