ARM: Change the mandatory barriers implementation
PATCH 1/4 ARM: Change the mandatory barriers implementation
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses dmb() for mb() and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory. If a DMA transfer is started by a write to Device memory, the data to be transfered may not reach the main memory (even if mapped as Normal Non-cacheable) before the device receives the notification to begin the transfer. The only barrier that would help in this situation is DSB which would completely drain the write buffers.
The patch also adds support for platform-defined barriers that can be defined in mach/barriers.h. This is required by at least two platforms - MSM and RealView (possible OMAP as well). On RealView with an outer cache (PL310 for example) stores to Normal Non-cacheable memory are buffered by the outer cache but the DSB doesn't go as far as this. A separate L2x0 sync command is required (a store to Strongly Ordered memory would do as well, similar to the MSM requirements and maybe faster).
Note that the SMP barriers are not affected as they only deal with ordering in Normal memory. There is however a situation with the use of IPIs. A DMB is not enough to ensure that a write to Normal memory is strictly ordered with respect to the IPI generation (and interrupt handling). A solution is for the users of smp_call_function() to use a mandatory barrier.
--- arch/arm/include/asm/system.h | 18 ++++++++---------- arch/arm/mm/Kconfig | 6 ++++++ 2 files changed, 14 insertions(+), 10 deletions(-)
arch/arm/include/asm/system.h +8 -10
@@ -138,14 +138,12 @@ extern unsigned int user_debug;#define dmb () __asm__ __volatile__ ("" : : : "memory")#endif -#if __LINUX_ARM_ARCH__ >=7 || defined(CONFIG_SMP ) -#define mb () dmb() -#define rmb () dmb() -#define wmb () dmb() +#ifdef CONFIG_ARCH_HAS_BARRIERS +#include <mach/barriers.h> #else -#define mb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define rmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define wmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) +#define mb () dsb() +#define rmb () dmb() +#define wmb () dsb()#endif #ifndef CONFIG_SMP
@@ -153,9 +151,9 @@ extern unsigned int user_debug;#define smp_rmb () barrier()#define smp_wmb () barrier()#else -#define smp_mb () mb() -#define smp_rmb () rmb() -#define smp_wmb () wmb() +#define smp_mb () dmb() +#define smp_rmb () dmb() +#define smp_wmb () dmb()#endif #define read_barrier_depends () do { } while(0)
arch/arm/mm/Kconfig +6 -0
@@ -802,3 +802,9 @@ config ARM_L1_CACHE_SHIFT int default 6 if ARCH_OMAP3 || ARCH_S5PC1XX default 5 + +config ARCH_HAS_BARRIERS + bool + help + This option allows the use of custom mandatory barriers + included via the mach/barriers.h file.
PATCH 1/4 ARM: Change the mandatory barriers implementation
arch/arm/include/asm/system.h extern unsigned int user_debug; · line 146
Quoted 6 lines · Catalin Marinas
arch/arm/include/asm/system.h +3
-3-#define mb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define rmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define wmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) +#define mb () dsb() +#define rmb () dmb() +#define wmb () dsb()
What is the reason for getting rid of the arch_is_coherent() bit here and imposing non-compiler barriers on everything? I'd assume that Lennert knew what he was doing with Xscale3.
This also breaks ARMv3 since you now issue a mcr for the write buffer on architectures which have no write buffer.
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 10 lines · Russell King - ARM Linux
arch/arm/include/asm/system.h extern unsigned int user_debug; · line 146
Quoted 6 lines · Catalin Marinas
arch/arm/include/asm/system.h +3
-3-#define mb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define rmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define wmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) +#define mb () dsb() +#define rmb () dmb() +#define wmb () dsb()What is the reason for getting rid of the
arch_is_coherent()bit here and imposing non-compiler barriers on everything?
The original code defines the mandatory barriers only if __LINUX_ARCH_ARM__ >= 7 || CONFIG_SMP. However we still need them for earlier architectures (v6). We can change the #if here to test for earlier architectures as well and only do the arch_is_coherent() check for some early implementations that don't have barriers.
Quoted 2 lines
This also breaks ARMv3 since you now issue a mcr for the write buffer on architectures which have no write buffer.
Is this the case for another #ifdef? I can add it.
Signature · 1 line
Catalin
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Russell King - ARM Linux
arch/arm/include/asm/system.h extern unsigned int user_debug; · line 146
Quoted 6 lines · Catalin Marinas
arch/arm/include/asm/system.h +3
-3-#define mb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define rmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define wmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) +#define mb () dsb() +#define rmb () dmb() +#define wmb () dsb()What is the reason for getting rid of the
arch_is_coherent()bit here and imposing non-compiler barriers on everything?
Quoted 2 lines
The original code defines the mandatory barriers only if
__LINUX_ARCH_ARM__>= 7 ||CONFIG_SMP.
No. Please read the code.
The original code defines mandatory barriers in the following cases:
- ARMv7+
- SMP
- Any architecture which sets arch_is_coherent() (for coherent DMA) - which at the moment is only Xscale 3 on IXP23xx platforms.
So, we end up with mandatory barriers for: ARMv7+, ARMv6 SMP, XScale 3. We end up with mb() and friends being compiler barriers on everything else.
However, because dsb() replaced the 'drain write buffer' instruction in things like the TLB operations, dsb() et al need to be defined for the other architectures - and TLB operations execute dsb() according to the TLB type attributes.
What your patch does is make mb() and friends unconditionally call dsb() and friends. As I've pointed out, this is totally wrong. We only need mb() and friends to be dsb() et al for ARMv6+ and anything which has arch_is_coherent() enabled.
So, all you need to do is to leave the #else clause as-is, and change:
#if __LINUX_ARM_ARCH__ >= 7 || defined(CONFIG_SMP)
to be
#if __LINUX_ARM_ARCH__ >= 6
No need for any additional ifdefs.
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 24 lines · Russell King - ARM Linux
Quoted 12 lines · Catalin Marinas
Quoted 10 lines · Russell King - ARM Linux
arch/arm/include/asm/system.h extern unsigned int user_debug; · line 146
Quoted 6 lines · Catalin Marinas
arch/arm/include/asm/system.h +3
-3-#define mb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define rmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define wmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) +#define mb () dsb() +#define rmb () dmb() +#define wmb () dsb()What is the reason for getting rid of the
arch_is_coherent()bit here and imposing non-compiler barriers on everything?Quoted 2 lines
The original code defines the mandatory barriers only if
__LINUX_ARCH_ARM__>= 7 ||CONFIG_SMP.No. Please read the code.
The original code defines mandatory barriers in the following cases:
- ARMv7+
- SMP
- Any architecture which setsarch_is_coherent()(for coherent DMA) - which at the moment is only Xscale 3 on IXP23xx platforms.
Quoted 3 lines
So, we end up with mandatory barriers for: ARMv7+, ARMv6 SMP, XScale 3. We end up with
mb()and friends being compiler barriers on everything else.
The scenario I have in mind is when using mb() in relation with DMA coherent mappings. We only use Normal Uncached for this case on ARMv7. Earlier architectures, including ARMv6 SMP, we use strongly ordered which would be fine without any barrier.
Since mb() isn't meant for SMP use, does it still make sense to have it defined in the ARMv6 SMP case? Would people not use the smp_* variants?
Or is mb() meant to cover all the cases that smp_mb() would cover on SMP?
Quoted 7 lines
So, all you need to do is to leave the #else clause as-is, and change:
#if
__LINUX_ARM_ARCH__>= 7 ||defined(CONFIG_SMP)to be
#if
__LINUX_ARM_ARCH__>= 6
Yes, sounds fine. Just need to check whether the minimum version should be 6 or 7 (see above).
Signature · 1 line
Catalin
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 7 lines · Catalin Marinas
The scenario I have in mind is when using
mb()in relation with DMA coherent mappings. We only use Normal Uncached for this case on ARMv7. Earlier architectures, including ARMv6 SMP, we use strongly ordered which would be fine without any barrier.Since
mb()isn't meant for SMP use, does it still make sense to have it defined in the ARMv6 SMP case? Would people not use the smp_* variants?
Part of the reason for that is that smp_mb() are, afaik, supposed to be as strong as mb() in the SMP case, or reduce to compiler barriers in the UP case.
I'm not entirely convinced by the part of your patch which changes the SMP barriers yet. For instance, some drivers contain:
/* We need for force the visibility of tp->intr_mask
* for other CPUs, as we can loose an MSI interrupt
* and potentially wait for a retransmit timeout if we don't.
* The posted write to IntrMask is safe, as it will
* eventually make it to the chip and we won't loose anything
* until it does.
*/
tp->intr_mask = 0xffff;
smp_wmb();
RTL_W16(IntrMask, tp->intr_event);
The second write is a write to hardware, and thus would be to a device region. The first is a write to a memory structure.
It seems to me given your description in the patch, that having smp_wmb() be a dmb(), rather than a wmb() would be insufficient here.
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 6 lines · Russell King - ARM Linux
Quoted 5 lines · Catalin Marinas
The scenario I have in mind is when using
mb()in relation with DMA coherent mappings. We only use Normal Uncached for this case on ARMv7. Earlier architectures, including ARMv6 SMP, we use strongly ordered which would be fine without any barrier.
Quoted 6 lines
Quoted 2 lines
Since
mb()isn't meant for SMP use, does it still make sense to have it defined in the ARMv6 SMP case? Would people not use the smp_* variants?Part of the reason for that is that
smp_mb()are, afaik, supposed to be as strong asmb()in the SMP case, or reduce to compiler barriers in the UP case.
I looked again at Documentation/memory-barriers.txt and I haven't seen anything that would suggest the above. The only reference to these types of barriers together seems to be:
Note that SMP memory barriers _must_ be used to control the ordering of references to shared memory on SMP systems [...].
Mandatory barriers should not be used to control SMP effects, since mandatory barriers unnecessarily impose overhead on UP systems. They may, however, be used to control MMIO effects on accesses through relaxed memory I/O windows.
They don't seem to imply that one is stronger than the other, only that they are meant for different scenarios.
Quoted 14 lines
I'm not entirely convinced by the part of your patch which changes the SMP barriers yet. For instance, some drivers contain:
/* We need for force the visibility of
tp->intr_mask
* for other CPUs, as we can loose an MSI interrupt
* and potentially wait for a retransmit timeout if we don't.
* The posted write to IntrMask is safe, as it will
* eventually make it to the chip and we won't loose anything
* until it does.
*/
tp->intr_mask=0xffff;
smp_wmb();
RTL_W16(IntrMask, tp->intr_event);
Quoted 5 lines
The second write is a write to hardware, and thus would be to a device region. The first is a write to a memory structure.
It seems to me given your description in the patch, that having
smp_wmb()be admb(), rather than awmb()would be insufficient here.
Yes, a DMB is insufficient here (see the Mailbox example in the Barrier Litmus document from Richard G). That's actually the case with the GIC currently - CPU0 writes some data, barrier and then IPI to CPU1. If only DMB is used, the IPI may arrive at CPU1 before the data written by CPU0 reaches the memory.
My proposal for this would be to place an explicit DSB at the beginning of gic_raise_irq(). Otherwise, we can change smp_wmb() to be a DSB but we may have some performance penalty for other cases where ordering with Device accesses is not required.
Signature · 1 line
Catalin
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 17 lines · Catalin Marinas
Quoted 17 lines
I'm not entirely convinced by the part of your patch which changes the SMP barriers yet. For instance, some drivers contain:
/* We need for force the visibility of
tp->intr_mask
* for other CPUs, as we can loose an MSI interrupt
* and potentially wait for a retransmit timeout if we don't.
* The posted write to IntrMask is safe, as it will
* eventually make it to the chip and we won't loose anything
* until it does.
*/
tp->intr_mask=0xffff;
smp_wmb();
RTL_W16(IntrMask, tp->intr_event);The second write is a write to hardware, and thus would be to a device region. The first is a write to a memory structure.
Quoted 7 lines
Quoted 2 lines
It seems to me given your description in the patch, that having
smp_wmb()be admb(), rather than awmb()would be insufficient here.My proposal for this would be to place an explicit DSB at the beginning of
gic_raise_irq(). Otherwise, we can changesmp_wmb()to be a DSB but we may have some performance penalty for other cases where ordering with Device accesses is not required.
That doesn't solve the above case; this isn't GIC code, it's driver code.
Given what you've said, it would appear that smp_wmb() needs to be a wmb() in the SMP case, to ensure that the write to intr_mask is visible to other CPUs before the interrupt mask write hits the peripheral.
So, that leads us back to the:
#ifndef CONFIG_SMP
#define smp_mb() barrier() #define smp_rmb() barrier() #define smp_wmb() barrier() #else #define smp_mb() mb() #define smp_rmb() rmb() #define smp_wmb() wmb()
#endif
which we have at the moment.
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 21 lines · Russell King - ARM Linux
Quoted 20 lines · Catalin Marinas
Quoted 19 lines
I'm not entirely convinced by the part of your patch which changes the SMP barriers yet. For instance, some drivers contain:
/* We need for force the visibility of
tp->intr_mask
* for other CPUs, as we can loose an MSI interrupt
* and potentially wait for a retransmit timeout if we don't.
* The posted write to IntrMask is safe, as it will
* eventually make it to the chip and we won't loose anything
* until it does.
*/
tp->intr_mask=0xffff;
smp_wmb();
RTL_W16(IntrMask, tp->intr_event);The second write is a write to hardware, and thus would be to a device region. The first is a write to a memory structure.
It seems to me given your description in the patch, that having
smp_wmb()be admb(), rather than awmb()would be insufficient here.
Quoted 6 lines
Quoted 4 lines
My proposal for this would be to place an explicit DSB at the beginning of
gic_raise_irq(). Otherwise, we can changesmp_wmb()to be a DSB but we may have some performance penalty for other cases where ordering with Device accesses is not required.That doesn't solve the above case; this isn't GIC code, it's driver code.
As I mentioned in the previous e-mail, my view is that the driver could just use wmb() rather than smp_wmb() in this case. Are the smp_*() barriers in Linux required to ensure the relative ordering of anything other than shared normal memory accesses?
Signature · 1 line
Catalin
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 27 lines · Catalin Marinas
Quoted 26 lines · Russell King - ARM Linux
Quoted 24 lines · Catalin Marinas
Quoted 14 lines
I'm not entirely convinced by the part of your patch which changes the SMP barriers yet. For instance, some drivers contain:
/* We need for force the visibility of
tp->intr_mask
* for other CPUs, as we can loose an MSI interrupt
* and potentially wait for a retransmit timeout if we don't.
* The posted write to IntrMask is safe, as it will
* eventually make it to the chip and we won't loose anything
* until it does.
*/
tp->intr_mask=0xffff;
smp_wmb();
RTL_W16(IntrMask, tp->intr_event);Quoted 5 lines
The second write is a write to hardware, and thus would be to a device region. The first is a write to a memory structure.
It seems to me given your description in the patch, that having
smp_wmb()be admb(), rather than awmb()would be insufficient here.My proposal for this would be to place an explicit DSB at the beginning of
gic_raise_irq(). Otherwise, we can changesmp_wmb()to be a DSB but we may have some performance penalty for other cases where ordering with Device accesses is not required.
Quoted 6 lines
Quoted 1 line
That doesn't solve the above case; this isn't GIC code, it's driver code.
As I mentioned in the previous e-mail, my view is that the driver could just use
wmb()rather thansmp_wmb()in this case. Are the smp_*() barriers in Linux required to ensure the relative ordering of anything other than shared normal memory accesses?
Historically, smp_*() was just this:
#ifdef SMP
#define smp_mb() mb()
#else
#define smb_mb() barrier()
#endif
The idea was to write smp_*() everywhere that you know you need a barrier iff CONFIG_SMP.
So at one time, the driver code might have been fine, assuming it's comment is correct.
It's only more recently that SMP + smp_*() became weaker than *() on some architectures.
Which might mean the above driver code is now buggy.
smp_*() are not required to order regular memory vs. device MMIO, or vs. interrupts, or vs. DMA, because on UP systems, they are usually defined as barrier() which guarantees nothing for MMIO.
-- Jamie
PATCH 1/4 ARM: Change the mandatory barriers implementation
In reply to Russell King - ARM LinuxQuoted 20 lines · Russell King - ARM Linux
Quoted 19 lines · Catalin Marinas
Quoted 19 lines
I'm not entirely convinced by the part of your patch which changes the SMP barriers yet. For instance, some drivers contain:
/* We need for force the visibility of
tp->intr_mask
* for other CPUs, as we can loose an MSI interrupt
* and potentially wait for a retransmit timeout if we don't.
* The posted write to IntrMask is safe, as it will
* eventually make it to the chip and we won't loose anything
* until it does.
*/
tp->intr_mask=0xffff;
smp_wmb();
RTL_W16(IntrMask, tp->intr_event);The second write is a write to hardware, and thus would be to a device region. The first is a write to a memory structure.
It seems to me given your description in the patch, that having
smp_wmb()be admb(), rather than awmb()would be insufficient here.
[...]
Quoted 8 lines
Given what you've said, it would appear that
smp_wmb()needs to be awmb()in the SMP case, to ensure that the write tointr_maskis visible to other CPUs before the interrupt mask write hits the peripheral.So, that leads us back to the:
#ifndef
CONFIG_SMP
Quoted 8 lines
#define smp_mb() barrier() #define smp_rmb() barrier() #define smp_wmb() barrier() #else #define smp_mb() mb() #define smp_rmb() rmb() #define smp_wmb() wmb()#endif
A better implementation would be this:
#ifndef CONFIG_SMP
#define smp_mb() barrier()
#define smp_rmb() barrier()
#define smp_wmb() barrier()
#else
#define smp_mb() dsb()
#define smp_rmb() mb()
#define smp_wmb() dsb()
#endif
Since the mb() may have other effects like draining the L2 write buffer which is definitely not needed for the SMP barriers.
Anyway, the above change to smp_*mb() would probably have a performance impact especially with spinlocks.
I can see that the driver situation you described appears in other drivers as well. Whether this is a correct usage model I can't tell. It may be worth going with this on linux-arch. PowerPC for example uses a light barrier for the smp_wmb() case which doesn't ensure ordering between accesses to normal vs I/O memory.
Signature · 1 line
Catalin
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 15 lines · Catalin Marinas
A better implementation would be this:
#ifndef
CONFIG_SMP
#definesmp_mb()barrier()
#definesmp_rmb()barrier()
#definesmp_wmb()barrier()
#else
#definesmp_mb()dsb()
#definesmp_rmb()mb()
#definesmp_wmb()dsb()
#endifSince the
mb()may have other effects like draining the L2 write buffer which is definitely not needed for the SMP barriers.
Quoted 8 lines
Anyway, the above change to smp_*mb() would probably have a performance impact especially with spinlocks.
I can see that the driver situation you described appears in other drivers as well. Whether this is a correct usage model I can't tell. It may be worth going with this on linux-arch. PowerPC for example uses a light barrier for the
smp_wmb()case which doesn't ensure ordering between accesses to normal vs I/O memory.
I agree, it looks like some confusion about the meaning of smp_wmb() for ordering w.r.t. I/O, DMA and interrupts has crept in. It would be good to clarify it.
-- Jamie
PATCH 1/4 ARM: Change the mandatory barriers implementation
In reply to Russell King - ARM LinuxQuoted 12 lines · Russell King - ARM Linux
Review
Quoted 6 lines · Catalin Marinas
(quoted diff) +3
-3-#define mb() do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define rmb() do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define wmb() do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) +#define mb() dsb() +#define rmb() dmb() +#define wmb() dsb()What is the reason for getting rid of the
arch_is_coherent()bit here and imposing non-compiler barriers on everything? I'd assume that Lennert knew what he was doing with Xscale3.
Quoted 2 lines
This also breaks ARMv3 since you now issue a mcr for the write buffer on architectures which have no write buffer.
What about the patch below?
ARM: Change the mandatory barriers implementation
From: Catalin Marinas <[email protected]>
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses dmb() for mb() and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory. If a DMA transfer is started by a write to Device memory, the data to be transfered may not reach the main memory (even if mapped as Normal Non-cacheable) before the device receives the notification to begin the transfer. The only barrier that would help in this situation is DSB which would completely drain the write buffers.
The patch also adds support for platform-defined barriers that can be defined in mach/barriers.h. This is required by at least two platforms - MSM and RealView (possible OMAP as well). On RealView with an outer cache (PL310 for example) stores to Normal Non-cacheable memory are buffered by the outer cache but the DSB doesn't go as far as this. A separate L2x0 sync command is required (a store to Strongly Ordered memory would do as well, similar to the MSM requirements and maybe faster).
Note that the SMP barriers are not affected as they only deal with ordering in Normal memory. There is however a situation with the use of IPIs. A DMB is not enough to ensure that a write to Normal memory is strictly ordered with respect to the IPI generation (and interrupt handling). A solution is for the users of smp_call_function() to use a mandatory barrier.
--- arch/arm/include/asm/system.h | 14 ++++++++------ arch/arm/mm/Kconfig | 6 ++++++ 2 files changed, 14 insertions(+), 6 deletions(-)
arch/arm/include/asm/system.h +8 -6
@@ -138,10 +138,12 @@ extern unsigned int user_debug;#define dmb () __asm__ __volatile__ ("" : : : "memory")#endif -#if __LINUX_ARM_ARCH__ >=7 || defined(CONFIG_SMP ) -#define mb () dmb() +#ifdef CONFIG_ARCH_HAS_BARRIERS +#include <mach/barriers.h> +#elif __LINUX_ARM_ARCH__ >=5 +#define mb () dsb()#define rmb () dmb() -#define wmb () dmb() +#define wmb () dsb()#else #define mb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0)#define rmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0)
@@ -153,9 +155,9 @@ extern unsigned int user_debug;#define smp_rmb () barrier()#define smp_wmb () barrier()#else -#define smp_mb () mb() -#define smp_rmb () rmb() -#define smp_wmb () wmb() +#define smp_mb () dmb() +#define smp_rmb () dmb() +#define smp_wmb () dmb()#endif #define read_barrier_depends () do { } while(0)
arch/arm/mm/Kconfig +6 -0
@@ -802,3 +802,9 @@ config ARM_L1_CACHE_SHIFT int default 6 if ARCH_OMAP3 || ARCH_S5PC1XX default 5 + +config ARCH_HAS_BARRIERS + bool + help + This option allows the use of custom mandatory barriers + included via the mach/barriers.h file.
Signature · 1 line
Catalin
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 7 lines · Catalin Marinas
arch/arm/include/asm/system.h +0
-0@@ -138,10 +138,12 @@ extern unsigned int user_debug;#define dmb () __asm__ __volatile__ ("" : : : "memory")#endif
arch/arm/include/asm/system.h extern unsigned int user_debug; · line 146
Quoted 10 lines
arch/arm/include/asm/system.h +5
-3-#if __LINUX_ARM_ARCH__ >=7 || defined(CONFIG_SMP ) -#define mb () dmb() +#ifdef CONFIG_ARCH_HAS_BARRIERS +#include <mach/barriers.h> +#elif __LINUX_ARM_ARCH__ >=5 +#define mb () dsb()#define rmb () dmb() -#define wmb () dmb() +#define wmb () dsb()
Why do we need these to be anything more than compiler barriers on PXA?
PATCH 1/4 ARM: Change the mandatory barriers implementation
In reply to Catalin MarinasCatalin,
Thanks for these patches.
Quoted 88 lines
-----Original Message-----
From: linux-arm-kernel-bounces at lists.infradead.org [mailto:linux-arm-kernel- bounces at lists.infradead.org] On Behalf Of Catalin Marinas
Sent: Tuesday, February 23, 2010 4:31 PM
To: linux-arm-kernel at lists.infradead.org
Cc: Tony Lindgren; Larry Bassel; Daniel Walker; Russell King
Subject: [PATCH 1/4] ARM: Change the mandatory barriers implementationThe mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses
dmb()formb()and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory. If a DMA transfer is started by a write to Device memory, the data to be transfered may not reach the main memory (even if mapped as Normal Non-cacheable) before the device receives the notification to begin the transfer. The only barrier that would help in this situation is DSB which would completely drain the write buffers.The patch also adds support for platform-defined barriers that can be defined in
mach/barriers.h. This is required by at least two platforms - MSM and RealView (possible OMAP as well). On RealView with an outer cache (PL310 for example) stores to Normal Non-cacheable memory are buffered by the outer cache but the DSB doesn't go as far as this. A separate L2x0 sync command is required (a store to Strongly Ordered memory would do as well, similar to the MSM requirements and maybe faster).Note that the SMP barriers are not affected as they only deal with ordering in Normal memory. There is however a situation with the use of IPIs. A DMB is not enough to ensure that a write to Normal memory is strictly ordered with respect to the IPI generation (and interrupt handling). A solution is for the users of
smp_call_function()to use a mandatory barrier.Signed-off-by Catalin Marinas <[email protected]>Cc Russell King <[email protected]>Cc Daniel Walker <[email protected]>Cc Larry Bassel <[email protected]>Cc Tony Lindgren <[email protected]>--- arch/arm/include/asm/system.h | 18 ++++++++---------- arch/arm/mm/Kconfig | 6 ++++++ 2 files changed, 14 insertions(+), 10 deletions(-)arch/arm/include/asm/system.h +8
-10@@ -138,14 +138,12 @@ extern unsigned int user_debug;#define dmb () __asm__ __volatile__ ("" : : : "memory")#endif -#if __LINUX_ARM_ARCH__ >=7 || defined(CONFIG_SMP ) -#define mb () dmb() -#define rmb () dmb() -#define wmb () dmb() +#ifdef CONFIG_ARCH_HAS_BARRIERS +#include <mach/barriers.h> #else -#define mb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define rmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) -#define wmb () do { if (arch_is_coherent()) dmb(); else barrier(); } while (0) +#define mb () dsb() +#define rmb () dmb() +#define wmb () dsb()#endif #ifndef CONFIG_SMP @@ -153,9 +151,9 @@ extern unsigned int user_debug;#define smp_rmb () barrier()#define smp_wmb () barrier()#else -#define smp_mb () mb() -#define smp_rmb () rmb() -#define smp_wmb () wmb() +#define smp_mb () dmb() +#define smp_rmb () dmb() +#define smp_wmb () dmb()#endif #define read_barrier_depends () do { } while(0)arch/arm/mm/Kconfig +0
-0
arch/arm/mm/Kconfig config ARM_L1_CACHE_SHIFT · line 810
Quoted 9 lines
arch/arm/mm/Kconfig +6
-0@@ -802,3 +802,9 @@ config ARM_L1_CACHE_SHIFT int default 6 if ARCH_OMAP3 || ARCH_S5PC1XX default 5 + +config ARCH_HAS_BARRIERS + bool + help + This option allows the use of custom mandatory barriers + included via the mach/barriers.h file.
OMAP3 and family would be ok with this much needed mandatory barriers but for OMAP4, we need platform specified barriers as you mentioned. Will have a look at this series and do some testing on OMAP4. Will send a follow up patch then.
Regards,
Santosh
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 4 lines · Santosh
OMAP3 and family would be ok with this much needed mandatory barriers but for OMAP4, we need platform specified barriers as you mentioned. Will have a look at this series and do some testing on OMAP4. Will send a follow up patch then.
Wait for v2 - it introduces barriers onto the older CPUs which do not need it, and therefore is definitely not right.
Eg, ARMv4 CPUs don't need the drain-write-buffer instruction for mb() and wmb() - we have no relaxed behaviours on those CPUs, so this is pointless. As I also pointed out, it breaks ARMv3 because the drain- write-buffer instruction there is an undefined instruction.
Plus, non-coherent Xscale has the same requirements as ARMv4, so getting rid of the arch_is_coherent() stuff was completely wrong.
Basically, the patch is not suitable as it currently stands, and is in need of rework.
PATCH 1/4 ARM: Change the mandatory barriers implementation
In reply to Catalin MarinasQuoted 6 lines · Catalin Marinas
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses
dmb()formb()and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory.
Erm, I also don't think your statement here is right. DMB is defined in the ARM ARM to be by default a full-system, read/write memory barrier. It has this property:
If the required shareability is Full system then the operation applies to all observers within the system.
...
Any observer with the same required shareability domain as Pe observes all members of Group A before it observes any member of Group B to the extent that those group members are required to be observed, as determined by the shareability and cacheability of the memory locations accessed by the group members. Where members of Group A and Group B access the same memory-mapped peripheral, all members of Group A will be visible at the memory-mapped peripheral before any members of Group B are visible at that peripheral.
This most definitely is not "single observer" - it's all observers within the same "shareability domain". That may encompass all CPUs and not devices and DMA agents.
I'd go further - in the case of:
write to non-cacheable memory
dmb();
write to peripheral
then, because the dmb() is a full system dmb, all observers within the system should see the write to non-cacheable memory before the write to peripheral.
Either that or the ARM ARM is unclear/wrong about what "all observers" means.
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 24 lines · Russell King - ARM Linux
Quoted 6 lines · Catalin Marinas
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses
dmb()formb()and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory.Erm, I also don't think your statement here is right. DMB is defined in the ARM ARM to be by default a full-system, read/write memory barrier. It has this property:
If the required shareability is Full system then the operation applies to all observers within the system.
...
Any observer with the same required shareability domain as Pe observes all members of Group A before it observes any member of Group B to the extent that those group members are required to be observed, as determined by the shareability and cacheability of the memory locations accessed by the group members. Where members of Group A and Group B access the same memory-mapped peripheral, all members of Group A will be visible at the memory-mapped peripheral before any members of Group B are visible at that peripheral.
Quoted 3 lines
This most definitely is not "single observer" - it's all observers within the same "shareability domain". That may encompass all CPUs and not devices and DMA agents.
Yes, that's correct but see below (the issue is the definition of "observability").
Quoted 6 lines
I'd go further - in the case of:
write to non-cacheable memory
dmb();
write to peripheral
Quoted 6 lines
then, because the
dmb()is a full system dmb, all observers within the system should see the write to non-cacheable memory before the write to peripheral.Either that or the ARM ARM is unclear/wrong about what "all observers" means.
The ARM ARM wasn't clear to me either but the answer I got from Richard G on this issue is that the "write to peripheral" above doesn't count as "observed" since the "observability" definition only applies to master accesses. In the above case, the CPU writes to a slave port of the peripheral and the DMB has no effect on when such event would be observed (if the device is mapped as strongly ordered, there are some stricter requirements). You could ask RG directly, he would probably be happy to clarify this for you.
Linux doesn't have a barrier defined to flush the write buffer, even though the Documentation/DMA-API.txt mention that the user of coherent DMA mappings may need to do this. Since many drivers use wmb/mb for this situation, I find this barrier as the best place to put a DSB.
IMHO, mb() has at least the smp_mb() semantics and it can be safely used in the IPI case you mentioned instead of the smp_* one (when implemented as DSB).
Signature · 1 line
Catalin
PATCH 1/4 ARM: Change the mandatory barriers implementation
Quoted 25 lines · Catalin Marinas
Quoted 24 lines · Russell King - ARM Linux
Quoted 6 lines · Catalin Marinas
The mandatory barriers (mb, rmb, wmb) are used even on uniprocessor systems for things like ordering Normal Non-cacheable memory accesses with DMA transfer (via Device memory writes). The current implementation uses
dmb()formb()and friends but this is not sufficient. The DMB only ensures the ordering of accesses with regards to a single observer accessing the same memory.Erm, I also don't think your statement here is right. DMB is defined in the ARM ARM to be by default a full-system, read/write memory barrier. It has this property:
If the required shareability is Full system then the operation applies to all observers within the system.
...
Any observer with the same required shareability domain as Pe observes all members of Group A before it observes any member of Group B to the extent that those group members are required to be observed, as determined by the shareability and cacheability of the memory locations accessed by the group members. Where members of Group A and Group B access the same memory-mapped peripheral, all members of Group A will be visible at the memory-mapped peripheral before any members of Group B are visible at that peripheral.
Quoted 6 lines
Quoted 3 lines
This most definitely is not "single observer" - it's all observers within the same "shareability domain". That may encompass all CPUs and not devices and DMA agents.
Yes, that's correct but see below (the issue is the definition of "observability").
I meant that the definitions in the ARM ARM encompass all CPUs, devices/DMA agents (but only when acting as masters).
Signature · 1 line
Catalin