b.L: Memory barriers and miscellaneous tidyups

For architectural correctness even Strongly-Ordered memory accesses require barriers in order to guarantee that multiple CPUs have a coherent view of the ordering of memory accesses.

Virtually everything done by this early code is done via explicit memory access only, so DSBs are seldom required. Existing barriers are demoted to DMB, except where a DSB is needed to synchronise non-memory signalling (i.e., before a SEV). If a particular platform performs cache maintenance in its power_up_setup function, it should force it to complete explicitly including a DSB, instead of relying on the bL_head framework code to do it.

Some additional DMBs are added to ensure all the memory ordering properties required by the race avoidance algorithm. DMBs are also moved out of loops, and for clarity some are moved so that most directly follow the memory operation which needs to be synchronised.

The setting of a CPU's bL_entry_vectors[] entry is also required to act as a synchronisation point, so a DMB is added after checking that entry to ensure that other CPUs do not observe gated operations leaking across the opening of the gate.

Signed-off-by Dave Martin <[email protected]>

---

+ Catalin, RMK

Dave,

Quoted 23 lines · Dave Martin

For architectural correctness even Strongly-Ordered memory accesses require barriers in order to guarantee that multiple CPUs have a coherent view of the ordering of memory accesses.

Virtually everything done by this early code is done via explicit memory access only, so DSBs are seldom required. Existing barriers are demoted to DMB, except where a DSB is needed to synchronise non-memory signalling (i.e., before a SEV). If a particular platform performs cache maintenance in its power_up_setup function, it should force it to complete explicitly including a DSB, instead of relying on the bL_head framework code to do it.

Some additional DMBs are added to ensure all the memory ordering properties required by the race avoidance algorithm. DMBs are also moved out of loops, and for clarity some are moved so that most directly follow the memory operation which needs to be synchronised.

The setting of a CPU's bL_entry_vectors[] entry is also required to act as a synchronisation point, so a DMB is added after checking that entry to ensure that other CPUs do not observe gated operations leaking across the opening of the gate.

Quoted 2 lines

Signed-off-by Dave Martin <[email protected]>

---

Sorry to pick on this again but I am not able to understand why the strongly ordered access needs barriers. At least from the ARM point of view, a strongly ordered write will be more of blocking write and the further interconnect also is suppose to respect that

rule. SO read writes are like adding barrier after every load store

so adding explicit barriers doesn't make sense. Is this a side effect of some "write early response" kind of optimizations at interconnect level ?

Will you be able to point to specs or documents which puts this requirement ?

Quoted 28 lines · Santosh Shilimkar

+ Catalin, RMK

Dave,

Quoted 23 lines · Dave Martin

For architectural correctness even Strongly-Ordered memory accesses require barriers in order to guarantee that multiple CPUs have a coherent view of the ordering of memory accesses.

Virtually everything done by this early code is done via explicit memory access only, so DSBs are seldom required. Existing barriers are demoted to DMB, except where a DSB is needed to synchronise non-memory signalling (i.e., before a SEV). If a particular platform performs cache maintenance in its power_up_setup function, it should force it to complete explicitly including a DSB, instead of relying on the bL_head framework code to do it.

Some additional DMBs are added to ensure all the memory ordering properties required by the race avoidance algorithm. DMBs are also moved out of loops, and for clarity some are moved so that most directly follow the memory operation which needs to be synchronised.

The setting of a CPU's bL_entry_vectors[] entry is also required to act as a synchronisation point, so a DMB is added after checking that entry to ensure that other CPUs do not observe gated operations leaking across the opening of the gate.

Quoted 7 lines

Quoted 2 lines

Signed-off-by Dave Martin <[email protected]>

---

Sorry to pick on this again but I am not able to understand why the strongly ordered access needs barriers. At least from the ARM point of view, a strongly ordered write will be more of blocking write and the further interconnect also is suppose to respect that

This is what I originally assumed (hence the absence of barriers in the initial patch).

Quoted 1 line

rule. SO read writes are like adding barrier after every load store

This assumption turns out to be wrong, unfortunately, although in a uniprocessor scenario is makes no difference. A SO memory access does block the CPU making the access, but explicitly does not block the interconnect.

In a typical boot scenario for example, all secondary CPUs are quiescent or powered down, so there's no problem. But we can't make the same assumptions when we're trying to coordinate between multiple active CPUs.

Quoted 3 lines

so adding explicit barriers doesn't make sense. Is this a side effect of some "write early response" kind of optimizations at interconnect level ?

Strongly-Ordered accesses are always non-shareable, so there is no explicit guarantee of coherency between multiple masters.

If there is only one master, it makes no difference, but if there are multiple masters, there is no guarantee that they are conntected to a slave device (DRAM controller in this case) via a single slave port.

The architecture only guarantees global serialisation when there is a single slave device, but provides no way to know whether two accesses from different masters will reach the same slave port. This is in the realms of "implementation defined."

Unfortunately, a high-performance component like a DRAM controller is exactly the kind of component which may implement multiple master ports, so you can't guarantee that accesses are serialised in the same order from the perspective of all masters. There may be some pipelining and caching between each master port and the actual memory, for example. This is allowed, because there is no requirement for the DMC to look like a single slave device from the perspective of multiple masters.

A multi-ported slave might provide transparent coherency between master ports, but it is only required to guarantee this when the accesses are shareable (SO is always non-shared), or when explicit barriers are used to force synchronisation between the device's master ports.

Of course, a given platform may have a DMC with only one slave port, in which case the barriers should not be needed. But I wanted this code to be generic enough to be reusable -- hence the addition of the barriers. The CPU does not need to wait for a DMB to "complete" in any sense, so this does not necessarily have a meaningful impact on performance.

This is my understanding anyway.

Quoted 2 lines

Will you be able to point to specs or documents which puts this requirement ?

Unfortunately, this is one of this things which we require not because there is a statement in the ARM ARM to say that we need it -- rather, there is no statement in the ARM ARM to say that we don't.

Quoted 40 lines · Dave Martin

Quoted 28 lines · Santosh Shilimkar

+ Catalin, RMK

Dave,

Quoted 23 lines · Dave Martin

For architectural correctness even Strongly-Ordered memory accesses require barriers in order to guarantee that multiple CPUs have a coherent view of the ordering of memory accesses.

Virtually everything done by this early code is done via explicit memory access only, so DSBs are seldom required. Existing barriers are demoted to DMB, except where a DSB is needed to synchronise non-memory signalling (i.e., before a SEV). If a particular platform performs cache maintenance in its power_up_setup function, it should force it to complete explicitly including a DSB, instead of relying on the bL_head framework code to do it.

Some additional DMBs are added to ensure all the memory ordering properties required by the race avoidance algorithm. DMBs are also moved out of loops, and for clarity some are moved so that most directly follow the memory operation which needs to be synchronised.

The setting of a CPU's bL_entry_vectors[] entry is also required to act as a synchronisation point, so a DMB is added after checking that entry to ensure that other CPUs do not observe gated operations leaking across the opening of the gate.

Quoted 7 lines

Quoted 2 lines

Signed-off-by Dave Martin <[email protected]>

---

Sorry to pick on this again but I am not able to understand why the strongly ordered access needs barriers. At least from the ARM point of view, a strongly ordered write will be more of blocking write and the further interconnect also is suppose to respect that

This is what I originally assumed (hence the absence of barriers in the initial patch).

Quoted 7 lines

Quoted 1 line

rule. SO read writes are like adding barrier after every load store

This assumption turns out to be wrong, unfortunately, although in a uniprocessor scenario is makes no difference. A SO memory access does block the CPU making the access, but explicitly does not block the interconnect.

I suspected the interconnect part when you described the barrier need for SO memory region.

Quoted 5 lines

In a typical boot scenario for example, all secondary CPUs are quiescent or powered down, so there's no problem. But we can't make the same assumptions when we're trying to coordinate between multiple active CPUs.

Quoted 7 lines

Quoted 3 lines

so adding explicit barriers doesn't make sense. Is this a side effect of some "write early response" kind of optimizations at interconnect level ?

Strongly-Ordered accesses are always non-shareable, so there is no explicit guarantee of coherency between multiple masters.

This is where probably issue then. My understanding is exactly opposite here and hence I wasn't worried about multi-master CPU scenario since sharable attributes would be taking care of it considering the same page tables being used in SMP system.

ARM documentation says -
------------
Shareability and the S bit, with TEX remap
The memory type of a region, as indicated in the Memory type column of Table B3-12 on page B3-1350, provides
the first level of control of whether the region is shareable:
? If the memory type is Strongly-ordered then the region is Shareable
------------------------------------------------------------

Quoted 68 lines · Santosh Shilimkar

Quoted 40 lines · Dave Martin

Quoted 28 lines · Santosh Shilimkar

+ Catalin, RMK

Dave,

Quoted 23 lines · Dave Martin

For architectural correctness even Strongly-Ordered memory accesses require barriers in order to guarantee that multiple CPUs have a coherent view of the ordering of memory accesses.

Virtually everything done by this early code is done via explicit memory access only, so DSBs are seldom required. Existing barriers are demoted to DMB, except where a DSB is needed to synchronise non-memory signalling (i.e., before a SEV). If a particular platform performs cache maintenance in its power_up_setup function, it should force it to complete explicitly including a DSB, instead of relying on the bL_head framework code to do it.

Some additional DMBs are added to ensure all the memory ordering properties required by the race avoidance algorithm. DMBs are also moved out of loops, and for clarity some are moved so that most directly follow the memory operation which needs to be synchronised.

The setting of a CPU's bL_entry_vectors[] entry is also required to act as a synchronisation point, so a DMB is added after checking that entry to ensure that other CPUs do not observe gated operations leaking across the opening of the gate.

Quoted 7 lines

Quoted 2 lines

Signed-off-by Dave Martin <[email protected]>

---

Sorry to pick on this again but I am not able to understand why the strongly ordered access needs barriers. At least from the ARM point of view, a strongly ordered write will be more of blocking write and the further interconnect also is suppose to respect that

This is what I originally assumed (hence the absence of barriers in the initial patch).

Quoted 7 lines

Quoted 1 line

rule. SO read writes are like adding barrier after every load store

This assumption turns out to be wrong, unfortunately, although in a uniprocessor scenario is makes no difference. A SO memory access does block the CPU making the access, but explicitly does not block the interconnect.

I suspected the interconnect part when you described the barrier need for SO memory region.

Quoted 5 lines

In a typical boot scenario for example, all secondary CPUs are quiescent or powered down, so there's no problem. But we can't make the same assumptions when we're trying to coordinate between multiple active CPUs.

Quoted 7 lines

Quoted 3 lines

so adding explicit barriers doesn't make sense. Is this a side effect of some "write early response" kind of optimizations at interconnect level ?

Strongly-Ordered accesses are always non-shareable, so there is no explicit guarantee of coherency between multiple masters.

This is where probably issue then. My understanding is exactly opposite here and hence I wasn't worried about multi-master CPU scenario since sharable attributes would be taking care of it considering the same page tables being used in SMP system.

Quoted 8 lines

ARM documentation says -
------------
Shareability and the S bit, with TEX remap
The memory type of a region, as indicated in the Memory type column of Table B3-12 on page B3-1350, provides
the first level of control of whether the region is shareable:
? If the memory type is Strongly-ordered then the region is Shareable
------------------------------------------------------------

Hmmm, it looks like you're right here. My assumption that SO implies non-shareable is wrong. This is backed up by:

A3.5.6 Device and Strongly-ordered memory

"Address locations marked as Strongly-ordered [...] are always treated as Shareable."

I think this is sufficient to ensure that if two CPUs access the same location with SO accesses, each will see an access order to any single location which is consistent with the program order of the accesses on the other CPUs. (This comes from the glossary definition of Coherent.)

However, I can't see any general guarantee for accesses to _different_ locations, beyond the guarantees for certain special cases given in A3.8.2 Ordering requirements for memory accesses (address and control dependencies etc.)

This may make some of the dmbs unnecessary, but it is not clear whether they are all unnecessary.

I'll need to follow up on this and see if we can get an answer.

Quoted 18 lines · Dave Martin

For architectural correctness even Strongly-Ordered memory accesses require barriers in order to guarantee that multiple CPUs have a coherent view of the ordering of memory accesses.

Virtually everything done by this early code is done via explicit memory access only, so DSBs are seldom required. Existing barriers are demoted to DMB, except where a DSB is needed to synchronise non-memory signalling (i.e., before a SEV). If a particular platform performs cache maintenance in its power_up_setup function, it should force it to complete explicitly including a DSB, instead of relying on the bL_head framework code to do it.

Some additional DMBs are added to ensure all the memory ordering properties required by the race avoidance algorithm. DMBs are also moved out of loops, and for clarity some are moved so that most directly follow the memory operation which needs to be synchronised.

Quoted 6 lines

The setting of a CPU's bL_entry_vectors[] entry is also required to act as a synchronisation point, so a DMB is added after checking that entry to ensure that other CPUs do not observe gated operations leaking across the opening of the gate.

Signed-off-by Dave Martin <[email protected]>

Sorry to pick on this again but I am not able to understand why the strongly ordered access needs barriers. At least from the ARM point of view, a strongly ordered write will be more of blocking write and the further interconnect also is suppose to respect that rule. SO read writes are like adding barrier after every load store so adding explicit barriers doesn't make sense. Is this a side effect of some "write early response" kind of optimizations at interconnect level ?

Santosh,

Quoted 26 lines · Santosh Shilimkar

Quoted 18 lines · Dave Martin

For architectural correctness even Strongly-Ordered memory accesses require barriers in order to guarantee that multiple CPUs have a coherent view of the ordering of memory accesses.

Virtually everything done by this early code is done via explicit memory access only, so DSBs are seldom required. Existing barriers are demoted to DMB, except where a DSB is needed to synchronise non-memory signalling (i.e., before a SEV). If a particular platform performs cache maintenance in its power_up_setup function, it should force it to complete explicitly including a DSB, instead of relying on the bL_head framework code to do it.

Some additional DMBs are added to ensure all the memory ordering properties required by the race avoidance algorithm. DMBs are also moved out of loops, and for clarity some are moved so that most directly follow the memory operation which needs to be synchronised.

Quoted 6 lines

The setting of a CPU's bL_entry_vectors[] entry is also required to act as a synchronisation point, so a DMB is added after checking that entry to ensure that other CPUs do not observe gated operations leaking across the opening of the gate.

Signed-off-by Dave Martin <[email protected]>

Quoted 8 lines

Sorry to pick on this again but I am not able to understand why the strongly ordered access needs barriers. At least from the ARM point of view, a strongly ordered write will be more of blocking write and the further interconnect also is suppose to respect that rule. SO read writes are like adding barrier after every load store so adding explicit barriers doesn't make sense. Is this a side effect of some "write early response" kind of optimizations at interconnect level ?

SO or Device memory accesses are *not* like putting a proper barrier between each access, though it may behave in some situations like having a barrier. The ARM ARM (A3.8.3, fig 3.5) defines how accesses must *arrive* at a peripheral or block of memory depending on the memory type and in case of Device or SO we don't need additional barriers because such accesses would *arrive* in order (given the minimum 1KB range restriction). But it does not say anything about *observability* by a different *master*. That's because you can't guarantee that your memory accesses go to the same slave port.

For observability by a different master, you need an explicit DMB even though the memory type is Device or SO. While it may work fine in most cases (especially when the accesses by various masters go to the same slave port), you can't be sure what the memory controller or whatever interconnect do.

As Dave said, it's more about what the ARM ARM doesn't say rather than what it explicitly states.

Santosh,

Quoted 26 lines · Santosh Shilimkar

Quoted 18 lines · Dave Martin

For architectural correctness even Strongly-Ordered memory accesses require barriers in order to guarantee that multiple CPUs have a coherent view of the ordering of memory accesses.

Virtually everything done by this early code is done via explicit memory access only, so DSBs are seldom required. Existing barriers are demoted to DMB, except where a DSB is needed to synchronise non-memory signalling (i.e., before a SEV). If a particular platform performs cache maintenance in its power_up_setup function, it should force it to complete explicitly including a DSB, instead of relying on the bL_head framework code to do it.

Some additional DMBs are added to ensure all the memory ordering properties required by the race avoidance algorithm. DMBs are also moved out of loops, and for clarity some are moved so that most directly follow the memory operation which needs to be synchronised.

Quoted 6 lines

The setting of a CPU's bL_entry_vectors[] entry is also required to act as a synchronisation point, so a DMB is added after checking that entry to ensure that other CPUs do not observe gated operations leaking across the opening of the gate.

Signed-off-by Dave Martin <[email protected]>

Quoted 8 lines

Sorry to pick on this again but I am not able to understand why the strongly ordered access needs barriers. At least from the ARM point of view, a strongly ordered write will be more of blocking write and the further interconnect also is suppose to respect that rule. SO read writes are like adding barrier after every load store so adding explicit barriers doesn't make sense. Is this a side effect of some "write early response" kind of optimizations at interconnect level ?

SO or Device memory accesses are *not* like putting a proper barrier between each access, though it may behave in some situations like having a barrier. The ARM ARM (A3.8.3, fig 3.5) defines how accesses must *arrive* at a peripheral or block of memory depending on the memory type and in case of Device or SO we don't need additional barriers because such accesses would *arrive* in order (given the minimum 1KB range restriction). But it does not say anything about *observability* by a different *master*. That's because you can't guarantee that your memory accesses go to the same slave port.

For observability by a different master, you need an explicit DMB even though the memory type is Device or SO. While it may work fine in most cases (especially when the accesses by various masters go to the same slave port), you can't be sure what the memory controller or whatever interconnect do.

As Dave said, it's more about what the ARM ARM doesn't say rather than what it explicitly states.

After much head-scratching and discussion, I have concluded that we need comprehensive memory barriers in order to ensure that the low-level synchronisation code executes robustly on all platforms.

DSBs are excessive on most situations though, so many DSBs can be replces with DMBs.

As was observed in review, providing a C interface to the vlocks makes little sense, so this series gets rid of it.

Quoted 4 lines · Dave Martin

After much head-scratching and discussion, I have concluded that we need comprehensive memory barriers in order to ensure that the low-level synchronisation code executes robustly on all platforms.

Quoted 5 lines

DSBs are excessive on most situations though, so many DSBs can be replces with DMBs.

As was observed in review, providing a C interface to the vlocks makes little sense, so this series gets rid of it.

Thanks. I have already removed the C stuff in my version but I'll merge the rest no problem.