Add note of release-acquire store vulnerability

Documentation/memory-barriers.txt And a couple of implicit varieties: · line 503

Quoted 12 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it. This commit therefore updates the documentation to call this vulnerability out explicitly.

Reported-by Alan Stern <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Documentation/memory-barriers.txt +3 -0

+     However, please note that a chain of RELEASE+ACQUIRE pairs may be
+     broken by a store by another thread that overwrites the RELEASE
+     operation's store before the ACQUIRE operation's read.

This is the powerpc lwsync quirk, right? Where the barrier disappears when it looses the store.

Or is there more to it? Its not entirely clear from the Changelog, which I feel should describe the reason for the behaviour.

Quoted 14 lines · Peter Zijlstra

Documentation/memory-barriers.txt And a couple of implicit varieties: · line 503

Quoted 12 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it. This commit therefore updates the documentation to call this vulnerability out explicitly.

Reported-by Alan Stern <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Documentation/memory-barriers.txt +3 -0

+     However, please note that a chain of RELEASE+ACQUIRE pairs may be
+     broken by a store by another thread that overwrites the RELEASE
+     operation's store before the ACQUIRE operation's read.

Quoted 5 lines

This is the powerpc lwsync quirk, right? Where the barrier disappears when it looses the store.

Or is there more to it? Its not entirely clear from the Changelog, which I feel should describe the reason for the behaviour.

If I've groked it correctly, it's for cases like:

PO:
Wx=1
WyRel=1

P1:
Wy=2

P2:
RyAcq=2
Rx=0

Final value of y is 2.

This is permitted on arm64. If you make P1's store a store-release, then it's forbidden, but I suspect that's not generally true of the kernel memory model.

Quoted 35 lines · Will Deacon

Quoted 14 lines · Peter Zijlstra

Documentation/memory-barriers.txt And a couple of implicit varieties: · line 503

Quoted 12 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it. This commit therefore updates the documentation to call this vulnerability out explicitly.

Reported-by Alan Stern <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Documentation/memory-barriers.txt +3 -0

+     However, please note that a chain of RELEASE+ACQUIRE pairs may be
+     broken by a store by another thread that overwrites the RELEASE
+     operation's store before the ACQUIRE operation's read.

Quoted 5 lines

This is the powerpc lwsync quirk, right? Where the barrier disappears when it looses the store.

Or is there more to it? Its not entirely clear from the Changelog, which I feel should describe the reason for the behaviour.

If I've groked it correctly, it's for cases like:

PO:
Wx=1
WyRel=1

P1:
Wy=2

P2:
RyAcq=2
Rx=0

Quoted 6 lines

Final value of y is 2.

This is permitted on arm64. If you make P1's store a store-release, then it's forbidden, but I suspect that's not generally true of the kernel memory model.

Right, I think that on PowerPC, even if P1 does store-release you can still get this, since the two stores conflict one can loose out, and the lwsync associated with the loosing store gets removed along with it.

So yes, I think this needs more clarification.

Quoted 14 lines · Peter Zijlstra

Documentation/memory-barriers.txt And a couple of implicit varieties: · line 503

Quoted 12 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it. This commit therefore updates the documentation to call this vulnerability out explicitly.

Reported-by Alan Stern <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Documentation/memory-barriers.txt +3 -0

+     However, please note that a chain of RELEASE+ACQUIRE pairs may be
+     broken by a store by another thread that overwrites the RELEASE
+     operation's store before the ACQUIRE operation's read.

Quoted 5 lines

This is the powerpc lwsync quirk, right? Where the barrier disappears when it looses the store.

Or is there more to it? Its not entirely clear from the Changelog, which I feel should describe the reason for the behaviour.

If I've groked it correctly, it's for cases like:

PO:
Wx=1
WyRel=1

P1:
Wy=2

P2:
RyAcq=2
Rx=0

Final value of y is 2.

This is permitted on arm64. If you make P1's store a store-release, then it's forbidden, but I suspect that's not generally true of the kernel memory model.

Quoted 35 lines · Will Deacon

Quoted 14 lines · Peter Zijlstra

Documentation/memory-barriers.txt And a couple of implicit varieties: · line 503

Quoted 12 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it. This commit therefore updates the documentation to call this vulnerability out explicitly.

Reported-by Alan Stern <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Documentation/memory-barriers.txt +3 -0

+     However, please note that a chain of RELEASE+ACQUIRE pairs may be
+     broken by a store by another thread that overwrites the RELEASE
+     operation's store before the ACQUIRE operation's read.

Quoted 5 lines

This is the powerpc lwsync quirk, right? Where the barrier disappears when it looses the store.

Or is there more to it? Its not entirely clear from the Changelog, which I feel should describe the reason for the behaviour.

If I've groked it correctly, it's for cases like:

PO:
Wx=1
WyRel=1

P1:
Wy=2

P2:
RyAcq=2
Rx=0

Quoted 6 lines

Final value of y is 2.

This is permitted on arm64. If you make P1's store a store-release, then it's forbidden, but I suspect that's not generally true of the kernel memory model.

That is the one! And to Peter's point, powerpc does the same for the example as shown. However, on powerpc, upgrading P1's store to release has no effect because there is no earlier access for the resulting lwsync to influence. For whatever it might be worth, C11 won't guarantee ordering in that case, either. Nor will the current Linux-kernel memory model. (Yes, I did just try it to make sure. Why do you ask?)

So you guys are fishing for an expanded commit log, for example, like the following? ;-)

Thanx, Paul

------------------------------------------------------------------------

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		smp_store_release(y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel memory model.

Quoted 80 lines · Paul E. McKenney

Quoted 35 lines · Will Deacon

Quoted 14 lines · Peter Zijlstra

Documentation/memory-barriers.txt And a couple of implicit varieties: · line 503

Quoted 12 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it. This commit therefore updates the documentation to call this vulnerability out explicitly.

Reported-by Alan Stern <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Documentation/memory-barriers.txt +3 -0

+     However, please note that a chain of RELEASE+ACQUIRE pairs may be
+     broken by a store by another thread that overwrites the RELEASE
+     operation's store before the ACQUIRE operation's read.

Quoted 5 lines

This is the powerpc lwsync quirk, right? Where the barrier disappears when it looses the store.

Or is there more to it? Its not entirely clear from the Changelog, which I feel should describe the reason for the behaviour.

If I've groked it correctly, it's for cases like:

PO:
Wx=1
WyRel=1

P1:
Wy=2

P2:
RyAcq=2
Rx=0

Quoted 6 lines

Final value of y is 2.

This is permitted on arm64. If you make P1's store a store-release, then it's forbidden, but I suspect that's not generally true of the kernel memory model.

That is the one! And to Peter's point, powerpc does the same for the example as shown. However, on powerpc, upgrading P1's store to release has no effect because there is no earlier access for the resulting lwsync to influence. For whatever it might be worth, C11 won't guarantee ordering in that case, either. Nor will the current Linux-kernel memory model. (Yes, I did just try it to make sure. Why do you ask?)

So you guys are fishing for an expanded commit log, for example, like the following? ;-)

Thanx, Paul

------------------------------------------------------------------------

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		smp_store_release(y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Quoted 3 lines

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel memory model.

FWIW, ARM doesn't allow this and arm64 only allows it if P1 uses WRITE_ONCE instead of store-release.

Quoted 81 lines · Will Deacon

Quoted 80 lines · Paul E. McKenney

Quoted 35 lines · Will Deacon

Quoted 14 lines · Peter Zijlstra

Documentation/memory-barriers.txt And a couple of implicit varieties: · line 503

Quoted 12 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it. This commit therefore updates the documentation to call this vulnerability out explicitly.

Reported-by Alan Stern <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Documentation/memory-barriers.txt +3 -0

+     However, please note that a chain of RELEASE+ACQUIRE pairs may be
+     broken by a store by another thread that overwrites the RELEASE
+     operation's store before the ACQUIRE operation's read.

Quoted 5 lines

This is the powerpc lwsync quirk, right? Where the barrier disappears when it looses the store.

Or is there more to it? Its not entirely clear from the Changelog, which I feel should describe the reason for the behaviour.

If I've groked it correctly, it's for cases like:

PO:
Wx=1
WyRel=1

P1:
Wy=2

P2:
RyAcq=2
Rx=0

Quoted 6 lines

Final value of y is 2.

This is permitted on arm64. If you make P1's store a store-release, then it's forbidden, but I suspect that's not generally true of the kernel memory model.

That is the one! And to Peter's point, powerpc does the same for the example as shown. However, on powerpc, upgrading P1's store to release has no effect because there is no earlier access for the resulting lwsync to influence. For whatever it might be worth, C11 won't guarantee ordering in that case, either. Nor will the current Linux-kernel memory model. (Yes, I did just try it to make sure. Why do you ask?)

So you guys are fishing for an expanded commit log, for example, like the following? ;-)

Thanx, Paul

------------------------------------------------------------------------

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		smp_store_release(y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Quoted 6 lines

Quoted 3 lines

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel memory model.

FWIW, ARM doesn't allow this and arm64 only allows it if P1 uses WRITE_ONCE instead of store-release.

Good catch, apologies for the error. The following, then?

Thanx, Paul

------------------------------------------------------------------------

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		WRITE_ONCE(*y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel

memory model.

Quoted 118 lines · Paul E. McKenney

Quoted 81 lines · Will Deacon

Quoted 80 lines · Paul E. McKenney

Quoted 35 lines · Will Deacon

Quoted 14 lines · Peter Zijlstra

Review

Quoted 12 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it. This commit therefore updates the documentation to call this vulnerability out explicitly.

Reported-by Alan Stern <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

(quoted diff) +3 -0

+     However, please note that a chain of RELEASE+ACQUIRE pairs may be
+     broken by a store by another thread that overwrites the RELEASE
+     operation's store before the ACQUIRE operation's read.

Quoted 5 lines

This is the powerpc lwsync quirk, right? Where the barrier disappears when it looses the store.

Or is there more to it? Its not entirely clear from the Changelog, which I feel should describe the reason for the behaviour.

If I've groked it correctly, it's for cases like:

PO:
Wx=1
WyRel=1

P1:
Wy=2

P2:
RyAcq=2
Rx=0

Quoted 6 lines

Final value of y is 2.

This is permitted on arm64. If you make P1's store a store-release, then it's forbidden, but I suspect that's not generally true of the kernel memory model.

That is the one! And to Peter's point, powerpc does the same for the example as shown. However, on powerpc, upgrading P1's store to release has no effect because there is no earlier access for the resulting lwsync to influence. For whatever it might be worth, C11 won't guarantee ordering in that case, either. Nor will the current Linux-kernel memory model. (Yes, I did just try it to make sure. Why do you ask?)

So you guys are fishing for an expanded commit log, for example, like the following? ;-)

Thanx, Paul

------------------------------------------------------------------------

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		smp_store_release(y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Quoted 6 lines

Quoted 3 lines

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel memory model.

FWIW, ARM doesn't allow this and arm64 only allows it if P1 uses WRITE_ONCE instead of store-release.

Good catch, apologies for the error. The following, then?

Thanx, Paul

------------------------------------------------------------------------

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		WRITE_ONCE(*y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Quoted 2 lines

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel

And the above needs to be (r1!=2 || r2 != 0)... Sigh!

Quoted 1 line

memory model.

Quoted 19 lines

Quoted 19 lines

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		WRITE_ONCE(*y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Quoted 4 lines

Quoted 2 lines

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel

And the above needs to be (r1!=2 || r2 != 0)... Sigh!

Make that (y==2 && r1==2 && r2 == 0).

Any further bids? ;-)

Quoted 22 lines · Paul E. McKenney

Quoted 22 lines

Quoted 21 lines

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		WRITE_ONCE(*y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel

Quoted 5 lines

Quoted 1 line

And the above needs to be (r1!=2 || r2 != 0)... Sigh!

Make that (y==2 && r1==2 && r2 == 0).

Any further bids? ;-)

Isn't that the trivial P1,P2,P0 order again?

How about something like so on PPC?

P0(int *x, int *y)

{
	WRITE_ONCE(*x, 1);
	smp_store_release(y, 1);
}

P1(int *x, int *y)

{
	WRITE_ONCE(x, 2);

	smp_store_release(y, 2);
}

P2(int *x, int *y)

{
	r1 = smp_load_acquire(y);
	r2 = READ_ONCE(*x);
}

(((x==1 && y==2) | (x==2 && y==1)) && (r1==1 || r1==2) && r2==0)

If you execute P0 and P1 concurrently and one store of each 'wins' the LWSYNC of either is null and void, and therefore P2 is unordered and can observe r2==0.

Quoted 11 lines · Peter Zijlstra

How about something like so on PPC?

P0(int *x, int *y)

{
	WRITE_ONCE(*x, 1);
	smp_store_release(y, 1);
}

P1(int *x, int *y)

{
	WRITE_ONCE(x, 2);

Need "WRITE_ONCE(*x, 2)" here.

Quoted 9 lines

	smp_store_release(y, 2);
}

P2(int *x, int *y)

{
	r1 = smp_load_acquire(y);
	r2 = READ_ONCE(*x);
}

Quoted 1 line

(((x==1 && y==2) | (x==2 && y==1)) && (r1==1 || r1==2) && r2==0)

That exists-clause is quite dazzling... So if each of P0 and P1 win, but on different stores, and if P2 follows one or the other of P0 or P1, can r2 get the pre-initialization value for x?

Quoted 3 lines

If you execute P0 and P1 concurrently and one store of each 'wins' the LWSYNC of either is null and void, and therefore P2 is unordered and can observe r2==0.

That vaguely resembles the infamous Z6.3, but only vaguely. The Linux-kernel memory model says "forbidden" to this:

	C C-WillDeacon-AcqRelStore.litmus

	{
	}

	P0(int *x, int *y)
	{
	      WRITE_ONCE(*x, 1);
	      smp_store_release(y, 1);
	}

	P1(int *x, int *y)
	{
	      WRITE_ONCE(*x, 2);
	      smp_store_release(y, 2);
	}

	P2(int *x, int *y)
	{
	      r1 = smp_load_acquire(y);
	      r2 = READ_ONCE(*x);
	}

	exists
	(((x=1 /\ y=2) \/ (x=2 /\ y=1)) /\ (2:r1=1 \/ 2:r1=2) /\ 2:r2=0)

Quoted 25 lines · Paul E. McKenney

Quoted 11 lines · Peter Zijlstra

How about something like so on PPC?

P0(int *x, int *y)

{
	WRITE_ONCE(*x, 1);
	smp_store_release(y, 1);
}

P1(int *x, int *y)

{
	WRITE_ONCE(x, 2);

Need "WRITE_ONCE(*x, 2)" here.

Quoted 9 lines

	smp_store_release(y, 2);
}

P2(int *x, int *y)

{
	r1 = smp_load_acquire(y);
	r2 = READ_ONCE(*x);
}

Quoted 5 lines

Quoted 1 line

(((x==1 && y==2) | (x==2 && y==1)) && (r1==1 || r1==2) && r2==0)

That exists-clause is quite dazzling... So if each of P0 and P1 win, but on different stores, and if P2 follows one or the other of P0 or P1, can r2 get the pre-initialization value for x?

In fact, this is more than you need. It's enough to specify

exists (2:r1=1 \/ 2:r1=2) /\ 2:r2=0

This much already is forbidden. For the sake of argument, say r1=1. Then P2 has read from P1's store-release. By definition, P1's write to x is visible to P2, so r2 will get the value from that write or from one that is later in x's coherence order. In other words, r2 will end up equal to either 1 or 2, but not 0.

Quoted 3 lines

Quoted 3 lines

If you execute P0 and P1 concurrently and one store of each 'wins' the LWSYNC of either is null and void, and therefore P2 is unordered and can observe r2==0.

Not so. lwsync instructions cannot be "voided".

Quoted 26 lines

That vaguely resembles the infamous Z6.3, but only vaguely. The Linux-kernel memory model says "forbidden" to this:

	C C-WillDeacon-AcqRelStore.litmus

	{
	}

	P0(int *x, int *y)
	{
	      WRITE_ONCE(*x, 1);
	      smp_store_release(y, 1);
	}

	P1(int *x, int *y)
	{
	      WRITE_ONCE(*x, 2);
	      smp_store_release(y, 2);
	}

	P2(int *x, int *y)
	{
	      r1 = smp_load_acquire(y);
	      r2 = READ_ONCE(*x);
	}

Quoted 2 lines

	exists
	(((x=1 /\ y=2) \/ (x=2 /\ y=1)) /\ (2:r1=1 \/ 2:r1=2) /\ 2:r2=0)

As above, you can leave out the part about the final values for x and y. The test will still be forbidden.

On the other hand, there's no guarantee that if r1=1 at the end then r2 will also be 1. It's quite possible that r1=1 and r2=2, or vice versa.

Quoted 3 lines

Quoted 3 lines

If you execute P0 and P1 concurrently and one store of each 'wins' the LWSYNC of either is null and void, and therefore P2 is unordered and can observe r2==0.

Not so. lwsync instructions cannot be "voided".

Quoted 25 lines · Peter Zijlstra

Quoted 24 lines · Paul E. McKenney

Quoted 23 lines

Quoted 19 lines

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		WRITE_ONCE(*y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Quoted 2 lines

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel

And the above needs to be (r1!=2 || r2 != 0)... Sigh!

Quoted 5 lines

Quoted 3 lines

Make that (y==2 && r1==2 && r2 == 0).

Any further bids? ;-)

Isn't that the trivial P1,P2,P0 order again?

I don't believe so. Wouldn't the final P0 would leave y==1?

Quoted 11 lines · Peter Zijlstra

How about something like so on PPC?

P0(int *x, int *y)

{
	WRITE_ONCE(*x, 1);
	smp_store_release(y, 1);
}

P1(int *x, int *y)

{
	WRITE_ONCE(x, 2);

Need "WRITE_ONCE(*x, 2)" here.

Quoted 10 lines

	smp_store_release(y, 2);
}

P2(int *x, int *y)

{
	r1 = smp_load_acquire(y);
	r2 = READ_ONCE(*x);
}

(((x==1 && y==2) | (x==2 && y==1)) && (r1==1 || r1==2) && r2==0)

That exists-clause is quite dazzling... So if each of P0 and P1 win, but on different stores, and if P2 follows one or the other of P0 or P1, can r2 get the pre-initialization value for x?

Quoted 3 lines

If you execute P0 and P1 concurrently and one store of each 'wins' the LWSYNC of either is null and void, and therefore P2 is unordered and can observe r2==0.

That vaguely resembles the infamous Z6.3, but only vaguely. The Linux-kernel memory model says "forbidden" to this:

So let's try PPCMEM. If PPCMEM allows it, then the kernel model is clearly broken.

	PPC PeterZijlstra+o-r+o-r+a-o-SB.litmus
	{
	0:r1=1; 0:r2=2; 0:r3=x; 0:r4=y;
	1:r1=1; 1:r2=2; 1:r3=x; 1:r4=y;
			2:r3=x; 2:r4=y;
	}
	 P0           | P1           | P2           ;
	 stw r1,0(r3) | stw r2,0(r3) | lwz r1,0(r4) ;
	 lwsync       | lwsync       | lwsync       ;
	 stw r1,0(r4) | stw r2,0(r4) | lwz r2,0(r3) ;
	exists
	(((x=1 /\ y=2) \/ (x=2 /\ y=1)) /\ (2:r1=1 \/ 2:r1=2) /\ 2:r2=0)

Or did I incorrectly translate your litmus test?

Quoted 35 lines · Paul E. McKenney

Quoted 11 lines · Peter Zijlstra

How about something like so on PPC?

P0(int *x, int *y)

{
	WRITE_ONCE(*x, 1);
	smp_store_release(y, 1);
}

P1(int *x, int *y)

{
	WRITE_ONCE(x, 2);

Need "WRITE_ONCE(*x, 2)" here.

Quoted 10 lines

	smp_store_release(y, 2);
}

P2(int *x, int *y)

{
	r1 = smp_load_acquire(y);
	r2 = READ_ONCE(*x);
}

(((x==1 && y==2) | (x==2 && y==1)) && (r1==1 || r1==2) && r2==0)

That exists-clause is quite dazzling... So if each of P0 and P1 win, but on different stores, and if P2 follows one or the other of P0 or P1, can r2 get the pre-initialization value for x?

Quoted 3 lines

If you execute P0 and P1 concurrently and one store of each 'wins' the LWSYNC of either is null and void, and therefore P2 is unordered and can observe r2==0.

Quoted 2 lines

That vaguely resembles the infamous Z6.3, but only vaguely. The Linux-kernel memory model says "forbidden" to this:

https://www.cl.cam.ac.uk/~pes20/ppc-supplemental/ppc710.html

That one, right?

Hmm, I seem to remember something else.. /me goes poke through history and comes up with:

https://lkml.kernel.org/r/[email protected]

So what was that about then? I remember it being a completely nonsensical case, but a weird one.

Quoted 16 lines

So let's try PPCMEM. If PPCMEM allows it, then the kernel model is clearly broken.

	PPC PeterZijlstra+o-r+o-r+a-o-SB.litmus
	{
	0:r1=1; 0:r2=2; 0:r3=x; 0:r4=y;
	1:r1=1; 1:r2=2; 1:r3=x; 1:r4=y;
			2:r3=x; 2:r4=y;
	}
	 P0           | P1           | P2           ;
	 stw r1,0(r3) | stw r2,0(r3) | lwz r1,0(r4) ;
	 lwsync       | lwsync       | lwsync       ;
	 stw r1,0(r4) | stw r2,0(r4) | lwz r2,0(r3) ;
	exists
	(((x=1 /\ y=2) \/ (x=2 /\ y=1)) /\ (2:r1=1 \/ 2:r1=2) /\ 2:r2=0)

Quoted 1 line

Or did I incorrectly translate your litmus test?

Looks about right.

Still not seeing how that is prohibited though. My reasoning is as follows:

- P0 and P1 both store to x, one looses (say P0). Effectively only P1 does a store.

- P0 and P1 both store to y, one looses (say P1). Effectively only P0 does a store.

- P2 reads y, sees the value from P0.

- P2 does lwsync, which constraints P2 to not issue the load of x before this. It also forms a (local) sync-point with P0 for having seen its store or y.

- P2 reads x, sees the initial value because the store from P1 hasn't been propagated yet.

It will not see the store P0 did to x, since that didn't happen.

Assuming I'm wrong on that last part, is then the following possible?

(x=2 /\ y=1 /\ 2:r1=1 /\ 2:r2=1)

Where we see a store that didn't happen?

PowerPC does not "obscure" stores, so both stores really are there and the lwsync really has effect on all CPUs. From what I understand, even CPUs that do obscure stores only do so in the case of repeated stores by the same CPU to the same variable, and the above litmus test doesn't have this.

So all the stores happen, and each CPU's stores are at least locally ordered.

Quoted 6 lines · Paul E. McKenney

PowerPC does not "obscure" stores, so both stores really are there and the lwsync really has effect on all CPUs. From what I understand, even CPUs that do obscure stores only do so in the case of repeated stores by the same CPU to the same variable, and the above litmus test doesn't have this.

Quoted 2 lines

So all the stores happen, and each CPU's stores are at least locally ordered.

OK, when I'm not sure I ever understood the case where smp_wmb() went wonky on PPC, sadly I cannot now find the email where you mentioned that :/

Quoted 81 lines · Will Deacon

Quoted 80 lines · Paul E. McKenney

Quoted 35 lines · Will Deacon

Quoted 14 lines · Peter Zijlstra

Documentation/memory-barriers.txt And a couple of implicit varieties: · line 503

Quoted 12 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it. This commit therefore updates the documentation to call this vulnerability out explicitly.

Reported-by Alan Stern <[email protected]>

Signed-off-by Paul E. McKenney <[email protected]>

Documentation/memory-barriers.txt +3 -0

+     However, please note that a chain of RELEASE+ACQUIRE pairs may be
+     broken by a store by another thread that overwrites the RELEASE
+     operation's store before the ACQUIRE operation's read.

Quoted 5 lines

This is the powerpc lwsync quirk, right? Where the barrier disappears when it looses the store.

Or is there more to it? Its not entirely clear from the Changelog, which I feel should describe the reason for the behaviour.

If I've groked it correctly, it's for cases like:

PO:
Wx=1
WyRel=1

P1:
Wy=2

P2:
RyAcq=2
Rx=0

Quoted 6 lines

Final value of y is 2.

This is permitted on arm64. If you make P1's store a store-release, then it's forbidden, but I suspect that's not generally true of the kernel memory model.

That is the one! And to Peter's point, powerpc does the same for the example as shown. However, on powerpc, upgrading P1's store to release has no effect because there is no earlier access for the resulting lwsync to influence. For whatever it might be worth, C11 won't guarantee ordering in that case, either. Nor will the current Linux-kernel memory model. (Yes, I did just try it to make sure. Why do you ask?)

So you guys are fishing for an expanded commit log, for example, like the following? ;-)

Thanx, Paul

------------------------------------------------------------------------

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		smp_store_release(y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Quoted 6 lines

Quoted 3 lines

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel memory model.

FWIW, ARM doesn't allow this and arm64 only allows it if P1 uses WRITE_ONCE instead of store-release.

Good catch, apologies for the error. The following, then?

Thanx, Paul

------------------------------------------------------------------------

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();

		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		WRITE_ONCE(*y, 2);

	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel memory model.

Quoted 9 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();

               ^^^^^^^^^^^

What is this smp_wmb() for?

Quoted 6 lines

		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		WRITE_ONCE(*y, 2);

If we change this WRITE_ONCE to a relaxed atomic operation(e.g. xchg_relaxed(y, 2)), both herd and ppcmem said the exist-clause "y = 2 /\ 2:r1 = 2 /\ 2:r2 = 0" wouldn't be triggered on PPC.

I guess we will get the same behavior on ARM/ARM64, Will?

If a normal store could break chain, while a RmW atomic won't, do we want to call it out in the document and build our memory model around this?

Quoted 6 lines · Boqun Feng

Quoted 5 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

Quoted 7 lines

Quoted 4 lines

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_wmb();

               ^^^^^^^^^^^

What is this smp_wmb() for?

Quoted 13 lines

Quoted 6 lines

		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		WRITE_ONCE(*y, 2);

If we change this WRITE_ONCE to a relaxed atomic operation(e.g. xchg_relaxed(y, 2)), both herd and ppcmem said the exist-clause "y = 2 /\ 2:r1 = 2 /\ 2:r2 = 0" wouldn't be triggered on PPC.

I guess we will get the same behavior on ARM/ARM64, Will?

Quoted 3 lines

If a normal store could break chain, while a RmW atomic won't, do we want to call it out in the document and build our memory model around this?

I think this is required to work by C11's definition of release sequences, so any architecture that claims to support those with the same instructions will need this to be forbidden.

Personally, I think that's a bug in C11, because I think it goes too far in forbidding some hardware optimisations around relaxed xchg, but it is what it is.

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		smp_store_release(y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel memory model.

This commit therefore updates the documentation to call this vulnerability out explicitly.

Quoted 22 lines · Paul E. McKenney

If two processes are related by a RELEASE+ACQUIRE pair, ordering can be broken if a third process overwrites the value written by the RELEASE operation before the ACQUIRE operation has a chance of reading it, for example:

	P0(int *x, int *y)
	{
		WRITE_ONCE(*x, 1);
		smp_store_release(y, 1);
	}

	P1(int *y)
	{
		smp_store_release(y, 2);
	}

	P2(int *x, int *y)
	{
		r1 = smp_load_acquire(y);
		r2 = READ_ONCE(*x);
	}

Quoted 6 lines

Both ARM and powerpc allow the "after the dust settles" outcome (r1=2 && r2=0), as does the current version of the early prototype Linux-kernel memory model.

This commit therefore updates the documentation to call this vulnerability out explicitly.

So its a pretty dumb thing to do in any case (and yes the kernel does this). Its also entirely expected in my book, that if you generate conflicting writes on a release, ordering is out the window.

Why do we need to call this out? Who in his right mind would want to do this and expect anything other than wreckage?

Not that we're not having too much 'fun' discussing this,.. but I do wonder why we need to call this out.

Quoted 7 lines · Peter Zijlstra

So its a pretty dumb thing to do in any case (and yes the kernel does this). Its also entirely expected in my book, that if you generate conflicting writes on a release, ordering is out the window.

Why do we need to call this out? Who in his right mind would want to do this and expect anything other than wreckage?

Quoted 2 lines

Not that we're not having too much 'fun' discussing this,.. but I do wonder why we need to call this out.

You lost me on this one... If no one does this, sure, we can leave it out. But if some part of the kernel does rely on this, we should call it out as forbidden. And fix the kernel, of course.

Or am I missing your point?

Quoted 12 lines · Paul E. McKenney

Quoted 9 lines · Peter Zijlstra

So its a pretty dumb thing to do in any case (and yes the kernel does this). Its also entirely expected in my book, that if you generate conflicting writes on a release, ordering is out the window.

Why do we need to call this out? Who in his right mind would want to do this and expect anything other than wreckage?

Not that we're not having too much 'fun' discussing this,.. but I do wonder why we need to call this out.

Quoted 3 lines

You lost me on this one... If no one does this, sure, we can leave it out. But if some part of the kernel does rely on this, we should call it out as forbidden. And fix the kernel, of course.

Well, the kernel does this, but doesn't rely on ordering. Do "git grep zap_locks". Its disgusting, can (and does) fail and generally is a sign of badly broken code (printk is all that).

Quoted 1 line

Or am I missing your point?

My point was, its obvious crack, anybody doing this needs to have his head examined. Then again, maybe we should just say that :-)