Skip to contentSkip to conversation

buffer: Fix I/O error due to ARM read-after-read hazard

PATCH In review Vincent Whitchurch 6 participants 15 messages Nov 12, 2019 – Nov 21, 2019

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 buffer: Fix I/O error due to ARM read-after-read hazard

On my dual-core ARM Cortex-A9, reading from squashfs (over dm-verity/ubi/mtd) in a loop for hundreds of hours invariably results in a read failure in squashfs_read_data(). The errors occur because the buffer_uptodate() check fails after wait_on_buffer(). Further debugging shows that the bh was in fact uptodate and that there is no actual I/O error in the lower layers.

The problem is caused by the read-after-read hazards in the ARM Cortex-A9 MPCore (erratum #761319, see [1]). The code generated by the compiler for the combination of the wait_on_buffer() and buffer_uptodate() calls reads the flags value twice from memory (see the excerpt of the assembly below). The new value of the BH_Lock flag is seen but the new value of BH_Uptodate is not even though both the bits are read from the same memory location.

 27c:	9d08      	ldr	r5, [sp, #32]
 27e:	2400      	movs	r4, #0
 280:	e006      	b.n	290 <squashfs_read_data+0x290>
 282:	6803      	ldr	r3, [r0, #0]
 284:	07da      	lsls	r2, r3, #31
 286:	f140 810d 	bpl.w	4a4 <squashfs_read_data+0x4a4>
 28a:	3401      	adds	r4, #1
 28c:	42bc      	cmp	r4, r7
 28e:	da08      	bge.n	2a2 <squashfs_read_data+0x2a2>
 290:	f855 0f04 	ldr.w	r0, [r5, #4]!
 294:	6803      	ldr	r3, [r0, #0]
 296:	0759      	lsls	r1, r3, #29
 298:	d5f3      	bpl.n	282 <squashfs_read_data+0x282>

29a: f7ff fffe bl 0 <__wait_on_buffer>

Work around this problem by adding a DMB between the two reads of bh->flags, as recommended in the ARM document. With this barrier, no failures have been seen in more than 5000 hours of the same test.

[1] http://infocenter.arm.com/help/topic/com.arm.doc.uan0004a/UAN0004A_a9_read_read.pdf

---
v2: Reword commit message.

 include/linux/buffer_head.h | 8 ++++++++
 1 file changed, 8 insertions(+)
include/linux/buffer_head.h +8 -0
index 7b73ef7f902d..4ef909a91f8c 100644
@@ -352,6 +352,14 @@ static inline void wait_on_buffer(struct buffer_head *bh)
 	might_sleep();
 	if (buffer_locked(bh))
 		__wait_on_buffer(bh);
+
+#ifdef CONFIG_ARM
+	/*
+	 * Work around ARM Cortex-A9 MPcore Read-after-Read Hazards (erratum
+	 * 761319).
+	 */
+	smp_rmb();
+#endif
 }
 
 static inline int trylock_buffer(struct buffer_head *bh)
Signature · 1 line
2.20.0
#1 lore raw
Quoted 28 lines · Vincent Whitchurch

On my dual-core ARM Cortex-A9, reading from squashfs (over dm-verity/ubi/mtd) in a loop for hundreds of hours invariably results in a read failure in squashfs_read_data(). The errors occur because the buffer_uptodate() check fails after wait_on_buffer(). Further debugging shows that the bh was in fact uptodate and that there is no actual I/O error in the lower layers.

The problem is caused by the read-after-read hazards in the ARM Cortex-A9 MPCore (erratum #761319, see [1]). The code generated by the compiler for the combination of the wait_on_buffer() and buffer_uptodate() calls reads the flags value twice from memory (see the excerpt of the assembly below). The new value of the BH_Lock flag is seen but the new value of BH_Uptodate is not even though both the bits are read from the same memory location.

 27c:	9d08      	ldr	r5, [sp, #32]
 27e:	2400      	movs	r4, #0
 280:	e006      	b.n	290 <squashfs_read_data+0x290>
 282:	6803      	ldr	r3, [r0, #0]
 284:	07da      	lsls	r2, r3, #31
 286:	f140 810d 	bpl.w	4a4 <squashfs_read_data+0x4a4>
 28a:	3401      	adds	r4, #1
 28c:	42bc      	cmp	r4, r7
 28e:	da08      	bge.n	2a2 <squashfs_read_data+0x2a2>
 290:	f855 0f04 	ldr.w	r0, [r5, #4]!
 294:	6803      	ldr	r3, [r0, #0]
 296:	0759      	lsls	r1, r3, #29
 298:	d5f3      	bpl.n	282 <squashfs_read_data+0x282>
Quoted 7 lines

29a: f7ff fffe bl 0 <__wait_on_buffer>

Work around this problem by adding a DMB between the two reads of bh->flags, as recommended in the ARM document. With this barrier, no failures have been seen in more than 5000 hours of the same test.

[1] http://infocenter.arm.com/help/topic/com.arm.doc.uan0004a/UAN0004A_a9_read_read.pdf

I thought we were going to fix the compiler. I found an old thread here:

https://gcc.gnu.org/ml/gcc-patches/2014-06/msg00714.html

Also cc'ing Richard Earnshaw as he may been involved in the gcc discussion at the time.

While you can add some barrier here, there may be other cases where this can go wrong.

Signature · 1 line
Catalin
#2 lore raw
Quoted 3 lines · Catalin Marinas

While you can add some barrier here, there may be other cases where this can go wrong.

Yeah, that patch isn't going to be accepted. We don't add random arch workarounds to core header files like that - particularly when it's not at all clear that there aren't hundreds of other cases where that cpu errata can trigger..

Linus

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#3 lore raw
In reply to Catalin Marinas
Quoted 39 lines · Catalin Marinas
Quoted 28 lines · Vincent Whitchurch

On my dual-core ARM Cortex-A9, reading from squashfs (over dm-verity/ubi/mtd) in a loop for hundreds of hours invariably results in a read failure in squashfs_read_data(). The errors occur because the buffer_uptodate() check fails after wait_on_buffer(). Further debugging shows that the bh was in fact uptodate and that there is no actual I/O error in the lower layers.

The problem is caused by the read-after-read hazards in the ARM Cortex-A9 MPCore (erratum #761319, see [1]). The code generated by the compiler for the combination of the wait_on_buffer() and buffer_uptodate() calls reads the flags value twice from memory (see the excerpt of the assembly below). The new value of the BH_Lock flag is seen but the new value of BH_Uptodate is not even though both the bits are read from the same memory location.

 27c:	9d08      	ldr	r5, [sp, #32]
 27e:	2400      	movs	r4, #0
 280:	e006      	b.n	290 <squashfs_read_data+0x290>
 282:	6803      	ldr	r3, [r0, #0]
 284:	07da      	lsls	r2, r3, #31
 286:	f140 810d 	bpl.w	4a4 <squashfs_read_data+0x4a4>
 28a:	3401      	adds	r4, #1
 28c:	42bc      	cmp	r4, r7
 28e:	da08      	bge.n	2a2 <squashfs_read_data+0x2a2>
 290:	f855 0f04 	ldr.w	r0, [r5, #4]!
 294:	6803      	ldr	r3, [r0, #0]
 296:	0759      	lsls	r1, r3, #29
 298:	d5f3      	bpl.n	282 <squashfs_read_data+0x282>
Quoted 7 lines

29a: f7ff fffe bl 0 <__wait_on_buffer>

Work around this problem by adding a DMB between the two reads of bh->flags, as recommended in the ARM document. With this barrier, no failures have been seen in more than 5000 hours of the same test.

[1] http://infocenter.arm.com/help/topic/com.arm.doc.uan0004a/UAN0004A_a9_read_read.pdf

I thought we were going to fix the compiler. I found an old thread here:

Quoted 7 lines

https://gcc.gnu.org/ml/gcc-patches/2014-06/msg00714.html

Also cc'ing Richard Earnshaw as he may been involved in the gcc discussion at the time.

While you can add some barrier here, there may be other cases where this can go wrong.

Hmm, and afaict, even if the compiler was modified to emit LDREX instructions for volatile loads, it wouldn't help in this case because test_bit() isn't using READ_ONCE().

It's also slightly odd that the proposed patch makes the code look like:

	for (i = 0; i < b; i++) {
		if (buffer_locked(bh)) {
			__wait_on_buffer(bh);
			smp_rmb();
		}
		if (!buffer_uptodate(bh[i]))
			goto block_release;
	}

whereas there are other potential RAR orderings between buffer_locked() and __wait_on_buffer() and also probably between successive iterations of the loop.

So, really, the only way I see to solve this is for us to use READ_ONCE consistently for all relaxed atomic loads (KCSAN is starting to tread on this), and then to patch READ_ONCE to emit a DMB at runtime for arch/arm/ (maybe a static key would work if you can avoid the recursion).

I've already got patches at [1] to allow architectures to override READ_ONCE, because Alpha needs to do something similar.

Will

[1] https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=lto

#4 lore raw
Quoted 45 lines · Will Deacon
Quoted 44 lines · Catalin Marinas
Quoted 28 lines · Vincent Whitchurch

On my dual-core ARM Cortex-A9, reading from squashfs (over dm-verity/ubi/mtd) in a loop for hundreds of hours invariably results in a read failure in squashfs_read_data(). The errors occur because the buffer_uptodate() check fails after wait_on_buffer(). Further debugging shows that the bh was in fact uptodate and that there is no actual I/O error in the lower layers.

The problem is caused by the read-after-read hazards in the ARM Cortex-A9 MPCore (erratum #761319, see [1]). The code generated by the compiler for the combination of the wait_on_buffer() and buffer_uptodate() calls reads the flags value twice from memory (see the excerpt of the assembly below). The new value of the BH_Lock flag is seen but the new value of BH_Uptodate is not even though both the bits are read from the same memory location.

 27c:	9d08      	ldr	r5, [sp, #32]
 27e:	2400      	movs	r4, #0
 280:	e006      	b.n	290 <squashfs_read_data+0x290>
 282:	6803      	ldr	r3, [r0, #0]
 284:	07da      	lsls	r2, r3, #31
 286:	f140 810d 	bpl.w	4a4 <squashfs_read_data+0x4a4>
 28a:	3401      	adds	r4, #1
 28c:	42bc      	cmp	r4, r7
 28e:	da08      	bge.n	2a2 <squashfs_read_data+0x2a2>
 290:	f855 0f04 	ldr.w	r0, [r5, #4]!
 294:	6803      	ldr	r3, [r0, #0]
 296:	0759      	lsls	r1, r3, #29
 298:	d5f3      	bpl.n	282 <squashfs_read_data+0x282>
Quoted 7 lines

29a: f7ff fffe bl 0 <__wait_on_buffer>

Work around this problem by adding a DMB between the two reads of bh->flags, as recommended in the ARM document. With this barrier, no failures have been seen in more than 5000 hours of the same test.

[1] http://infocenter.arm.com/help/topic/com.arm.doc.uan0004a/UAN0004A_a9_read_read.pdf

I thought we were going to fix the compiler. I found an old thread here:

https://gcc.gnu.org/ml/gcc-patches/2014-06/msg00714.html

Also cc'ing Richard Earnshaw as he may been involved in the gcc discussion at the time.

Quoted 6 lines
Quoted 2 lines

While you can add some barrier here, there may be other cases where this can go wrong.

Hmm, and afaict, even if the compiler was modified to emit LDREX instructions for volatile loads, it wouldn't help in this case because test_bit() isn't using READ_ONCE().

I think changing volatile accesses to LDREX in gcc wasn't acceptable since they may read Device memory and not allowed on ARM.

Quoted 7 lines

It's also slightly odd that the proposed patch makes the code look like:

	for (i = 0; i < b; i++) {
		if (buffer_locked(bh)) {
			__wait_on_buffer(bh);
			smp_rmb();
		}

The proposed patch actually keeps smp_rmb() outside the 'if' block but your point below still stands.

Quoted 4 lines
		if (!buffer_uptodate(bh[i]))
			goto block_release;
	}
Quoted 8 lines

whereas there are other potential RAR orderings between buffer_locked() and __wait_on_buffer() and also probably between successive iterations of the loop.

So, really, the only way I see to solve this is for us to use READ_ONCE consistently for all relaxed atomic loads (KCSAN is starting to tread on this), and then to patch READ_ONCE to emit a DMB at runtime for arch/arm/ (maybe a static key would work if you can avoid the recursion).

OK, so this includes changing test_bit() to perform a READ_ONCE.

Signature · 1 line
Catalin
#5 lore raw
Quoted 2 lines · Catalin Marinas

OK, so this includes changing test_bit() to perform a READ_ONCE.

That's not going to happen.

Linus

#6 lore raw
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Catalin Marinas

OK, so this includes changing test_bit() to perform a READ_ONCE.

That's not going to happen.

Ok, I'll stick my neck out here, but if test_bit() is being used to read a bitmap that is being concurrently modified (e.g. by set_bit() which boils down to atomic_long_or()), then why isn't READ_ONCE() required? Right now, test_bit takes a 'const volatile unsigned long *addr' argument, so I don't see that you'll get a change in codegen except on alpha and, with this erratum, arm32.

Will

#7 lore raw
Quoted 6 lines · Will Deacon
Quoted 5 lines · Linus Torvalds
Quoted 2 lines · Catalin Marinas

OK, so this includes changing test_bit() to perform a READ_ONCE.

Quoted 8 lines
Quoted 1 line

That's not going to happen.

Ok, I'll stick my neck out here, but if test_bit() is being used to read a bitmap that is being concurrently modified (e.g. by set_bit() which boils down to atomic_long_or()), then why isn't READ_ONCE() required? Right now, test_bit takes a 'const volatile unsigned long *addr' argument, so I don't see that you'll get a change in codegen except on alpha and, with this erratum, arm32.

I'm not entirely clear what you're suggesting, so I'll just pick the scenario that I think you're talking about - but I'm not sure it's the one you're intending.

Using test_bit() in one thread and set_bit() on the same bit in another thread without locking is going to be racy by definition. It's entirely possible for:

	Thread 1			Thread 2
	bit = test_bit(...);
					set_bit(...);
	/* use bit */

and here, bit == 0 but the bit has been set by thread 2. Use of the result from test_bit() is inherently a non-atomic operation.

This is why we have test_and_set_bit() and friends that atomically test that a bit is clear before setting it. Where this is especially important is for some filesystems, as they use test_and_xxx_bit() to manage their allocation bitmaps.

Signature · 3 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up
#8 lore raw

Hi Russell,

Quoted 24 lines · Russell King - ARM Linux admin
Quoted 6 lines · Will Deacon
Quoted 5 lines · Linus Torvalds
Quoted 2 lines · Catalin Marinas

OK, so this includes changing test_bit() to perform a READ_ONCE.

Quoted 8 lines
Quoted 1 line

That's not going to happen.

Ok, I'll stick my neck out here, but if test_bit() is being used to read a bitmap that is being concurrently modified (e.g. by set_bit() which boils down to atomic_long_or()), then why isn't READ_ONCE() required? Right now, test_bit takes a 'const volatile unsigned long *addr' argument, so I don't see that you'll get a change in codegen except on alpha and, with this erratum, arm32.

I'm not entirely clear what you're suggesting, so I'll just pick the scenario that I think you're talking about - but I'm not sure it's the one you're intending.

Using test_bit() in one thread and set_bit() on the same bit in another thread without locking is going to be racy by definition. It's entirely possible for:

Quoted 7 lines
	Thread 1			Thread 2
	bit = test_bit(...);
					set_bit(...);
	/* use bit */

and here, bit == 0 but the bit has been set by thread 2. Use of the result from test_bit() is inherently a non-atomic operation.

I think it's atomic in the same way that atomic_read() is atomic (which is typically defined using READ_ONCE()).

Quoted 4 lines

This is why we have test_and_set_bit() and friends that atomically test that a bit is clear before setting it. Where this is especially important is for some filesystems, as they use test_and_xxx_bit() to manage their allocation bitmaps.

Agreed, but what we don't want is something like:

	Thread 1			Thread 2
					set_bit(...); // bit is now 1
	test_bit(...); // returns 1
	test_bit(...); // returns 0

which is what can happen due to this erratum. It's generally good practice to use READ_ONCE() when reading something which can be updated concurrently because:

* It ensures that the value is (re-)loaded from memory

* It prevents the compiler from performing harmful optimisations, such as merging or tearing (although in this case I suspect these are ok because we're dealing with a single bit)

* On Alpha, it gives you a barrier so that dependency ordering can be relied upon from the load

* It keeps KCSAN happy

I think the current definition of test_bit() gives you the first two by casting to volatile explicitly, but not the second two.

So I'm mainly curious as to the disconnect between my thinking and Linus's "That's not going to happen" comment, given that it shouldn't have an impact on architectures that don't need magic here.

Will

#9 lore raw
Quoted 5 lines · Will Deacon

which is what can happen due to this erratum. It's generally good practice to use READ_ONCE() when reading something which can be updated concurrently because:

Quoted 8 lines

* It ensures that the value is (re-)loaded from memory

* It prevents the compiler from performing harmful optimisations, such as merging or tearing (although in this case I suspect these are ok because we're dealing with a single bit)

* On Alpha, it gives you a barrier so that dependency ordering can be relied upon from the load

The Alpha barrier matters for pointers, how could it make a difference for individual bits?

Cheers,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#10 lore raw
Quoted 12 lines · Herbert Xu
Quoted 11 lines · Will Deacon

which is what can happen due to this erratum. It's generally good practice to use READ_ONCE() when reading something which can be updated concurrently because:

* It ensures that the value is (re-)loaded from memory

* It prevents the compiler from performing harmful optimisations, such as merging or tearing (although in this case I suspect these are ok because we're dealing with a single bit)

Quoted 5 lines
Quoted 2 lines

* On Alpha, it gives you a barrier so that dependency ordering can be relied upon from the load

The Alpha barrier matters for pointers, how could it make a difference for individual bits?

I guess you could use the result of test_bit to index into an array or something?

Will

#11 lore raw
Quoted 6 lines · Will Deacon
Quoted 3 lines

The Alpha barrier matters for pointers, how could it make a difference for individual bits?

I guess you could use the result of test_bit to index into an array or something?

Can Alpha Assembly even do this without using a branch?

Cheers,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#12 lore raw
Quoted 9 lines · Herbert Xu
Quoted 6 lines · Will Deacon
Quoted 3 lines

The Alpha barrier matters for pointers, how could it make a difference for individual bits?

I guess you could use the result of test_bit to index into an array or something?

Can Alpha Assembly even do this without using a branch?

Don't see why not: you can add the base address to the scaled result of test_bit and use that as the address register into a load instruction.

Will

#13 lore raw
In reply to Will Deacon
Quoted 7 lines · Will Deacon

Ok, I'll stick my neck out here, but if test_bit() is being used to read a bitmap that is being concurrently modified (e.g. by set_bit() which boils down to atomic_long_or()), then why isn't READ_ONCE() required? Right now, test_bit takes a 'const volatile unsigned long *addr' argument, so I don't see that you'll get a change in codegen except on alpha and, with this erratum, arm32.

You're right. We've moved back to actually having it volatile (or possibly never got away from it). My bad.

At one point, we tried very hard to make test_bit() perform much better when you tested multiple bits, and I thought we ended up having that merged and didn't want to regress. But apparently we never did that, or it got undone.

test_bit() is a very unfortunate interface, in that we actually use it in some situations where we _really_ would want to merge reads (not split them, but merge them). There are several cases where we do constant test-bits on the same word, and don't care about ordering. Things like thread flags etc.

It's explicitly not ordered, so *merging* reads would be right and lovely, it's the "don't read twice" that is bad. But we have no way to tell the compiler that ;(

Anyway, READ_ONCE() is ok by me when I look at the code, because as you say, it's already volatile (like my memory ;).

Linus

#14 lore raw
Quoted 6 lines · Linus Torvalds

test_bit() is a very unfortunate interface, in that we actually use it in some situations where we _really_ would want to merge reads (not split them, but merge them). There are several cases where we do constant test-bits on the same word, and don't care about ordering. Things like thread flags etc.

Side note: test_bit() really isn't good for locking in the first place. The fact that the buffer heads use it for that is very non-optimal indeed.

Particularly for testing something like "is this buffer uptodate", it should be a "smp_load_acquire()", not a test_bit(). And READ_ONCE() doesn't really help.

So in many ways it would be much better to make the buffer head stuff use proper ordered accesses. But I suspect nobody is going to ever want to go through that pain for a legacy thing, so the papering it over with READ_ONCE() and a ugly ARM hw erratum hack is probably the best we'll do..

Linus