Skip to contentSkip to conversation

selftests: KVM: Handle compiler optimizations in ucall

PATCH Applied Raghavendra Rao Ananta 5 participants 10 messages Jun 15, 2022 – Jun 17, 2022

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 selftests: KVM: Handle compiler optimizations in ucall

The selftests, when built with newer versions of clang, is found to have over optimized guests' ucall() function, and eliminating the stores for uc.cmd (perhaps due to no immediate readers). This resulted in the userspace side always reading a value of '0', and causing multiple test failures.

As a result, prevent the compiler from optimizing the stores in ucall() with WRITE_ONCE().

---
 tools/testing/selftests/kvm/lib/aarch64/ucall.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
tools/testing/selftests/kvm/lib/aarch64/ucall.c +4 -5
index e0b0164e9af8..be1d9728c4ce 100644
@@ -73,20 +73,19 @@ void ucall_uninit(struct kvm_vm *vm)
 
 void ucall(uint64_t cmd, int nargs, ...)
 {
-	struct ucall uc = {
-		.cmd = cmd,
-	};
+	struct ucall uc = {};
 	va_list va;
 	int i;
 
+	WRITE_ONCE(uc.cmd, cmd);
 	nargs = nargs <= UCALL_MAX_ARGS ? nargs : UCALL_MAX_ARGS;
 
 	va_start(va, nargs);
 	for (i = 0; i < nargs; ++i)
-		uc.args[i] = va_arg(va, uint64_t);
+		WRITE_ONCE(uc.args[i], va_arg(va, uint64_t));
 	va_end(va);
 
-	*ucall_exit_mmio_addr = (vm_vaddr_t)&uc;
+	WRITE_ONCE(*ucall_exit_mmio_addr, (vm_vaddr_t)&uc);
 }
 
 uint64_t get_ucall(struct kvm_vm *vm, uint32_t vcpu_id, struct ucall *uc)
Signature · 1 line
2.36.1.476.g0c4daa206d-goog
Reviewed-by on v1 1/1
#1 lore raw
Quoted 48 lines · Raghavendra Rao Ananta

The selftests, when built with newer versions of clang, is found to have over optimized guests' ucall() function, and eliminating the stores for uc.cmd (perhaps due to no immediate readers). This resulted in the userspace side always reading a value of '0', and causing multiple test failures.

As a result, prevent the compiler from optimizing the stores in ucall() with WRITE_ONCE().

---
 tools/testing/selftests/kvm/lib/aarch64/ucall.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
tools/testing/selftests/kvm/lib/aarch64/ucall.c +4 -5
index e0b0164e9af8..be1d9728c4ce 100644
@@ -73,20 +73,19 @@ void ucall_uninit(struct kvm_vm *vm)
 
 void ucall(uint64_t cmd, int nargs, ...)
 {
-	struct ucall uc = {
-		.cmd = cmd,
-	};
+	struct ucall uc = {};
 	va_list va;
 	int i;
 
+	WRITE_ONCE(uc.cmd, cmd);
 	nargs = nargs <= UCALL_MAX_ARGS ? nargs : UCALL_MAX_ARGS;
 
 	va_start(va, nargs);
 	for (i = 0; i < nargs; ++i)
-		uc.args[i] = va_arg(va, uint64_t);
+		WRITE_ONCE(uc.args[i], va_arg(va, uint64_t));
 	va_end(va);
 
-	*ucall_exit_mmio_addr = (vm_vaddr_t)&uc;
+	WRITE_ONCE(*ucall_exit_mmio_addr, (vm_vaddr_t)&uc);
 }
 
 uint64_t get_ucall(struct kvm_vm *vm, uint32_t vcpu_id, struct ucall *uc)
Signature · 1 line
2.36.1.476.g0c4daa206d-goog

Thanks,
drew

#2 lore raw
In reply to Andrew Jones

From: Andrew Jones

Quoted 44 lines

Sent: 16 June 2022 13:03

Quoted 9 lines · Raghavendra Rao Ananta

The selftests, when built with newer versions of clang, is found to have over optimized guests' ucall() function, and eliminating the stores for uc.cmd (perhaps due to no immediate readers). This resulted in the userspace side always reading a value of '0', and causing multiple test failures.

As a result, prevent the compiler from optimizing the stores in ucall() with WRITE_ONCE().

Review

Quoted 8 lines
---
 tools/testing/selftests/kvm/lib/aarch64/ucall.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
(quoted diff) +0 -0

b/tools/testing/selftests/kvm/lib/aarch64/ucall.c

Quoted 23 lines

index e0b0164e9af8..be1d9728c4ce 100644

tools/testing/selftests/kvm/lib/aarch64/ucall.c +3 -4
@@ -73,20 +73,19 @@ void ucall_uninit(struct kvm_vm *vm)
 
 void ucall(uint64_t cmd, int nargs, ...)
 {
-	struct ucall uc = {
-		.cmd = cmd,
-	};
+	struct ucall uc = {};
 	va_list va;
 	int i;
 
+	WRITE_ONCE(uc.cmd, cmd);
 	nargs = nargs <= UCALL_MAX_ARGS ? nargs : UCALL_MAX_ARGS;
 
 	va_start(va, nargs);
 	for (i = 0; i < nargs; ++i)
-		uc.args[i] = va_arg(va, uint64_t);
+		WRITE_ONCE(uc.args[i], va_arg(va, uint64_t));
 	va_end(va);
 
Quoted 3 lines

tools/testing/selftests/kvm/lib/aarch64/ucall.c void ucall_uninit(struct kvm_vm *vm) · line 89

Quoted 3 lines
tools/testing/selftests/kvm/lib/aarch64/ucall.c +1 -1
-	*ucall_exit_mmio_addr = (vm_vaddr_t)&uc;
+	WRITE_ONCE(*ucall_exit_mmio_addr, (vm_vaddr_t)&uc);
 }

Am I misreading things again?
That function looks like it writes the address of an on-stack item into global data.

Maybe 'uc' ought to be static?

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#3 lore raw
Quoted 45 lines · David Laight

From: Andrew Jones

Quoted 44 lines

Sent: 16 June 2022 13:03

Quoted 9 lines · Raghavendra Rao Ananta

The selftests, when built with newer versions of clang, is found to have over optimized guests' ucall() function, and eliminating the stores for uc.cmd (perhaps due to no immediate readers). This resulted in the userspace side always reading a value of '0', and causing multiple test failures.

As a result, prevent the compiler from optimizing the stores in ucall() with WRITE_ONCE().

Review

Quoted 8 lines
---
 tools/testing/selftests/kvm/lib/aarch64/ucall.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
(quoted diff) +0 -0

b/tools/testing/selftests/kvm/lib/aarch64/ucall.c

Quoted 23 lines

index e0b0164e9af8..be1d9728c4ce 100644

tools/testing/selftests/kvm/lib/aarch64/ucall.c +3 -4
@@ -73,20 +73,19 @@ void ucall_uninit(struct kvm_vm *vm)
 
 void ucall(uint64_t cmd, int nargs, ...)
 {
-	struct ucall uc = {
-		.cmd = cmd,
-	};
+	struct ucall uc = {};
 	va_list va;
 	int i;
 
+	WRITE_ONCE(uc.cmd, cmd);
 	nargs = nargs <= UCALL_MAX_ARGS ? nargs : UCALL_MAX_ARGS;
 
 	va_start(va, nargs);
 	for (i = 0; i < nargs; ++i)
-		uc.args[i] = va_arg(va, uint64_t);
+		WRITE_ONCE(uc.args[i], va_arg(va, uint64_t));
 	va_end(va);
 
Quoted 7 lines
Quoted 3 lines

tools/testing/selftests/kvm/lib/aarch64/ucall.c void ucall_uninit(struct kvm_vm *vm) · line 89

Quoted 3 lines
tools/testing/selftests/kvm/lib/aarch64/ucall.c +1 -1
-	*ucall_exit_mmio_addr = (vm_vaddr_t)&uc;
+	WRITE_ONCE(*ucall_exit_mmio_addr, (vm_vaddr_t)&uc);
 }

Am I misreading things again?
That function looks like it writes the address of an on-stack item into global data.

The write to the address that the global points at causes a switch from guest to host context. The guest's stack remains intact while executing host code and the host can access the uc stack variable directly by its address. Take a look at lib/aarch64/ucall.c to see all the details.

Thanks,
drew

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

#4 lore raw

From: Andrew Jones

Quoted 56 lines

Sent: 16 June 2022 17:26

Quoted 45 lines · David Laight

From: Andrew Jones

Quoted 44 lines

Sent: 16 June 2022 13:03

Quoted 9 lines · Raghavendra Rao Ananta

The selftests, when built with newer versions of clang, is found to have over optimized guests' ucall() function, and eliminating the stores for uc.cmd (perhaps due to no immediate readers). This resulted in the userspace side always reading a value of '0', and causing multiple test failures.

As a result, prevent the compiler from optimizing the stores in ucall() with WRITE_ONCE().

Review

Quoted 8 lines
---
 tools/testing/selftests/kvm/lib/aarch64/ucall.c | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
(quoted diff) +0 -0

b/tools/testing/selftests/kvm/lib/aarch64/ucall.c

Quoted 23 lines

index e0b0164e9af8..be1d9728c4ce 100644

tools/testing/selftests/kvm/lib/aarch64/ucall.c +3 -4
@@ -73,20 +73,19 @@ void ucall_uninit(struct kvm_vm *vm)
 
 void ucall(uint64_t cmd, int nargs, ...)
 {
-	struct ucall uc = {
-		.cmd = cmd,
-	};
+	struct ucall uc = {};
 	va_list va;
 	int i;
 
+	WRITE_ONCE(uc.cmd, cmd);
 	nargs = nargs <= UCALL_MAX_ARGS ? nargs : UCALL_MAX_ARGS;
 
 	va_start(va, nargs);
 	for (i = 0; i < nargs; ++i)
-		uc.args[i] = va_arg(va, uint64_t);
+		WRITE_ONCE(uc.args[i], va_arg(va, uint64_t));
 	va_end(va);
 
Quoted 7 lines
Quoted 3 lines

tools/testing/selftests/kvm/lib/aarch64/ucall.c void ucall_uninit(struct kvm_vm *vm) · line 89

Quoted 3 lines
tools/testing/selftests/kvm/lib/aarch64/ucall.c +1 -1
-	*ucall_exit_mmio_addr = (vm_vaddr_t)&uc;
+	WRITE_ONCE(*ucall_exit_mmio_addr, (vm_vaddr_t)&uc);
 }

Am I misreading things again?
That function looks like it writes the address of an on-stack item into global data.

Quoted 5 lines

The write to the address that the global points at causes a switch from guest to host context. The guest's stack remains intact while executing host code and the host can access the uc stack variable directly by its address. Take a look at lib/aarch64/ucall.c to see all the details.

No wonder I was confused.
It's not surprising the compiler optimises it all away.

It doesn't seem right to be 'abusing' WRITE_ONCE() here. Just adding barrier() should be enough and much more descriptive.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#5 lore raw
Quoted 5 lines · David Laight

No wonder I was confused.
It's not surprising the compiler optimises it all away.

It doesn't seem right to be 'abusing' WRITE_ONCE() here. Just adding barrier() should be enough and much more descriptive.

I had the same thought, although I do not believe barrier() is sufficient on its own. barrier_data() with a pointer to uc passed through is required to keep clang from eliminating the dead store.

--
Thanks,
Oliver

#6 lore raw

From: [email protected]

Quoted 7 lines

Sent: 16 June 2022 19:45

Quoted 3 lines · David Laight

No wonder I was confused.
It's not surprising the compiler optimises it all away.

Quoted 6 lines
Quoted 2 lines

It doesn't seem right to be 'abusing' WRITE_ONCE() here. Just adding barrier() should be enough and much more descriptive.

I had the same thought, although I do not believe barrier() is sufficient on its own. barrier_data() with a pointer to uc passed through is required to keep clang from eliminating the dead store.

A barrier() (full memory clobber) ought to be stronger than the partial one than barrier_data() generates.

I can't quite decide whether you need a barrier() both sides of the 'magic write'.
Plausibly the compiler could discard the on-stack data after the barrier() and before the 'magic write'.

Certainly putting the 'magic write' inside a asm block that has a memory clobber is a more correct solution.

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#7 lore raw
Quoted 18 lines · David Laight

From: [email protected]

Quoted 7 lines

Sent: 16 June 2022 19:45

Quoted 3 lines · David Laight

No wonder I was confused.
It's not surprising the compiler optimises it all away.

Quoted 6 lines
Quoted 2 lines

It doesn't seem right to be 'abusing' WRITE_ONCE() here. Just adding barrier() should be enough and much more descriptive.

I had the same thought, although I do not believe barrier() is sufficient on its own. barrier_data() with a pointer to uc passed through is required to keep clang from eliminating the dead store.

A barrier() (full memory clobber) ought to be stronger than the partial one than barrier_data() generates.

Quoted 7 lines

I can't quite decide whether you need a barrier() both sides of the 'magic write'.
Plausibly the compiler could discard the on-stack data after the barrier() and before the 'magic write'.

Certainly putting the 'magic write' inside a asm block that has a memory clobber is a more correct solution.

Indeed, since the magic write is actually a guest MMIO write, then it should be using writeq().

Thanks,
drew

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

maintainer
#8 lore raw
Quoted 24 lines · Andrew Jones
Quoted 23 lines · David Laight

From: [email protected]

Quoted 7 lines

Sent: 16 June 2022 19:45

Quoted 3 lines · David Laight

No wonder I was confused.
It's not surprising the compiler optimises it all away.

Quoted 6 lines
Quoted 2 lines

It doesn't seem right to be 'abusing' WRITE_ONCE() here. Just adding barrier() should be enough and much more descriptive.

I had the same thought, although I do not believe barrier() is sufficient on its own. barrier_data() with a pointer to uc passed through is required to keep clang from eliminating the dead store.

A barrier() (full memory clobber) ought to be stronger than the partial one than barrier_data() generates.

I can't quite decide whether you need a barrier() both sides of the 'magic write'.
Plausibly the compiler could discard the on-stack data after the barrier() and before the 'magic write'.

Quoted 5 lines
Quoted 2 lines

Certainly putting the 'magic write' inside a asm block that has a memory clobber is a more correct solution.

Indeed, since the magic write is actually a guest MMIO write, then it should be using writeq().

It doesn't need to use writeq() because no special precautions are needed with respect to cacheability or instruction reordering (as is the case with hardware registers).

WRITE_ONCE is okay, especially since the code never reads it (and if it did it would also use READ_ONCE).

Paolo

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm