Skip to contentSkip to conversation

rcutorture: Tweak kvm options

PATCH New bigeasy at linutronix.de (Sebastian Andrzej Siewior) 1 participant 13 messages Apr 24, 2019 – May 3, 2019

Subsystems (3)
lore
Share & tools
lore

bigeasy at linutronix.de (Sebastian Andrzej Siewior)

#0 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

In one of my rcutorture tests the TSC clocksource got marked unstable due to a large difference in the TSC value. I'm not sure if the guest run for a long time with disabled interrupts or if the host was very busy and didn't schedule the guest for some time.
I took a look on the qemu/KVM options and decided to update the options:
- Use kvm{32|64} as CPU. We could probably use `host' (like ARM does) for maximum available features but since we don't run any userland I'm not sure if it makes any difference.

- Drop the "noapic" option, enable TSC deadline timer. There is no history why the APIC was disabled, I see no reason for it. The deadline timer is probably "nicer".

- Additional config options. It ensures that the kernel knowns that it runs as a kvm guest and can use virt devices like the kvm-clock as clocksource. The kvm-clock was the main motivation here.

- I didn't add a random HW device. It would make the random device ready earlier (not it doesn't complete the initialisation at all) but I doubt that there is any need for this.

Signed-off-by Sebastian Andrzej Siewior <bigeasy at linutronix.de>
---
 tools/testing/selftests/rcutorture/bin/functions.sh | 13 ++++++++++++-
 .../selftests/rcutorture/configs/rcu/CFcommon       |  4 ++++
 2 files changed, 16 insertions(+), 1 deletion(-)
tools/testing/selftests/rcutorture/bin/functions.sh +12 -1
index 6bcb8b5b2ff22..be3c5c73d7e79 100644
@@ -172,7 +172,7 @@ identify_qemu_append () {
 	local console=ttyS0
 	case "$1" in
 	qemu-system-x86_64|qemu-system-i386)
-		echo noapic selinux=0 initcall_debug debug
+		echo selinux=0 initcall_debug debug
 		;;
 	qemu-system-aarch64)
 		console=ttyAMA0
@@ -191,8 +191,19 @@ identify_qemu_append () {
 # Output arguments for qemu arguments based on the TORTURE_QEMU_MAC
 # and TORTURE_QEMU_INTERACTIVE environment variables.
 identify_qemu_args () {
+	local KVM_CPU=""
+	case "$1" in
+	qemu-system-x86_64)
+		KVM_CPU=kvm64
+		;;
+	qemu-system-i386)
+		KVM_CPU=kvm32
+		;;
+	esac
 	case "$1" in
 	qemu-system-x86_64|qemu-system-i386)
+		echo -machine q35,accel=kvm
+		echo -cpu ${KVM_CPU},x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on
 		;;
 	qemu-system-aarch64)
 		echo -machine virt,gic-version=host -cpu host
tools/testing/selftests/rcutorture/configs/rcu/CFcommon +4 -0
index d2d2a86139db1..322d5d40443cd 100644
@@ -1,2 +1,6 @@
 CONFIG_RCU_TORTURE_TEST=y
 CONFIG_PRINTK_TIME=y
+CONFIG_HYPERVISOR_GUEST=y
+CONFIG_PARAVIRT=y
+CONFIG_PARAVIRT_SPINLOCKS=y
+CONFIG_KVM_GUEST=y
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 17 lines · Sebastian Andrzej Siewior

In one of my rcutorture tests the TSC clocksource got marked unstable due to a large difference in the TSC value. I'm not sure if the guest run for a long time with disabled interrupts or if the host was very busy and didn't schedule the guest for some time.
I took a look on the qemu/KVM options and decided to update the options:
- Use kvm{32|64} as CPU. We could probably use `host' (like ARM does) for maximum available features but since we don't run any userland I'm not sure if it makes any difference.

- Drop the "noapic" option, enable TSC deadline timer. There is no history why the APIC was disabled, I see no reason for it. The deadline timer is probably "nicer".

- Additional config options. It ensures that the kernel knowns that it runs as a kvm guest and can use virt devices like the kvm-clock as clocksource. The kvm-clock was the main motivation here.

Quoted 5 lines

- I didn't add a random HW device. It would make the random device ready earlier (not it doesn't complete the initialisation at all) but I doubt that there is any need for this.

Signed-off-by Sebastian Andrzej Siewior <bigeasy at linutronix.de>

Thank you, Sebastian! Queued for review and testing.

Thanx, Paul

Quoted 52 lines
---
 tools/testing/selftests/rcutorture/bin/functions.sh | 13 ++++++++++++-
 .../selftests/rcutorture/configs/rcu/CFcommon       |  4 ++++
 2 files changed, 16 insertions(+), 1 deletion(-)
tools/testing/selftests/rcutorture/bin/functions.sh +12 -1
index 6bcb8b5b2ff22..be3c5c73d7e79 100644
@@ -172,7 +172,7 @@ identify_qemu_append () {
 	local console=ttyS0
 	case "$1" in
 	qemu-system-x86_64|qemu-system-i386)
-		echo noapic selinux=0 initcall_debug debug
+		echo selinux=0 initcall_debug debug
 		;;
 	qemu-system-aarch64)
 		console=ttyAMA0
@@ -191,8 +191,19 @@ identify_qemu_append () {
 # Output arguments for qemu arguments based on the TORTURE_QEMU_MAC
 # and TORTURE_QEMU_INTERACTIVE environment variables.
 identify_qemu_args () {
+	local KVM_CPU=""
+	case "$1" in
+	qemu-system-x86_64)
+		KVM_CPU=kvm64
+		;;
+	qemu-system-i386)
+		KVM_CPU=kvm32
+		;;
+	esac
 	case "$1" in
 	qemu-system-x86_64|qemu-system-i386)
+		echo -machine q35,accel=kvm
+		echo -cpu ${KVM_CPU},x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on
 		;;
 	qemu-system-aarch64)
 		echo -machine virt,gic-version=host -cpu host
tools/testing/selftests/rcutorture/configs/rcu/CFcommon +4 -0
index d2d2a86139db1..322d5d40443cd 100644
@@ -1,2 +1,6 @@
 CONFIG_RCU_TORTURE_TEST=y
 CONFIG_PRINTK_TIME=y
+CONFIG_HYPERVISOR_GUEST=y
+CONFIG_PARAVIRT=y
+CONFIG_PARAVIRT_SPINLOCKS=y
+CONFIG_KVM_GUEST=y
Signature · 1 line
2.20.1
#2 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 18 lines · Paul E. McKenney
Quoted 17 lines · Sebastian Andrzej Siewior

In one of my rcutorture tests the TSC clocksource got marked unstable due to a large difference in the TSC value. I'm not sure if the guest run for a long time with disabled interrupts or if the host was very busy and didn't schedule the guest for some time.
I took a look on the qemu/KVM options and decided to update the options:
- Use kvm{32|64} as CPU. We could probably use `host' (like ARM does) for maximum available features but since we don't run any userland I'm not sure if it makes any difference.

- Drop the "noapic" option, enable TSC deadline timer. There is no history why the APIC was disabled, I see no reason for it. The deadline timer is probably "nicer".

- Additional config options. It ensures that the kernel knowns that it runs as a kvm guest and can use virt devices like the kvm-clock as clocksource. The kvm-clock was the main motivation here.

Quoted 7 lines
Quoted 5 lines

- I didn't add a random HW device. It would make the random device ready earlier (not it doesn't complete the initialisation at all) but I doubt that there is any need for this.

Signed-off-by Sebastian Andrzej Siewior <bigeasy at linutronix.de>

Thank you, Sebastian! Queued for review and testing.

And it doesn't like my (admittedly ancient) QEMU, complaining about not knowing about "x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on". If I remove these, it works. I will be upgrading soon (famous last words), so what I am going to do is queue the following separate not-for-upstream patch that makes it work on my setup.

Thanx, Paul

------------------------------------------------------------------------

commit d33dffe7d1f05adfc798836a9d30b9f6bf69635d

Author: Paul E. McKenney <paulmck at linux.ibm.com>
Date:   Wed Apr 24 05:20:07 2019 -0700

EXP torture: Hack patch to support ancient QEMU

To be removed before upstreaming or when Paul gets his QEMU act together, whichever comes first. ;-)

Signed-off-by Paul E. McKenney <paulmck at linux.ibm.com>
tools/testing/selftests/rcutorture/bin/functions.sh +1 -1
index be3c5c73d7e7..c3a49fb4d6f6 100644
@@ -203,7 +203,7 @@ identify_qemu_args () {
 	case "$1" in
 	qemu-system-x86_64|qemu-system-i386)
 		echo -machine q35,accel=kvm
-		echo -cpu ${KVM_CPU},x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on
+		echo -cpu ${KVM_CPU}
 		;;
 	qemu-system-aarch64)
 		echo -machine virt,gic-version=host -cpu host
#3 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 25 lines · Paul E. McKenney
Quoted 24 lines · Paul E. McKenney
Quoted 22 lines · Sebastian Andrzej Siewior

In one of my rcutorture tests the TSC clocksource got marked unstable due to a large difference in the TSC value. I'm not sure if the guest run for a long time with disabled interrupts or if the host was very busy and didn't schedule the guest for some time.
I took a look on the qemu/KVM options and decided to update the options:
- Use kvm{32|64} as CPU. We could probably use `host' (like ARM does) for maximum available features but since we don't run any userland I'm not sure if it makes any difference.

- Drop the "noapic" option, enable TSC deadline timer. There is no history why the APIC was disabled, I see no reason for it. The deadline timer is probably "nicer".

- Additional config options. It ensures that the kernel knowns that it runs as a kvm guest and can use virt devices like the kvm-clock as clocksource. The kvm-clock was the main motivation here.

- I didn't add a random HW device. It would make the random device ready earlier (not it doesn't complete the initialisation at all) but I doubt that there is any need for this.

Signed-off-by Sebastian Andrzej Siewior <bigeasy at linutronix.de>
Quoted 7 lines
Quoted 1 line

Thank you, Sebastian! Queued for review and testing.

And it doesn't like my (admittedly ancient) QEMU, complaining about not knowing about "x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on". If I remove these, it works. I will be upgrading soon (famous last words), so what I am going to do is queue the following separate not-for-upstream patch that makes it work on my setup.

Also, the !SMP scenarios get this:

:CONFIG_PARAVIRT_SPINLOCKS=y: improperly set

Would it make sense to only set this on CONFIG_SMP=y runs? The easy way to do this is to move it from CFcommon to the scenario files not having CONFIG_SMP=n. Or would something else work better?

Or am I doing something wrong?

Thanx, Paul

#4 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 9 lines · Paul E. McKenney
Quoted 5 lines

And it doesn't like my (admittedly ancient) QEMU, complaining about not knowing about "x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on". If I remove these, it works. I will be upgrading soon (famous last words), so what I am going to do is queue the following separate not-for-upstream patch that makes it work on my setup.

Also, the !SMP scenarios get this:

:CONFIG_PARAVIRT_SPINLOCKS=y: improperly set

I ignored that (because there is a CHECK option to ensure that certain options are set).

Quoted 5 lines

Would it make sense to only set this on CONFIG_SMP=y runs? The easy way to do this is to move it from CFcommon to the scenario files not having CONFIG_SMP=n. Or would something else work better?

Or am I doing something wrong?

You should also get that warning on ARM for instance since those PARAVIRT SPINLOCKS are only implemented on x86.
I have no numbers to compare the performance with and without that option. So we could drop that option or you tell me where to look after a run with/without that option so I can tell if it is worth the effort.

Quoted 1 line

Thanx, Paul

Sebastian

#5 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 7 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Paul E. McKenney
Quoted 5 lines

And it doesn't like my (admittedly ancient) QEMU, complaining about not knowing about "x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on". If I remove these, it works. I will be upgrading soon (famous last words), so what I am going to do is queue the following separate not-for-upstream patch that makes it work on my setup.

Quoted 6 lines
Quoted 3 lines

Also, the !SMP scenarios get this:

:CONFIG_PARAVIRT_SPINLOCKS=y: improperly set

I ignored that (because there is a CHECK option to ensure that certain options are set).

True enough, but these things are a bit like compiler warnings in that it is good to have a clean run.

Quoted 3 lines
Quoted 3 lines

Would it make sense to only set this on CONFIG_SMP=y runs? The easy way to do this is to move it from CFcommon to the scenario files not having CONFIG_SMP=n.

(Which is how I am currently running, I should have added.)

Quoted 6 lines
Quoted 3 lines

having CONFIG_SMP=n. Or would something else work better?

Or am I doing something wrong?

You should also get that warning on ARM for instance since those PARAVIRT SPINLOCKS are only implemented on x86.

OK, so we would want this to only be added for x86 runs.

Quoted 3 lines

I have no numbers to compare the performance with and without that option. So we could drop that option or you tell me where to look after a run with/without that option so I can tell if it is worth the effort.

One place to look is in the summary output:

TREE01 ------- 17540 GPs (58.4667/s) [rcu: g130629 f0x0 ]

The "58.4667/s" is the number of grace periods per second. I would be surprised if CONFIG_PARAVIRT_SPINLOCKS made a noticeable difference in grace-period rate (given the natural variation), but you never know.

Thanx, Paul

#6 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 7 lines · Paul E. McKenney

One place to look is in the summary output:

TREE01 ------- 17540 GPs (58.4667/s) [rcu: g130629 f0x0 ]

The "58.4667/s" is the number of grace periods per second. I would be surprised if CONFIG_PARAVIRT_SPINLOCKS made a noticeable difference in grace-period rate (given the natural variation), but you never know.

I did four runs of the different parts of the patch:
- 5.1-rc6
- #1 + kvm64 CPU + some config options
- #2 + tsc-deadline=on and so on (the whole line)
- #3 + CONFIG_PARAVIRT_SPINLOCKS (now everything)

the test command was

	tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 112 --duration 60 --configs "16*TREE08" --memory 4G

and the results:

| HEAD is now at 085b7755808a... Linux 5.1-rc6
| (28.5942 +27.4658 +28.0203 +27.2061 +28.0731 +26.9078 +27.8494 + 27.3392 +26.4339 +28.025 +27.4797 +27.6775 +28.0653 +28.0742 +27.9581 +28.6508)/ 16
| 27.738775
| 
| HEAD is now at 36a12aa9761a... tune #1
| (28.5761 +26.6514 +26.6989 +27.4375 +27.3442 +28.3228 +26.6353 +27.5461+28.5531 +27.7006 +27.8078 +27.9753 +27.4269 +28.0464 +27.6314 +27.8356) / 16
| 27.6368375
| 
| HEAD is now at af5cd7196436... tune #2
| (28.4867 +26.3675 +27.6364 +28.3344 +27.4153 +27.9306 +27.1703 +26.8461+27.3194 +28.5486 +27.8975 +27.4356 +28.12 +28.4397 +29.0186 +26.9328 )/ 16
| 27.74371875
| 
| HEAD is now at 3701f64943f5... tune #3
| (28.2431 +27.7831 +28.39 +28.2586 +27.7408 +27.9258 +26.6236 +26.7817+29.1178 +26.9564 +29.0525 +27.4258 +27.4931 +27.8928 +26.9308 +28.4833)/ 16
| 27.8187

This 28.… is the number of GP/s. Based on the results in looks like noise to me. Also I have no idea why you have more than twice as many GP/s as I do.
Different box doing:

	tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 112 --duration 60 --configs "42*TREE01" --memory 4G

and got this:

| HEAD is now at 085b7755808aa... Linux 5.1-rc6
| (15.2878 + 15.8664 + 15.6369 + 15.6714 + 15.3667 + 16.4706 + 15.7844 +
| 16.2119 + 15.6108 + 15.84 + 16.0003 + 16.2886 + 15.8728 + 15.5347 +
| 15.6753 + 15.6628 + 15.8628 + 15.8158 + 15.8419 + 16.0053 + 15.7878 +
| 16.465 + 16.2267 + 16.6881 + 16.3186 + 16.1778 + 15.7069 + 16.0178 +
| 15.7156 + 16.0083 + 15.7181 + 15.8961 + 15.5253 + 16.1569 + 15.7692 +
| 16.2622 + 16.2931 + 15.9531 + 15.6697 + 15.4539 + 15.6478 + 15.8047) /
| 42
|	~15.89452142857142857143
|
| HEAD is now at 3701f64943f5a... tune #3
| ; (15.8461 + 15.8653 + 16.0392 + 15.8906 + 15.7606 + 15.6169 + 16.1425 +
| 15.9089 + 16.2169 + 16.1694 + 16.2122 + 15.6417 + 15.8022 + 16.1178 +
| 15.1689 + 16.1303 + 16.0181 + 16.3797 + 16.0614 + 16.2839 + 15.4583 +
| 15.9178 + 16.0589 + 16.3428 + 15.5486 + 16.0839 + 15.9931 + 15.8417 +
| 16.0981 + 15.8075 + 15.9925 + 15.7311 + 15.9172 + 16.1164 + 15.6303 +
| 15.9383 + 16.0714 + 16.2786 + 15.8394 + 15.9597 + 16.0175 + 15.3908) /
| 42
|	~15.93586904761904761905

Noise.

So from RCUtorture point of view there is no improvement right? In that case I would suggest to drop the problematic parts…

Quoted 1 line

Thanx, Paul

Sebastian

#7 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 35 lines · Sebastian Andrzej Siewior
Quoted 7 lines · Paul E. McKenney

One place to look is in the summary output:

TREE01 ------- 17540 GPs (58.4667/s) [rcu: g130629 f0x0 ]

The "58.4667/s" is the number of grace periods per second. I would be surprised if CONFIG_PARAVIRT_SPINLOCKS made a noticeable difference in grace-period rate (given the natural variation), but you never know.

I did four runs of the different parts of the patch:
- 5.1-rc6
- #1 + kvm64 CPU + some config options
- #2 + tsc-deadline=on and so on (the whole line)
- #3 + CONFIG_PARAVIRT_SPINLOCKS (now everything)

the test command was

	tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 112 --duration 60 --configs "16*TREE08" --memory 4G

and the results:

| HEAD is now at 085b7755808a... Linux 5.1-rc6
| (28.5942 +27.4658 +28.0203 +27.2061 +28.0731 +26.9078 +27.8494 + 27.3392 +26.4339 +28.025 +27.4797 +27.6775 +28.0653 +28.0742 +27.9581 +28.6508)/ 16
| 27.738775
| 
| HEAD is now at 36a12aa9761a... tune #1
| (28.5761 +26.6514 +26.6989 +27.4375 +27.3442 +28.3228 +26.6353 +27.5461+28.5531 +27.7006 +27.8078 +27.9753 +27.4269 +28.0464 +27.6314 +27.8356) / 16
| 27.6368375
| 
| HEAD is now at af5cd7196436... tune #2
| (28.4867 +26.3675 +27.6364 +28.3344 +27.4153 +27.9306 +27.1703 +26.8461+27.3194 +28.5486 +27.8975 +27.4356 +28.12 +28.4397 +29.0186 +26.9328 )/ 16
| 27.74371875
| 
| HEAD is now at 3701f64943f5... tune #3
| (28.2431 +27.7831 +28.39 +28.2586 +27.7408 +27.9258 +26.6236 +26.7817+29.1178 +26.9564 +29.0525 +27.4258 +27.4931 +27.8928 +26.9308 +28.4833)/ 16
| 27.8187
Quoted 3 lines

This 28.… is the number of GP/s. Based on the results in looks like noise to me. Also I have no idea why you have more than twice as many GP/s as I do.

My guess is that because you have more CPUs, the for_each_online_cpu() loop takes longer on your system.

Quoted 24 lines

Different box doing:

	tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 112 --duration 60 --configs "42*TREE01" --memory 4G

and got this:

| HEAD is now at 085b7755808aa... Linux 5.1-rc6
| (15.2878 + 15.8664 + 15.6369 + 15.6714 + 15.3667 + 16.4706 + 15.7844 +
| 16.2119 + 15.6108 + 15.84 + 16.0003 + 16.2886 + 15.8728 + 15.5347 +
| 15.6753 + 15.6628 + 15.8628 + 15.8158 + 15.8419 + 16.0053 + 15.7878 +
| 16.465 + 16.2267 + 16.6881 + 16.3186 + 16.1778 + 15.7069 + 16.0178 +
| 15.7156 + 16.0083 + 15.7181 + 15.8961 + 15.5253 + 16.1569 + 15.7692 +
| 16.2622 + 16.2931 + 15.9531 + 15.6697 + 15.4539 + 15.6478 + 15.8047) /
| 42
|	~15.89452142857142857143
|
| HEAD is now at 3701f64943f5a... tune #3
| ; (15.8461 + 15.8653 + 16.0392 + 15.8906 + 15.7606 + 15.6169 + 16.1425 +
| 15.9089 + 16.2169 + 16.1694 + 16.2122 + 15.6417 + 15.8022 + 16.1178 +
| 15.1689 + 16.1303 + 16.0181 + 16.3797 + 16.0614 + 16.2839 + 15.4583 +
| 15.9178 + 16.0589 + 16.3428 + 15.5486 + 16.0839 + 15.9931 + 15.8417 +
| 16.0981 + 15.8075 + 15.9925 + 15.7311 + 15.9172 + 16.1164 + 15.6303 +
| 15.9383 + 16.0714 + 16.2786 + 15.8394 + 15.9597 + 16.0175 + 15.3908) /
| 42
|	~15.93586904761904761905
Quoted 4 lines

Noise.

So from RCUtorture point of view there is no improvement right? In that case I would suggest to drop the problematic parts…

Thank you for testing this! I will adjust accordingly.

Thanx, Paul

#8 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 36 lines · Paul E. McKenney
Quoted 35 lines · Sebastian Andrzej Siewior
Quoted 7 lines · Paul E. McKenney

One place to look is in the summary output:

TREE01 ------- 17540 GPs (58.4667/s) [rcu: g130629 f0x0 ]

The "58.4667/s" is the number of grace periods per second. I would be surprised if CONFIG_PARAVIRT_SPINLOCKS made a noticeable difference in grace-period rate (given the natural variation), but you never know.

I did four runs of the different parts of the patch:
- 5.1-rc6
- #1 + kvm64 CPU + some config options
- #2 + tsc-deadline=on and so on (the whole line)
- #3 + CONFIG_PARAVIRT_SPINLOCKS (now everything)

the test command was

	tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 112 --duration 60 --configs "16*TREE08" --memory 4G

and the results:

| HEAD is now at 085b7755808a... Linux 5.1-rc6
| (28.5942 +27.4658 +28.0203 +27.2061 +28.0731 +26.9078 +27.8494 + 27.3392 +26.4339 +28.025 +27.4797 +27.6775 +28.0653 +28.0742 +27.9581 +28.6508)/ 16
| 27.738775
| 
| HEAD is now at 36a12aa9761a... tune #1
| (28.5761 +26.6514 +26.6989 +27.4375 +27.3442 +28.3228 +26.6353 +27.5461+28.5531 +27.7006 +27.8078 +27.9753 +27.4269 +28.0464 +27.6314 +27.8356) / 16
| 27.6368375
| 
| HEAD is now at af5cd7196436... tune #2
| (28.4867 +26.3675 +27.6364 +28.3344 +27.4153 +27.9306 +27.1703 +26.8461+27.3194 +28.5486 +27.8975 +27.4356 +28.12 +28.4397 +29.0186 +26.9328 )/ 16
| 27.74371875
| 
| HEAD is now at 3701f64943f5... tune #3
| (28.2431 +27.7831 +28.39 +28.2586 +27.7408 +27.9258 +26.6236 +26.7817+29.1178 +26.9564 +29.0525 +27.4258 +27.4931 +27.8928 +26.9308 +28.4833)/ 16
| 27.8187
Quoted 6 lines
Quoted 3 lines

This 28.… is the number of GP/s. Based on the results in looks like noise to me. Also I have no idea why you have more than twice as many GP/s as I do.

My guess is that because you have more CPUs, the for_each_online_cpu() loop takes longer on your system.

OK, that is rather oversimplified, to say the least. A better way to put this is that the probability of some CPU holding things up is larger the more CPUs you have. RCU doe take explicit steps to slow down grace periods, but that doesn't start kicking in until 256 CPUs.

Thanx, Paul

Quoted 32 lines
Quoted 24 lines

Different box doing:

	tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 112 --duration 60 --configs "42*TREE01" --memory 4G

and got this:

| HEAD is now at 085b7755808aa... Linux 5.1-rc6
| (15.2878 + 15.8664 + 15.6369 + 15.6714 + 15.3667 + 16.4706 + 15.7844 +
| 16.2119 + 15.6108 + 15.84 + 16.0003 + 16.2886 + 15.8728 + 15.5347 +
| 15.6753 + 15.6628 + 15.8628 + 15.8158 + 15.8419 + 16.0053 + 15.7878 +
| 16.465 + 16.2267 + 16.6881 + 16.3186 + 16.1778 + 15.7069 + 16.0178 +
| 15.7156 + 16.0083 + 15.7181 + 15.8961 + 15.5253 + 16.1569 + 15.7692 +
| 16.2622 + 16.2931 + 15.9531 + 15.6697 + 15.4539 + 15.6478 + 15.8047) /
| 42
|	~15.89452142857142857143
|
| HEAD is now at 3701f64943f5a... tune #3
| ; (15.8461 + 15.8653 + 16.0392 + 15.8906 + 15.7606 + 15.6169 + 16.1425 +
| 15.9089 + 16.2169 + 16.1694 + 16.2122 + 15.6417 + 15.8022 + 16.1178 +
| 15.1689 + 16.1303 + 16.0181 + 16.3797 + 16.0614 + 16.2839 + 15.4583 +
| 15.9178 + 16.0589 + 16.3428 + 15.5486 + 16.0839 + 15.9931 + 15.8417 +
| 16.0981 + 15.8075 + 15.9925 + 15.7311 + 15.9172 + 16.1164 + 15.6303 +
| 15.9383 + 16.0714 + 16.2786 + 15.8394 + 15.9597 + 16.0175 + 15.3908) /
| 42
|	~15.93586904761904761905
Quoted 4 lines

Noise.

So from RCUtorture point of view there is no improvement right? In that case I would suggest to drop the problematic parts…

Thank you for testing this! I will adjust accordingly.

Thanx, Paul

#9 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 41 lines · Paul E. McKenney
Quoted 40 lines · Paul E. McKenney
Quoted 38 lines · Sebastian Andrzej Siewior
Quoted 7 lines · Paul E. McKenney

One place to look is in the summary output:

TREE01 ------- 17540 GPs (58.4667/s) [rcu: g130629 f0x0 ]

The "58.4667/s" is the number of grace periods per second. I would be surprised if CONFIG_PARAVIRT_SPINLOCKS made a noticeable difference in grace-period rate (given the natural variation), but you never know.

I did four runs of the different parts of the patch:
- 5.1-rc6
- #1 + kvm64 CPU + some config options
- #2 + tsc-deadline=on and so on (the whole line)
- #3 + CONFIG_PARAVIRT_SPINLOCKS (now everything)

the test command was

	tools/testing/selftests/rcutorture/bin/kvm.sh --cpus 112 --duration 60 --configs "16*TREE08" --memory 4G

and the results:

| HEAD is now at 085b7755808a... Linux 5.1-rc6
| (28.5942 +27.4658 +28.0203 +27.2061 +28.0731 +26.9078 +27.8494 + 27.3392 +26.4339 +28.025 +27.4797 +27.6775 +28.0653 +28.0742 +27.9581 +28.6508)/ 16
| 27.738775
| 
| HEAD is now at 36a12aa9761a... tune #1
| (28.5761 +26.6514 +26.6989 +27.4375 +27.3442 +28.3228 +26.6353 +27.5461+28.5531 +27.7006 +27.8078 +27.9753 +27.4269 +28.0464 +27.6314 +27.8356) / 16
| 27.6368375
| 
| HEAD is now at af5cd7196436... tune #2
| (28.4867 +26.3675 +27.6364 +28.3344 +27.4153 +27.9306 +27.1703 +26.8461+27.3194 +28.5486 +27.8975 +27.4356 +28.12 +28.4397 +29.0186 +26.9328 )/ 16
| 27.74371875
| 
| HEAD is now at 3701f64943f5... tune #3
| (28.2431 +27.7831 +28.39 +28.2586 +27.7408 +27.9258 +26.6236 +26.7817+29.1178 +26.9564 +29.0525 +27.4258 +27.4931 +27.8928 +26.9308 +28.4833)/ 16
| 27.8187

This 28.… is the number of GP/s. Based on the results in looks like noise to me. Also I have no idea why you have more than twice as many GP/s as I do.

Quoted 7 lines
Quoted 2 lines

My guess is that because you have more CPUs, the for_each_online_cpu() loop takes longer on your system.

OK, that is rather oversimplified, to say the least. A better way to put this is that the probability of some CPU holding things up is larger the more CPUs you have. RCU doe take explicit steps to slow down grace periods, but that doesn't start kicking in until 256 CPUs.

And I ended up with the following variant of your patch. If I don't hear otherwise from you, I will assume that you are OK with it. So if something bothers you about it, please don't suffer in silence!

Thanx, Paul

------------------------------------------------------------------------

commit 4a04229cf73ac9bc1ae15357beb32a0b37be1480

Author: Sebastian Andrzej Siewior <bigeasy at linutronix.de>
Date:   Wed Apr 24 09:34:46 2019 +0200

rcutorture: Tweak kvm options

In one of my rcutorture tests the TSC clocksource got marked unstable due to a large difference in the TSC value. I'm not sure if the guest run for a long time with disabled interrupts or if the host was very busy and didn't schedule the guest for some time.

I took a look on the qemu/KVM options and decided to update the options:

- Use kvm{32|64} as CPU. We could probably use `host' (like ARM does) for maximum available features but since we don't run any userland I'm not sure if it makes any difference.

- Drop the "noapic" option. There is no history why the APIC was disabled, I see no reason for it. Once old qemu versions fade away, we can add "x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on".

- Additional config options. It ensures that the kernel knowns that it runs as a kvm guest and can use virt devices like the kvm-clock as clocksource. The kvm-clock was the main motivation here.

- I didn't add a random HW device. It would make the random device ready earlier (not it doesn't complete the initialisation at all) but I doubt that there is any need for this.

Signed-off-by: Sebastian Andrzej Siewior <bigeasy at linutronix.de> [ paulmck: The world is not quite ready for CONFIG_PARAVIRT_SPINLOCKS=y
and x2apic, so they are omitted for the time being. ]
Signed-off-by: Paul E. McKenney <paulmck at linux.ibm.com>

tools/testing/selftests/rcutorture/bin/functions.sh +12 -1
index 6bcb8b5b2ff2..c3a49fb4d6f6 100644
@@ -172,7 +172,7 @@ identify_qemu_append () {
 	local console=ttyS0
 	case "$1" in
 	qemu-system-x86_64|qemu-system-i386)
-		echo noapic selinux=0 initcall_debug debug
+		echo selinux=0 initcall_debug debug
 		;;
 	qemu-system-aarch64)
 		console=ttyAMA0
@@ -191,8 +191,19 @@ identify_qemu_append () {
 # Output arguments for qemu arguments based on the TORTURE_QEMU_MAC
 # and TORTURE_QEMU_INTERACTIVE environment variables.
 identify_qemu_args () {
+	local KVM_CPU=""
+	case "$1" in
+	qemu-system-x86_64)
+		KVM_CPU=kvm64
+		;;
+	qemu-system-i386)
+		KVM_CPU=kvm32
+		;;
+	esac
 	case "$1" in
 	qemu-system-x86_64|qemu-system-i386)
+		echo -machine q35,accel=kvm
+		echo -cpu ${KVM_CPU}
 		;;
 	qemu-system-aarch64)
 		echo -machine virt,gic-version=host -cpu host
tools/testing/selftests/rcutorture/configs/rcu/CFcommon +3 -0
index d2d2a86139db..e19a444a0684 100644
@@ -1,2 +1,5 @@
 CONFIG_RCU_TORTURE_TEST=y
 CONFIG_PRINTK_TIME=y
+CONFIG_HYPERVISOR_GUEST=y
+CONFIG_PARAVIRT=y
+CONFIG_KVM_GUEST=y
#10 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 3 lines · Paul E. McKenney

And I ended up with the following variant of your patch. If I don't hear otherwise from you, I will assume that you are OK with it. So if something bothers you about it, please don't suffer in silence!

perfect, thank you!

Quoted 1 line

Thanx, Paul

Sebastian

#11 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

In reply to bigeasy at linutronix.de (Sebastian Andrzej Siewior)
Quoted 9 lines · Sebastian Andrzej Siewior

In one of my rcutorture tests the TSC clocksource got marked unstable due to a large difference in the TSC value. I'm not sure if the guest run for a long time with disabled interrupts or if the host was very busy and didn't schedule the guest for some time.
I took a look on the qemu/KVM options and decided to update the options:
- Use kvm{32|64} as CPU. We could probably use `host' (like ARM does) for maximum available features but since we don't run any userland I'm not sure if it makes any difference.

Quoted 3 lines

- Drop the "noapic" option, enable TSC deadline timer. There is no history why the APIC was disabled, I see no reason for it. The deadline timer is probably "nicer".

I was wondering why the tsc deadline timer can't just be the default in the kernel if it is "nicer" / "better" , and why does it need to be an option.

Quoted 8 lines

- Additional config options. It ensures that the kernel knowns that it runs as a kvm guest and can use virt devices like the kvm-clock as clocksource. The kvm-clock was the main motivation here.

- I didn't add a random HW device. It would make the random device ready earlier (not it doesn't complete the initialisation at all) but I doubt that there is any need for this.

Didn't follow this point about "random HW device". It looks like there is no part of the patch that matches this comment. There was no "random HW device" needed for this change of the clocksource, so could you clarify what this means?

Otherwise lgtm, thanks!

Reviewed-by Joel Fernandes (Google) <joel at joelfernandes.org>

- Joel

Quoted 54 lines
Signed-off-by Sebastian Andrzej Siewior <bigeasy at linutronix.de>
---
 tools/testing/selftests/rcutorture/bin/functions.sh | 13 ++++++++++++-
 .../selftests/rcutorture/configs/rcu/CFcommon       |  4 ++++
 2 files changed, 16 insertions(+), 1 deletion(-)
tools/testing/selftests/rcutorture/bin/functions.sh +12 -1
index 6bcb8b5b2ff22..be3c5c73d7e79 100644
@@ -172,7 +172,7 @@ identify_qemu_append () {
 	local console=ttyS0
 	case "$1" in
 	qemu-system-x86_64|qemu-system-i386)
-		echo noapic selinux=0 initcall_debug debug
+		echo selinux=0 initcall_debug debug
 		;;
 	qemu-system-aarch64)
 		console=ttyAMA0
@@ -191,8 +191,19 @@ identify_qemu_append () {
 # Output arguments for qemu arguments based on the TORTURE_QEMU_MAC
 # and TORTURE_QEMU_INTERACTIVE environment variables.
 identify_qemu_args () {
+	local KVM_CPU=""
+	case "$1" in
+	qemu-system-x86_64)
+		KVM_CPU=kvm64
+		;;
+	qemu-system-i386)
+		KVM_CPU=kvm32
+		;;
+	esac
 	case "$1" in
 	qemu-system-x86_64|qemu-system-i386)
+		echo -machine q35,accel=kvm
+		echo -cpu ${KVM_CPU},x2apic=on,tsc-deadline=on,hypervisor=on,tsc_adjust=on
 		;;
 	qemu-system-aarch64)
 		echo -machine virt,gic-version=host -cpu host
tools/testing/selftests/rcutorture/configs/rcu/CFcommon +4 -0
index d2d2a86139db1..322d5d40443cd 100644
@@ -1,2 +1,6 @@
 CONFIG_RCU_TORTURE_TEST=y
 CONFIG_PRINTK_TIME=y
+CONFIG_HYPERVISOR_GUEST=y
+CONFIG_PARAVIRT=y
+CONFIG_PARAVIRT_SPINLOCKS=y
+CONFIG_KVM_GUEST=y
Signature · 1 line
2.20.1
#12 lore raw

PATCH 1/1 rcutorture: Tweak kvm options

Quoted 10 lines · Joel Fernandes
Quoted 9 lines · Sebastian Andrzej Siewior

In one of my rcutorture tests the TSC clocksource got marked unstable due to a large difference in the TSC value. I'm not sure if the guest run for a long time with disabled interrupts or if the host was very busy and didn't schedule the guest for some time.
I took a look on the qemu/KVM options and decided to update the options:
- Use kvm{32|64} as CPU. We could probably use `host' (like ARM does) for maximum available features but since we don't run any userland I'm not sure if it makes any difference.

Quoted 6 lines
Quoted 3 lines

- Drop the "noapic" option, enable TSC deadline timer. There is no history why the APIC was disabled, I see no reason for it. The deadline timer is probably "nicer".

I was wondering why the tsc deadline timer can't just be the default in the kernel if it is "nicer" / "better" , and why does it need to be an option.

The tsc-deadline=on part tells qemu to expose it. Otherwise the kernel can't use HW that isn't there.
I added q35 as the machine which should pass enough sane default options. If this tsc-deadline timer is a problem we could probably drop it. The local-apic should work.

Quoted 7 lines
Quoted 3 lines

- I didn't add a random HW device. It would make the random device ready earlier (not it doesn't complete the initialisation at all) but I doubt that there is any need for this.

Didn't follow this point about "random HW device". It looks like there is no part of the patch that matches this comment. There was no "random HW device" needed for this change of the clocksource, so could you clarify what this means?

I wanted to "upgrade" the kvm options and as part of it also add:
-object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0

With that change you would see
|random: crng init done

during boot. Now you should end up with "only"
|random: fast init done

I mentioned it because I didn't see a reason why to do so. If someone has an idea why it would make sense for rcutorture to use it, I can add it.

Quoted 5 lines

Otherwise lgtm, thanks!

Reviewed-by Joel Fernandes (Google) <joel at joelfernandes.org>

- Joel

Sebastian