Skip to contentSkip to conversation

cpuhotplug05.sh: Rewrite test case

PATCH New Xiao Yang 4 participants 7 messages Dec 2, 2019 – Dec 10, 2019

lore
Share & tools
lore

Xiao Yang

author
#0 lore raw

PATCH 1/1 cpuhotplug05.sh: Rewrite test case

1) Replace sar command with /proc/stat
2) Convert to new API
3) Remove unused/duplicated code

---
 runtest/cpuhotplug                            |   2 +-
 .../cpu_hotplug/functional/cpuhotplug05.sh    | 186 ++++++------------
 2 files changed, 63 insertions(+), 125 deletions(-)
runtest/cpuhotplug +1 -1
index ec7f11ed1..bd97e01b8 100644
@@ -4,6 +4,6 @@
 cpuhotplug02 cpuhotplug02.sh -c 1 -l 1
 cpuhotplug03 cpuhotplug03.sh -c 1 -l 1
 cpuhotplug04 cpuhotplug04.sh -l 1
-cpuhotplug05 cpuhotplug05.sh -c 1 -l 1 -d /tmp
+cpuhotplug05 cpuhotplug05.sh -c 1
 cpuhotplug06 cpuhotplug06.sh -c 1 -l 1
 cpuhotplug07 cpuhotplug07.sh -c 1 -l 1 -d /usr/src/linux
testcases/kernel/hotplug/cpu_hotplug/functional/cpuhotplug05.sh +62 -124

219 lines not on this page · raw message

Signature · 1 line
2.21.0
#1 lore raw

PATCH 1/1 cpuhotplug05.sh: Rewrite test case

testcases/kernel/hotplug/cpu_hotplug/functional/cpuhotplug05.sh line 59

Quoted 2 lines · Xiao Yang
testcases/kernel/hotplug/cpu_hotplug/functional/cpuhotplug05.sh +2 -0
+do_test()
+{

First of all, your patch changes the test scenario. The original scenario is this:
(setup) Deactivate selected CPU
1. Check that load statistics are zero or empty for selected CPU
2. Activate selected CPU
3. Check that load statistics are valid for selected CPU
4. Deactivate selected CPU
(cleanup) Activate selected CPU

I have a few kernel builds where step 3 correctly detects regression.

Your changed scenario looks like this:
1. Activate selected CPU
2. Check that load statistics are non-zero for selected CPU
3. Deactivate selected CPU
4. Check that load statistics are empty for selected CPU (cleanup) Return selected CPU back to original state

Your new scenario has two problems:

- You have to run at least two test loops to verify that reactivating a CPU doesn't break /proc/stat entries.

- You only check that load statistics are non-zero. This works with sar because the program already normalizes load statistics for you. But it won't detect the regression mentioned above because the bug sets one of the entries to something absurdly high. This will confuse sar enough to print all load values as 0 but it'll pass your checks.

In addition to checking that at least some values are non-zero, you need to check a few more things:
- Call getconf CLK_TCK to find /proc/stat timer resolution
- Calculate system uptime in seconds
- Check that sum of all /proc/stat values for selected CPU is less than or equal to (timer resolution * (uptime+1))
- Also consider doing the above check for all CPUs to increase test coverage.

testcases/kernel/hotplug/cpu_hotplug/functional/cpuhotplug05.sh line 81

Quoted 13 lines
testcases/kernel/hotplug/cpu_hotplug/functional/cpuhotplug05.sh +7 -3
+	for field in $(seq 2 11); do
+		field_value=$(grep "^cpu${cpu_num}" /proc/stat | awk "{print \$$field}")
+		[ "$field_value" != "0" ] && passed=1
 	done
 
-	if [ $check_passed -eq 0 ]; then
-		tst_resm TFAIL "No change in the CPU statistics"
-		tst_exit
+	if [ $passed -eq 0 ]; then
+		tst_res TFAIL \
+			"all field of online cpu{cpu_num} shows zero in /proc/stat"
+		return 1
 	fi

It'd be better to write this check (including the upper bound check explained above) entirely in Awk.

if ! awk "[test script]" /proc/stat; then
tst_res TFAIL ...
return 1
fi

Signature · 7 lines
Martin Doucha   [email protected]
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic
author
#2 lore raw

PATCH 1/1 cpuhotplug05.sh: Rewrite test case

Hi Martin,

Thanks for your comment.
Also sorry for the late reply.

Quoted 21 lines · Martin Doucha

testcases/kernel/hotplug/cpu_hotplug/functional/cpuhotplug05.sh line 59

Quoted 2 lines · Xiao Yang
testcases/kernel/hotplug/cpu_hotplug/functional/cpuhotplug05.sh +2 -0
+do_test()
+{

First of all, your patch changes the test scenario. The original scenario is this:
(setup) Deactivate selected CPU
1. Check that load statistics are zero or empty for selected CPU
2. Activate selected CPU
3. Check that load statistics are valid for selected CPU
4. Deactivate selected CPU
(cleanup) Activate selected CPU

I have a few kernel builds where step 3 correctly detects regression.

Your changed scenario looks like this:
1. Activate selected CPU
2. Check that load statistics are non-zero for selected CPU
3. Deactivate selected CPU
4. Check that load statistics are empty for selected CPU (cleanup) Return selected CPU back to original state

Quoted 4 lines

Your new scenario has two problems:

- You have to run at least two test loops to verify that reactivating a CPU doesn't break /proc/stat entries.

I think running cpuhotplug05.sh with -i 2 can verfiy this point.

Perhaps we can add -i option to runtest/cpuhotplug, or do you prefer to keep the original scenario?

Quoted 6 lines

- You only check that load statistics are non-zero. This works with sar because the program already normalizes load statistics for you. But it won't detect the regression mentioned above because the bug sets one of the entries to something absurdly high. This will confuse sar enough to print all load values as 0 but it'll pass your checks.

Good point.

I will looking into the code of sar.

Quoted 7 lines

In addition to checking that at least some values are non-zero, you need to check a few more things:
- Call getconf CLK_TCK to find /proc/stat timer resolution
- Calculate system uptime in seconds
- Check that sum of all /proc/stat values for selected CPU is less than or equal to (timer resolution * (uptime+1))

I will try to add them as you suggested.

Quoted 2 lines

- Also consider doing the above check for all CPUs to increase test coverage.

Is it necessary to add above check for all CPUs?

This test is designed to test specified CPU so you can test each CPU by using -c option.

Thanks,

Xiao Yang

Quoted 22 lines
Quoted 4 lines
testcases/kernel/hotplug/cpu_hotplug/functional/cpuhotplug05.sh +3 -0
+	for field in $(seq 2 11); do
+		field_value=$(grep "^cpu${cpu_num}" /proc/stat | awk "{print \$$field}")
+		[ "$field_value" != "0" ] && passed=1

testcases/kernel/hotplug/cpu_hotplug/functional/cpuhotplug05.sh line 81

Quoted 10 lines
testcases/kernel/hotplug/cpu_hotplug/functional/cpuhotplug05.sh +4 -3
  	done
 
-	if [ $check_passed -eq 0 ]; then
-		tst_resm TFAIL "No change in the CPU statistics"
-		tst_exit
+	if [ $passed -eq 0 ]; then
+		tst_res TFAIL \
+			"all field of online cpu{cpu_num} shows zero in /proc/stat"
+		return 1
  	fi

It'd be better to write this check (including the upper bound check explained above) entirely in Awk.

if ! awk "[test script]" /proc/stat; then
tst_res TFAIL ...
return 1
fi

#3 lore raw

PATCH 1/1 cpuhotplug05.sh: Rewrite test case

Quoted 9 lines · Xiao Yang
Quoted 4 lines

Your new scenario has two problems:

- You have to run at least two test loops to verify that reactivating a CPU doesn't break /proc/stat entries.

I think running cpuhotplug05.sh with -i 2 can verfiy this point.

Perhaps we can add -i option to runtest/cpuhotplug, or do you prefer to keep the original scenario?

I'd prefer keeping the original scenario (with your version of cleanup). Running extra iterations of the same test should not be required to get the full intended test coverage.

Quoted 7 lines
Quoted 2 lines

- Also consider doing the above check for all CPUs to increase test coverage.

Is it necessary to add above check for all CPUs?

This test is designed to test specified CPU so you can test each CPU by using -c option.

It's not necessary but it'd be a nice addition. Note that running the test script multiple times with different -c arguments does not fully cover this extra check. Turning a CPU core off and on again might reset its /proc/stat line into a sane state while breaking all the others.

Signature · 7 lines
Martin Doucha   [email protected]
QA Engineer for Software Maintenance
SUSE LINUX, s.r.o.
CORSO IIa
Krizikova 148/34
186 00 Prague 8
Czech Republic
#4 lore raw

PATCH 1/1 cpuhotplug05.sh: Rewrite test case

Hi,

Quoted 6 lines · Xiao Yang
Quoted 4 lines
Quoted 4 lines

Your new scenario has two problems:

- You have to run at least two test loops to verify that reactivating a CPU doesn't break /proc/stat entries.

Quoted 8 lines
Quoted 4 lines

I think running cpuhotplug05.sh with -i 2 can verfiy this point.

Perhaps we can add -i option to runtest/cpuhotplug, or do you prefer to keep the original scenario?

I'd prefer keeping the original scenario (with your version of cleanup). Running extra iterations of the same test should not be required to get the full intended test coverage.

Agree with Martin. Yang, could you, please, send new version, where you keep it?

Kind regards,
Petr

#5 lore raw

PATCH 1/1 cpuhotplug05.sh: Rewrite test case

Quoted 1 line · Petr Vorel

Agree with Martin. Yang, could you, please, send new version, where you keep it?

Hi Petr, Martin.

Sorry, I am busy with other things recently.
Don't worry, I will send new version this week or next week. :-)

Thanks,
Xiao Yang

Quoted 3 lines

Kind regards,
Petr

#6 lore raw

PATCH 1/1 cpuhotplug05.sh: Rewrite test case

Hi,

Quoted 2 lines

Sorry, I am busy with other things recently.
Don't worry, I will send new version this week or next week. :-)

Yang, sure, no rush.

Martin, thanks a lot for very good review!

Kind regards,
Petr