Skip to contentSkip to conversation

numa01.sh: Handle computation error

PATCH New Petr Vorel 5 participants 7 messages Jul 1, 2020 – Jul 10, 2020

lore
Share & tools
lore

Petr Vorel

author
#0 lore raw

PATCH 1/1 numa01.sh: Handle computation error

when numastat -p did not give a value, the resulting bc calculation was empty string instead of 0, thus shell -eq comparison lacked the first operator:

Mem_curr=$(echo "$(extract_numastat_p $pid $node) * $MB" |bc) if [ $(echo "$Mem_curr < $MB" |bc ) -eq 1 ]; then

(standard_in) 1: syntax error
(standard_in) 1: syntax error /root/ltp-install/testcases/bin/numa01.sh: line 93: [: -eq: unary operator expected

Also fix style (use local and lowercase local function variables).

Fixes 702

---
Hi,

Simple fix for https://github.com/linux-test-project/ltp/issues/702,
I guess there should be more checks. Not tested.
BTW I thought numa01.sh was intended to be replaced with C (@Cyril: am I correct?), thus I didn't bother to split syntax fix into 2 commits.

Kind regards,
Petr

 testcases/kernel/numa/numa01.sh | 95 ++++++++++++++++++---------------
 1 file changed, 51 insertions(+), 44 deletions(-)
testcases/kernel/numa/numa01.sh +51 -44
index 1d626327d..a217db033 100755
@@ -1,7 +1,7 @@
 #!/bin/sh
 # SPDX-License-Identifier: GPL-2.0-or-later
 # Copyright (c) International Business Machines Corp., 2007
-# Copyright (c) Linux Test Project, 2016
+# Copyright (c) Linux Test Project, 2016-2020
 # Author: Sivakumar Chinnaiah <[email protected]>
 #
 # Test Basic functionality of numactl command.
@@ -12,9 +12,9 @@
 # Test #5: Verifies localalloc
 # Test #6: Verifies memhog
 # Test #7: Verifies numa_node_size api
-# Test #8:Verifies Migratepages
-# Test #9:Verifies hugepage alloacted on specified node
-# Test #10:Verifies THP memory allocated on preferred node
+# Test #8: Verifies Migratepages
+# Test #9: Verifies hugepage alloacted on specified node
+# Test #10: Verifies THP memory allocated on preferred node
 
 TST_CNT=10
 TST_SETUP=setup
@@ -25,15 +25,24 @@ TST_NEEDS_CMDS="awk bc numactl numastat"
 
 . tst_test.sh
 
-# Extracts the value of given numa node from the `numastat -p` output.
-# $1 - Pid number.
-# $2 - Node number.
-extract_numastat_p()
+# Convert the value of given numa node from the `numastat -p` output,
+# multiply by size.
+# $1 - Pid number
+# $2 - Node number
+# $3 - Size for multiplication (e.g. 1024, $MB)
+get_mem_cur()
 {
 	local pid=$1
 	local node=$(($2 + 2))
+	local size=$3
+	local numstat=$(numastat -p $pid |awk '/^Total/ {print $'$node'}')
 
-	echo $(numastat -p $pid |awk '/^Total/ {print $'$node'}')
+	if [ -z "$numstat" ]; then
+		echo 0
+		return
+	fi
+
+	echo $(echo "$numstat * $size" | bc)
 }
 
 check_for_support_numa()
@@ -72,7 +81,7 @@ setup()
 # Verification of memory allocated on a node
 test1()
 {
-	Mem_curr=0
+	local mem_curr
 
 	for node in $nodes_list; do
 		numactl --cpunodebind=$node --membind=$node support_numa alloc_1MB &
@@ -80,8 +89,8 @@ test1()
 
 		TST_RETRY_FUNC "check_for_support_numa $pid" 0
 
-		Mem_curr=$(echo "$(extract_numastat_p $pid $node) * $MB" |bc)
-		if [ $(echo "$Mem_curr < $MB" | bc) -eq 1 ]; then
+		mem_curr=$(get_mem_cur $pid $node $MB)
+		if [ $(echo "$mem_curr < $MB" | bc) -eq 1 ]; then
 			tst_res TFAIL \
 				"NUMA memory allocated in node$node is less than expected"
 			kill -CONT $pid >/dev/null 2>&1
@@ -97,16 +106,16 @@ test1()
 # Verification of memory allocated on preferred node
 test2()
 {
-	Mem_curr=0
+	local mem_curr
+	local cnt=1
 
-	COUNTER=1
 	for node in $nodes_list; do
 
-		if [ $COUNTER -eq $total_nodes ]; then   #wrap up for last node
+		if [ $cnt -eq $total_nodes ]; then   #wrap up for last node
 			Preferred_node=$(echo $nodes_list | cut -d ' ' -f 1)
 		else
 			# always next node is preferred node
-			Preferred_node=$(echo $nodes_list | cut -d ' ' -f $((COUNTER+1)))
+			Preferred_node=$(echo $nodes_list | cut -d ' ' -f $((cnt+1)))
 		fi
 
 		numactl --cpunodebind=$node --preferred=$Preferred_node support_numa alloc_1MB &
@@ -114,15 +123,15 @@ test2()
 
 		TST_RETRY_FUNC "check_for_support_numa $pid" 0
 
-		Mem_curr=$(echo "$(extract_numastat_p $pid $Preferred_node) * $MB" |bc)
-		if [ $(echo "$Mem_curr < $MB" |bc ) -eq 1 ]; then
+		mem_curr=$(get_mem_cur $pid $Preferred_node $MB)
+		if [ $(echo "$mem_curr < $MB" |bc ) -eq 1 ]; then
 			tst_res TFAIL \
 				"NUMA memory allocated in node$Preferred_node is less than expected"
 			kill -CONT $pid >/dev/null 2>&1
 			return
 		fi
 
-		COUNTER=$((COUNTER+1))
+		cnt=$((cnt+1))
 		kill -CONT $pid >/dev/null 2>&1
 	done
 
@@ -132,7 +141,7 @@ test2()
 # Verification of memory interleaved on all nodes
 test3()
 {
-	Mem_curr=0
+	local mem_curr
 	# Memory will be allocated using round robin on nodes.
 	Exp_incr=$(echo "$MB / $total_nodes" |bc)
 
@@ -142,9 +151,9 @@ test3()
 	TST_RETRY_FUNC "check_for_support_numa $pid" 0
 
 	for node in $nodes_list; do
-		Mem_curr=$(echo "$(extract_numastat_p $pid $node) * $MB" |bc)
+		mem_curr=$(get_mem_cur $pid $node $MB)
 
-		if [ $(echo "$Mem_curr < $Exp_incr" |bc ) -eq 1 ]; then
+		if [ $(echo "$mem_curr < $Exp_incr" |bc ) -eq 1 ]; then
 			tst_res TFAIL \
 				"NUMA interleave memory allocated in node$node is less than expected"
 			kill -CONT $pid >/dev/null 2>&1
@@ -191,7 +200,7 @@ test4()
 # Verification of local node allocation
 test5()
 {
-	Mem_curr=0
+	local mem_curr
 
 	for node in $nodes_list; do
 		numactl --cpunodebind=$node --localalloc support_numa alloc_1MB &
@@ -199,8 +208,8 @@ test5()
 
 		TST_RETRY_FUNC "check_for_support_numa $pid" 0
 
-		Mem_curr=$(echo "$(extract_numastat_p $pid $node) * $MB" |bc)
-		if [ $(echo "$Mem_curr < $MB" |bc ) -eq 1 ]; then
+		mem_curr=$(get_mem_cur $pid $node $MB)
+		if [ $(echo "$mem_curr < $MB" |bc ) -eq 1 ]; then
 			tst_res TFAIL \
 				"NUMA localnode memory allocated in node$node is less than expected"
 			kill -CONT $pid >/dev/null 2>&1
@@ -221,7 +230,7 @@ check_ltp_numa_test8_log()
 # Verification of memhog with interleave policy
 test6()
 {
-	Mem_curr=0
+	local mem_curr
 	# Memory will be allocated using round robin on nodes.
 	Exp_incr=$(echo "$MB / $total_nodes" |bc)
 
@@ -231,9 +240,9 @@ test6()
 	TST_RETRY_FUNC "check_ltp_numa_test8_log" 0
 
 	for node in $nodes_list; do
-		Mem_curr=$(echo "$(extract_numastat_p $pid $node) * $MB" |bc)
+		mem_curr=$(get_mem_cur $pid $node $MB)
 
-		if [ $(echo "$Mem_curr < $Exp_incr" |bc ) -eq 1 ]; then
+		if [ $(echo "$mem_curr < $Exp_incr" |bc ) -eq 1 ]; then
 			tst_res TFAIL \
 				"NUMA interleave memhog in node$node is less than expected"
 			kill -KILL $pid >/dev/null 2>&1
@@ -283,15 +292,14 @@ test7()
 # Verification of migratepages
 test8()
 {
-	Mem_curr=0
-	COUNTER=1
+	local mem_curr
+	local cnt=1
 
 	for node in $nodes_list; do
-
-		if [ $COUNTER -eq $total_nodes ]; then
+		if [ $cnt -eq $total_nodes ]; then
 			Preferred_node=$(echo $nodes_list | cut -d ' ' -f 1)
 		else
-			Preferred_node=$(echo $nodes_list | cut -d ' ' -f $((COUNTER+1)))
+			Preferred_node=$(echo $nodes_list | cut -d ' ' -f $((cnt+1)))
 		fi
 
 		numactl --preferred=$node support_numa alloc_1MB &
@@ -301,15 +309,15 @@ test8()
 
 		migratepages $pid $node $Preferred_node
 
-		Mem_curr=$(echo "$(extract_numastat_p $pid $Preferred_node) * $MB" |bc)
-		if [ $(echo "$Mem_curr < $MB" |bc ) -eq 1 ]; then
+		mem_curr=$(get_mem_cur $pid $Preferred_node $MB)
+		if [ $(echo "$mem_curr < $MB" |bc ) -eq 1 ]; then
 			tst_res TFAIL \
 				"NUMA migratepages is not working fine"
 			kill -CONT $pid >/dev/null 2>&1
 			return
 		fi
 
-		COUNTER=$((COUNTER+1))
+		cnt=$((cnt+1))
 		kill -CONT $pid >/dev/null 2>&1
 	done
 
@@ -363,21 +371,20 @@ test9()
 # Verification of THP memory allocated on preferred node
 test10()
 {
-	Mem_curr=0
+	local mem_curr
+	local cnt=1
 
 	if ! grep -q '\[always\]' /sys/kernel/mm/transparent_hugepage/enabled; then
 		tst_res TCONF "THP is not supported/enabled"
 		return
 	fi
 
-	COUNTER=1
 	for node in $nodes_list; do
-
-		if [ $COUNTER -eq $total_nodes ]; then   #wrap up for last node
+		if [ $cnt -eq $total_nodes ]; then   #wrap up for last node
 			Preferred_node=$(echo $nodes_list | cut -d ' ' -f 1)
 		else
 			# always next node is preferred node
-			Preferred_node=$(echo $nodes_list | cut -d ' ' -f $((COUNTER+1)))
+			Preferred_node=$(echo $nodes_list | cut -d ' ' -f $((cnt+1)))
 		fi
 
 		numactl --cpunodebind=$node --preferred=$Preferred_node support_numa alloc_2HPSZ_THP &
@@ -385,15 +392,15 @@ test10()
 
 		TST_RETRY_FUNC "check_for_support_numa $pid" 0
 
-		Mem_curr=$(echo "$(extract_numastat_p $pid $Preferred_node) * 1024" |bc)
-		if [ $(echo "$Mem_curr < $HPAGE_SIZE * 2" |bc ) -eq 1 ]; then
+		mem_curr=$(get_mem_cur $pid $Preferred_node 1024)
+		if [ $(echo "$mem_curr < $HPAGE_SIZE * 2" |bc ) -eq 1 ]; then
 			tst_res TFAIL \
 				"NUMA memory allocated in node$Preferred_node is less than expected"
 			kill -CONT $pid >/dev/null 2>&1
 			return
 		fi
 
-		COUNTER=$((COUNTER+1))
+		cnt=$((cnt+1))
 		kill -CONT $pid >/dev/null 2>&1
 	done
 
Signature · 1 line
2.27.0.rc0
#1 lore raw

PATCH 1/1 numa01.sh: Handle computation error

Quoted 12 lines · Petr Vorel

when numastat -p did not give a value, the resulting bc calculation was empty string instead of 0, thus shell -eq comparison lacked the first operator:

Mem_curr=$(echo "$(extract_numastat_p $pid $node) * $MB" |bc) if [ $(echo "$Mem_curr < $MB" |bc ) -eq 1 ]; then

(standard_in) 1: syntax error
(standard_in) 1: syntax error /root/ltp-install/testcases/bin/numa01.sh: line 93: [: -eq: unary operator expected

Quoted 5 lines

Also fix style (use local and lowercase local function variables).

Fixes 702
Quoted 8 lines

---
Hi,

Simple fix for https://github.com/linux-test-project/ltp/issues/702,
I guess there should be more checks. Not tested.
BTW I thought numa01.sh was intended to be replaced with C (@Cyril: am I correct?), thus I didn't bother to split syntax fix into 2 commits.

I think yes, I remember the syscalls/set_mempolicy* is going to replace numa.sh tests.

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20200702/a3a7c2cb/attachment-0001.htm>
#2 lore raw

PATCH 1/1 numa01.sh: Handle computation error

In reply to Petr Vorel

Hi!

Quoted 4 lines

Simple fix for https://github.com/linux-test-project/ltp/issues/702,
I guess there should be more checks. Not tested.
BTW I thought numa01.sh was intended to be replaced with C (@Cyril: am I correct?), thus I didn't bother to split syntax fix into 2 commits.

In the long term yes, a few tests have been removed from it since they were reimplemented in C already. The rest is on TODO with a low priority. For instance we already have tests for migrate_pages, if we cleanup these we can remove the test8 from the numa01.sh etc...

Signature · 2 lines
Cyril Hrubis
[email protected]
#3 lore raw

PATCH 1/1 numa01.sh: Handle computation error

In reply to Petr Vorel

Hi Petr, Harish,

Though the root cause is from the non-ordered node in a special machine, I still think this patch makes sense to numa01, because the function get_mem_cur() make code more readable.

So I'm going to merge both this one and Harish's patch, after doing that, I will also follow Cyril's comment to remove test8(migrate_pages).

Any objections? or comments?

testcases/kernel/numa/numa01.sh TST_NEEDS_CMDS="awk bc numactl numastat" · line 42

Quoted 13 lines · Petr Vorel

...

testcases/kernel/numa/numa01.sh +6 -1
+get_mem_cur()
 {
        local pid=$1
        local node=$(($2 + 2))
+       local size=$3
+       local numstat=$(numastat -p $pid |awk '/^Total/ {print $'$node'}')
 
-       echo $(numastat -p $pid |awk '/^Total/ {print $'$node'}')
+       if [ -z "$numstat" ]; then
+               echo 0
+               return

Maybe we'd better do TBROK from here if numstat doesn't work well?

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20200703/6552fbff/attachment.htm>
#4 lore raw

PATCH 1/1 numa01.sh: Handle computation error

Quoted 11 lines · Li Wang

Hi?Petr, Harish,

Though the root cause is from the non-ordered node in a special machine, I still think this patch makes sense to numa01, because the function get_mem_cur() make code more readable.

So I'm going to merge both this one and Harish's patch, after doing that, I will also follow Cyril's comment to remove test8(migrate_pages).

Any objections? or comments?

Makes sense to me.

Regards,
Harish

Quoted 21 lines

On Thu, Jul 2, 2020 at 7:31 AM Petr Vorel <[email protected] <mailto:[email protected]>> wrote:

    ...
    +get_mem_cur()
    ?{
    ? ? ? ? local pid=$1
    ? ? ? ? local node=$(($2 + 2))
    +? ? ? ?local size=$3
    +? ? ? ?local numstat=$(numastat -p $pid |awk '/^Total/ {print
    $'$node'}')

    -? ? ? ?echo $(numastat -p $pid |awk '/^Total/ {print $'$node'}')
    +? ? ? ?if [ -z "$numstat" ]; then
    +? ? ? ? ? ? ? ?echo 0
    +? ? ? ? ? ? ? ?return
Quoted 5 lines

Maybe we'd better do TBROK from here if numstat doesn't work well?

Signature · 2 lines
Regards,
Li Wang

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20200703/a4d8ea13/attachment.htm>

#5 lore raw

PATCH 1/1 numa01.sh: Handle computation error

Quoted 8 lines · Harish

On 7/3/20 8:30 AM, Li Wang wrote:

Hi Petr, Harish,

Though the root cause is from the non-ordered node in a special machine, I still think this patch makes sense to numa01, because the function get_mem_cur() make code more readable.

Quoted 7 lines

So I'm going to merge both this one and Harish's patch, after doing that, I will also follow Cyril's comment to remove test8(migrate_pages).

Any objections? or comments?

Makes sense to me.

I have pushed both patches, and remove the test8 too.

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20200708/9377025f/attachment-0001.htm>
#6 lore raw

PATCH 1/1 numa01.sh: Handle computation error

Hi Li,

Quoted 1 line

I have pushed both patches, and remove the test8 too.

Thanks for merging all the fixes
(I was on 2 weeks vacation thus didn't reply).

Kind regards,
Petr