Add a test that triggers the blk_mq_realloc_hw_ctxs() error path
v3 cover Add a test that triggers the blk_mq_realloc_hw_ctxs() error path
Hi Omar,
Please consider this patch series for the upstream blktests repository.
Thanks,
Bart.
Changes compared to v2:
- Addressed Omar's review comments.
Changes between v1 and v2:
- Added three patches that refactor null_blk loading, unloading and configuration (Chaitanya).
Bart Van Assche (4):
Make _exit_null_blk remove all null_blk device instances Use _{init,exit}_null_blk instead of open-coding these functions Introduce the function _configure_null_blk()
Add a test that triggers the blk_mq_realloc_hw_ctxs() error path
common/multipath-over-rdma | 29 ++++++++------------------ common/null_blk | 21 ++++++++++++++++++- tests/block/022 | 3 --- tests/block/029 | 17 ++------------- tests/block/030 | 42 ++++++++++++++++++++++++++++++++++++++ tests/block/030.out | 1 + tests/nvmeof-mp/rc | 2 +- 7 files changed, 74 insertions(+), 41 deletions(-) create mode 100755 tests/block/030 create mode 100644 tests/block/030.out
PATCH v3 1/4 Make _exit_null_blk remove all null_blk device instances
Instead of making every test remove null_blk device instances before calling _exit_null_blk(), move the null_blk device instance removal code into _exit_null_blk().
--- common/null_blk | 7 ++++++- tests/block/022 | 3 --- tests/block/029 | 1 - 3 files changed, 6 insertions(+), 5 deletions(-)
common/null_blk +6 -1
@@ -8,11 +8,15 @@ _have_null_blk() { _have_modules null_blk } -_init_null_blk() { +_remove_null_blk_devices() { if [[ -d /sys/kernel/config/nullb ]]; then find /sys/kernel/config/nullb -mindepth 1 -maxdepth 1 \ -type d -delete fi +} + +_init_null_blk() { + _remove_null_blk_devices local zoned="" if (( RUN_FOR_ZONED )); then zoned="zoned=1"; fi
@@ -26,6 +30,7 @@ _init_null_blk() { } _exit_null_blk() { + _remove_null_blk_devices udevadm settle modprobe -r null_blk }
tests/block/022 +0 -3
@@ -50,9 +50,6 @@ test() { wait $pid1 } 2>/dev/null - rmdir /sys/kernel/config/nullb/1 - rmdir /sys/kernel/config/nullb/0 - _exit_null_blk echo "Test complete" }
tests/block/029 +0 -1
@@ -58,7 +58,6 @@ test() { else echo "Skipping test because $sq cannot be modified" >>"$FULL" fi - rmdir /sys/kernel/config/nullb/nullb0 _exit_null_blk echo Passed }
Re: [PATCH blktests v3 1/4] Make _exit_null_blk remove all null_blk device instances
Quoted 6 lines · Bart Van Assche
Instead of making every test remove
null_blkdevice instances before calling_exit_null_blk(), move thenull_blkdevice instance removal code into_exit_null_blk().Reviewed-by Chaitanya Kulkarni <[email protected]>Signed-off-by Bart Van Assche <[email protected]>
PATCH v3 2/4 Use _{init,exit}_null_blk instead of open-coding these functions
In reply to Bart Van AsscheThis patch reduces code duplication.
--- common/multipath-over-rdma | 12 +++--------- tests/nvmeof-mp/rc | 2 +- 2 files changed, 4 insertions(+), 10 deletions(-)
common/multipath-over-rdma +3 -9
@@ -5,6 +5,7 @@ # Functions and global variables used by both the srp and nvmeof-mp tests. . common/shellcheck +. common/null_blk debug= filesystem_type=ext4
@@ -634,13 +635,6 @@ configure_null_blk() { ls -l /dev/nullb* &>>"$FULL" } -unload_null_blk() { - local d - - for d in /sys/kernel/config/nullb/*; do [ -d "$d" ] && rmdir "$d"; done - unload_module null_blk -} - setup_rdma() { start_soft_rdma (
@@ -662,7 +656,7 @@ teardown_uncond() { rm -f /etc/multipath.conf stop_target stop_soft_rdma - unload_null_blk + _exit_null_blk } teardown() {
@@ -698,7 +692,7 @@ setup_test() { [ -e "/sys/module/$m" ] || modprobe "$m" || return $? done - modprobe null_blk nr_devices=0 || return $? + _init_null_blk nr_devices=0 || return $? configure_null_blk || return $?
tests/nvmeof-mp/rc +1 -1
@@ -278,7 +278,7 @@ stop_nvme_target() { ) unload_module nvmet_rdma && unload_module nvmet && - unload_null_blk + _exit_null_blk } start_target() {
Re: [PATCH blktests v3 2/4] Use _{init,exit}_null_blk instead of open-coding these functions
Quoted 4 lines · Bart Van Assche
This patch reduces code duplication.
Reviewed-by Chaitanya Kulkarni <[email protected]>Signed-off-by Bart Van Assche <[email protected]>
PATCH v3 3/4 Introduce the function _configure_null_blk()
In reply to Bart Van AsscheIntroduce a function for creating a null_blk device instance through configfs.
--- common/multipath-over-rdma | 17 +++++------------ common/null_blk | 14 ++++++++++++++ tests/block/029 | 16 ++-------------- 3 files changed, 21 insertions(+), 26 deletions(-)
common/multipath-over-rdma +5 -12
@@ -620,18 +620,11 @@ run_fio() { configure_null_blk() { local i - ( - cd /sys/kernel/config/nullb || return $? - for i in nullb0 nullb1; do ( - { mkdir -p $i && - cd $i && - echo 0 > completion_nsec && - echo 512 > blocksize && - echo $((ramdisk_size>>20)) > size && - echo 1 > memory_backed && - echo 1 > power; } || exit $? - ) done - ) + for i in nullb0 nullb1; do + _configure_null_blk nullb$i completion_nsec=0 blocksize=512 \ + size=$((ramdisk_size>>20)) memory_backed=1 \ + power=1 || return $? + done ls -l /dev/nullb* &>>"$FULL" }
common/null_blk +14 -0
@@ -29,6 +29,20 @@ _init_null_blk() { return 0 } +# Configure one null_blk instance with name $1 and parameters $2..${$#}. +_configure_null_blk() { + local nullb=/sys/kernel/config/nullb/$1 param val + + shift + mkdir "$nullb" || return $? + while [ -n "$1" ]; do + param="${1%%=*}" + val="${1#*=}" + shift + echo "$val" > "$nullb/$param" || return $? + done +} + _exit_null_blk() { _remove_null_blk_devices udevadm settle
tests/block/029 +2 -14
@@ -14,19 +14,6 @@ requires() { _have_null_blk } -# Configure one null_blk instance. -configure_null_blk() { - local nullb0="/sys/kernel/config/nullb/nullb0" - - mkdir "$nullb0" && - echo 0 > "$nullb0/completion_nsec" && - echo 512 > "$nullb0/blocksize" && - echo 16 > "$nullb0/size" && - echo 1 > "$nullb0/memory_backed" && - echo 1 > "$nullb0/power" && - ls -l /dev/nullb* &>>"$FULL" -} - modify_nr_hw_queues() { local deadline num_cpus
@@ -45,7 +32,8 @@ test() { : "${TIMEOUT:=30}" _init_null_blk nr_devices=0 queue_mode=2 && - configure_null_blk + _configure_null_blk nullb0 completion_nsec=0 blocksize=512 \ + size=16 memory_backed=1 power=1 && if { echo 1 >$sq; } 2>/dev/null; then modify_nr_hw_queues & fio --rw=randwrite --bs=4K --loops=$((10**6)) \
Re: [PATCH blktests v3 3/4] Introduce the function _configure_null_blk()
Hi Bart,
Quoted 24 lines · Bart Van Assche
Introduce a function for creating a
null_blkdevice instance through configfs.Suggested-by Chaitanya Kulkarni <[email protected]>Reviewed-by Chaitanya Kulkarni <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- common/multipath-over-rdma | 17 +++++------------ common/null_blk | 14 ++++++++++++++ tests/block/029 | 16 ++-------------- 3 files changed, 21 insertions(+), 26 deletions(-)common/multipath-over-rdma +0
-4@@ -620,18 +620,11 @@ run_fio() { configure_null_blk() { local i - ( - cd /sys/kernel/config/nullb || return $? - for i in nullb0 nullb1; do ( - { mkdir -p $i &&
common/multipath-over-rdma run_fio() { · line 624
Quoted 10 lines
common/multipath-over-rdma +2
-8- cd $i && - echo 0 > completion_nsec && - echo 512 > blocksize && - echo $((ramdisk_size>>20)) > size && - echo 1 > memory_backed && - echo 1 > power; } || exit $? - ) done - ) + for i in nullb0 nullb1; do + _configure_null_blk nullb$i completion_nsec=0 blocksize=512 \
I think this should be:
_configure_null_blk $i completion_nsec=0 blocksize=512 \
Rest looks good, feel free do add my Reviewed-by when spinning a new version.
Thanks,
Daniel
Re: [PATCH blktests v3 3/4] Introduce the function _configure_null_blk()
Quoted 26 lines · Daniel Wagner
Quoted 12 lines · Bart Van Assche
common/multipath-over-rdma +0
-4@@ -620,18 +620,11 @@ run_fio() { configure_null_blk() { local i - ( - cd /sys/kernel/config/nullb || return $? - for i in nullb0 nullb1; do ( - { mkdir -p $i &&common/multipath-over-rdma run_fio() { · line 624
Quoted 10 lines
common/multipath-over-rdma +2
-8- cd $i && - echo 0 > completion_nsec && - echo 512 > blocksize && - echo $((ramdisk_size>>20)) > size && - echo 1 > memory_backed && - echo 1 > power; } || exit $? - ) done - ) + for i in nullb0 nullb1; do + _configure_null_blk nullb$i completion_nsec=0 blocksize=512 \I think this should be:
Quoted 1 line
_configure_null_blk $i completion_nsec=0 blocksize=512 \
Good catch, will fix.
Thanks,
Bart.
PATCH v3 4/4 Add a test that triggers the blk_mq_realloc_hw_ctxs() error path
In reply to Bart Van AsscheAdd a test that triggers the code touched by commit d0930bb8f46b ("blk-mq:
Fix a recently introduced regression in blk_mq_realloc_hw_ctxs()"). This test only runs if a recently added fault injection feature is available, namely commit 596444e75705 ("null_blk: Add support for init_hctx() fault injection").
--- tests/block/030 | 42 ++++++++++++++++++++++++++++++++++++++++++ tests/block/030.out | 1 + 2 files changed, 43 insertions(+) create mode 100755 tests/block/030 create mode 100644 tests/block/030.out
tests/block/030 new file +42 -0
@@ -0,0 +1,42 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2020 Google LLC +# +# Trigger the blk_mq_realloc_hw_ctxs() error path. + +. tests/block/rc +. common/null_blk + +DESCRIPTION="trigger the blk_mq_realloc_hw_ctxs() error path" +QUICK=1 + +requires() { + _have_null_blk || return $? + _have_module_param null_blk init_hctx || return $? +} + +test() { + local i sq=/sys/kernel/config/nullb/nullb0/submit_queues + + : "${TIMEOUT:=30}" + if ! _init_null_blk nr_devices=0 queue_mode=2 "init_hctx=$(nproc),100,0,0"; then + echo "Loading null_blk failed" + return 1 + fi + if ! _configure_null_blk nullb0 completion_nsec=0 blocksize=512 size=16\ + submit_queues="$(nproc)" memory_backed=1 power=1; then + echo "Configuring null_blk failed" + return 1 + fi + if { echo "$(<"$sq")" >$sq; } 2>/dev/null; then + for ((i=0;i<100;i++)); do + echo 1 >$sq + nproc >$sq + done + else + SKIP_REASON="Skipping test because $sq cannot be modified" + fi + rmdir /sys/kernel/config/nullb/nullb0 + _exit_null_blk + echo Passed +}
tests/block/030.out new file +1 -0
@@ -0,0 +1 @@ +Passed
Hi Bart,
Quoted 5 lines · Bart Van Assche
Add a test that triggers the code touched by commit
d0930bb8f46b("blk-mq:
Fix a recently introduced regression inblk_mq_realloc_hw_ctxs()"). This test only runs if a recently added fault injection feature is available, namely commit596444e75705("null_blk: Add support forinit_hctx()fault injection").
[...]
tests/block/030 line 22
Quoted 5 lines
tests/block/030 new file +5
-0+test() { + local i sq=/sys/kernel/config/nullb/nullb0/submit_queues + + : "${TIMEOUT:=30}" + if ! _init_null_blk nr_devices=0 queue_mode=2 "init_hctx=$(nproc),100,0,0"; then
Doesn't make the $(nproc) the test subtil depending on the execution environment?
Thanks,
Daniel
Quoted 9 lines · Daniel Wagner
Quoted 5 lines · Bart Van Assche
Add a test that triggers the code touched by commit
d0930bb8f46b("blk-mq:
Fix a recently introduced regression inblk_mq_realloc_hw_ctxs()"). This test only runs if a recently added fault injection feature is available, namely commit596444e75705("null_blk: Add support forinit_hctx()fault injection").[...]
Quoted 8 lines
tests/block/030 line 22
Quoted 5 lines
tests/block/030 new file +5
-0+test() { + local i sq=/sys/kernel/config/nullb/nullb0/submit_queues + + : "${TIMEOUT:=30}" + if ! _init_null_blk nr_devices=0 queue_mode=2 "init_hctx=$(nproc),100,0,0"; thenDoesn't make the $(nproc) the test subtil depending on the execution environment?
Hi Daniel,
The value $(nproc) has been chosen on purpose. The following code from the test script:
+ echo 1 >$sq
+ nproc >$sq
triggers (nproc + 1) calls to null_init_hctx(). So injecting a failure after (nproc) null_init_hctx() calls triggers the following pattern:
* The first blk_mq_realloc_hw_ctxs() call fails after (nproc - 1) null_init_hctx() calls.
* The second blk_mq_realloc_hw_ctxs() call fails after (nproc - 2) null_init_hctx() calls.
...
* The (nproc) th blk_mq_realloc_hw_ctxs() call fails after one null_init_hctx() call.
* The (nproc + 1) th blk_mq_realloc_hw_ctxs() call succeeds.
I'm not sure to trigger this behavior without using the $(nproc) value?
Thanks,
Bart.
Hi Bart,
Quoted 7 lines · Bart Van Assche
Quoted 6 lines · Daniel Wagner
tests/block/030 line 22
Quoted 5 lines · Bart Van Assche
tests/block/030 new file +5
-0+test() { + local i sq=/sys/kernel/config/nullb/nullb0/submit_queues + + : "${TIMEOUT:=30}" + if ! _init_null_blk nr_devices=0 queue_mode=2 "init_hctx=$(nproc),100,0,0"; then
From the kernel code:
/* "<interval>,<probability>,<space>,<times>" */
Don't you need to set the times attribute to -1 in order to inject the everytime the interval is reached? If I understood it correctly, with times=0 no failure is injected.
BTW, I've had to change it to init_hctx=$(($(nproc)+1)) to pass the initial __configure_null_blk call before the first fail hits.
Quoted 19 lines
Quoted 2 lines
Doesn't make the $(nproc) the test subtil depending on the execution environment?
The value $(nproc) has been chosen on purpose. The following code from the test script:
tests/block/030 new file +2
-0+ echo 1 >$sq + nproc >$sqtriggers (nproc + 1) calls to null_init_hctx().So injecting a failure after (nproc)
null_init_hctx()calls triggers the following pattern:
* The firstblk_mq_realloc_hw_ctxs()call fails after (nproc - 1)null_init_hctx()calls.
* The secondblk_mq_realloc_hw_ctxs()call fails after (nproc - 2)null_init_hctx()calls.
...
* The (nproc) thblk_mq_realloc_hw_ctxs()call fails after onenull_init_hctx()call.
* The (nproc + 1) thblk_mq_realloc_hw_ctxs()call succeeds.
Quoted 1 line
I'm not sure to trigger this behavior without using the $(nproc) value?
Okay, I get the idea how you want to test.
Is the dependency on nproc because null_blk expects submit_queue <= online cpus?
Though why the 100?
for ((i=0;i<100;i++)); do echo 1 >$sq nproc >$sq done
And shouldn't be there a test for error?
Thanks,
Daniel
Quoted 13 lines · Daniel Wagner
Hi Bart,
Quoted 9 lines · Bart Van Assche
Quoted 6 lines · Daniel Wagner
tests/block/030 line 22
Quoted 5 lines · Bart Van Assche
tests/block/030 new file +5
-0+test() { + local i sq=/sys/kernel/config/nullb/nullb0/submit_queues + + : "${TIMEOUT:=30}" + if ! _init_null_blk nr_devices=0 queue_mode=2 "init_hctx=$(nproc),100,0,0"; thenFrom the kernel code:
Quoted 8 lines
/* "<interval>,<probability>,<space>,<times>" */Don't you need to set the times attribute to -1 in order to inject the everytime the interval is reached? If I understood it correctly, with times=0 no failure is injected.
BTW, I've had to change it to init_hctx=$(($(nproc)+1)) to pass the initial
__configure_null_blkcall before the first fail hits.
Hi Daniel,
I will make both changes in the init_hctx string. Not sure how this escaped from my attention.
Quoted 19 lines
Quoted 19 lines
Quoted 2 lines
Doesn't make the $(nproc) the test subtil depending on the execution environment?
The value $(nproc) has been chosen on purpose. The following code from the test script:
tests/block/030 new file +2
-0+ echo 1 >$sq + nproc >$sqtriggers (nproc + 1) calls to null_init_hctx().So injecting a failure after (nproc)
null_init_hctx()calls triggers the following pattern:
* The firstblk_mq_realloc_hw_ctxs()call fails after (nproc - 1)null_init_hctx()calls.
* The secondblk_mq_realloc_hw_ctxs()call fails after (nproc - 2)null_init_hctx()calls.
...
* The (nproc) thblk_mq_realloc_hw_ctxs()call fails after onenull_init_hctx()call.
* The (nproc + 1) thblk_mq_realloc_hw_ctxs()call succeeds.
Quoted 6 lines
Quoted 1 line
I'm not sure to trigger this behavior without using the $(nproc) value?
Okay, I get the idea how you want to test.
Is the dependency on nproc because
null_blkexpectssubmit_queue<= online cpus?
That's correct. I want to test with the maximum number of submit queues allowed, hence the use of $(nproc).
Quoted 6 lines
Though why the 100?
for ((i=0;i<100;i++)); do echo 1 >$sq nproc >$sq done
No particular reason other than "a significant number of iterations".
Quoted 1 line
And shouldn't be there a test for error?
All I want to test is the absence of kernel crashes. The blktests framework already inspects dmesg for the absence of kernel crashes. So I don't think that I have to check whether or not the quoted sysfs writes succeed.
Thanks,
Bart.
Hi Bart,
Quoted 5 lines
Quoted 2 lines
Is the dependency on nproc because
null_blkexpectssubmit_queue<= online cpus?That's correct. I want to test with the maximum number of submit queues allowed, hence the use of $(nproc).
Ok, I'd probably add this info as comment or have it in the commit message. Just in case someone in the future wonders why nproc.
Quoted 10 lines
Quoted 6 lines
Though why the 100?
for ((i=0;i<100;i++)); do echo 1 >$sq nproc >$sq doneNo particular reason other than "a significant number of iterations".
Quoted 1 line
And shouldn't be there a test for error?
Ah ok. I was getting paranoid about numbers :)
Quoted 4 lines
All I want to test is the absence of kernel crashes. The blktests framework already inspects dmesg for the absence of kernel crashes. So I don't think that I have to check whether or not the quoted sysfs writes succeed.
Maybe adding this info to the commit message what this test tries to cover wouldn't hurt.
Thanks,
Daniel