blktests: replace module removal with patient module removal
PATCH v2 1/1 blktests: replace module removal with patient module removal
A long time ago, in a galaxy far, far away...
I ran into some odd scsi_debug false positives with fstests. This prompted me to look into them given these false positives prevents me from moving forward with establishing a test baseline with high number of cycles. That is, this stupid issue was prevening creating high confidence in testing.
I reported it [0] and exchanged some ideas with Doug. However, in the end, despite efforts to help things with scsi_debug there were still issues lingering which seemed to defy our expectations upstream. One of the last hanging fruit issues is and always has been that userspace expectations for proper module removal has been broken, so in the end I have demonstrated this is a generic issue [1].
Long ago a WAIT option for module removal was added... that was then removed as it was deemed not needed as folks couldn't figure out when these races happened. The races are actually pretty easy to trigger, it was just never properly documented. A simpe blkdev_open() will easily bump a module refcnt, and these days many thing scan do that sort of thing.
The proper solution is to implement then a patient module removal on kmod and patches have been sent for that and those patches are under review. In the meantime we need a work around to open code a similar solution for users of old versions of kmod. I sent an open coded solution for fstests about since August 19th and has been used there for a few months now. Now that that stuff is merged and tested in fstests with more exposure, its time to match parity on blktests.
I've tested blktests with this for things which I can run virtually for a while now. More wider testig is welcomed.
[0] https://bugzilla.kernel.org/show_bug.cgi?id=212337
[1] https://bugzilla.kernel.org/show_bug.cgi?id=214015
---
This v2:
Goes tested with a series of fixes for nvme and srp. It also goes tested against shellcheck. The biggest change is to adapt the patient rmmod check support into a helper has_modprobe_patient(). The other bigger change is to ensure we use nvme_loop when using the patient removal as the directory we use to for the recfct *is* case sensitive, we can't use aliases.
I've tested this with as many things I could now and I can't find any regressions. For those curious my baseline for v5.17-rc can be found here:
https://github.com/mcgrof/kdevops/blob/master/workflows/blktests/expunges/5.17-rc7/failures.txt
common/multipath-over-rdma | 11 +-- common/null_blk | 9 ++- common/rc | 153 ++++++++++++++++++++++++++++++++++--- common/scsi_debug | 9 ++- tests/nvme/rc | 12 +-- tests/nvmeof-mp/rc | 15 ++-- tests/srp/rc | 19 ++--- 7 files changed, 176 insertions(+), 52 deletions(-)
common/multipath-over-rdma +3 -8
@@ -4,6 +4,7 @@ # # Functions and global variables used by both the srp and nvmeof-mp tests. +. common/rc . common/shellcheck . common/null_blk
@@ -427,14 +428,8 @@ stop_soft_rdma() { echo "$i ..." rdma link del "${i}" || echo "Failed to remove ${i}" done - if ! unload_module rdma_rxe 10; then - echo "Unloading rdma_rxe failed" - return 1 - fi - if ! unload_module siw 10; then - echo "Unloading siw failed" - return 1 - fi + _patient_rmmod rdma_rxe || return 1 + _patient_rmmod siw || return 1 } >>"$FULL" }
common/null_blk +5 -4
@@ -4,6 +4,8 @@ # # null_blk helper functions. +. common/rc + _have_null_blk() { _have_modules null_blk }
@@ -21,9 +23,8 @@ _init_null_blk() { local zoned="" if (( RUN_FOR_ZONED )); then zoned="zoned=1"; fi - if ! modprobe -r null_blk || ! modprobe null_blk "$@" "${zoned}" ; then - return 1 - fi + _patient_rmmod null_blk || return 1 + modprobe null_blk "$@" "${zoned}" || 1 udevadm settle return 0
@@ -46,5 +47,5 @@ _configure_null_blk() { _exit_null_blk() { _remove_null_blk_devices udevadm settle - modprobe -r null_blk + _patient_rmmod null_blk }
common/rc +143 -10
158 lines not on this page · raw message
common/scsi_debug +5 -4
@@ -4,14 +4,15 @@ # # scsi_debug helper functions. +. common/rc + _have_scsi_debug() { _have_modules scsi_debug } _init_scsi_debug() { - if ! modprobe -r scsi_debug || ! modprobe scsi_debug "$@"; then - return 1 - fi + _patient_rmmod scsi_debug || return 1 + modprobe scsi_debug "$@" || return 1 udevadm settle
@@ -51,5 +52,5 @@ _exit_scsi_debug() { unset SCSI_DEBUG_TARGETS unset SCSI_DEBUG_DEVICES udevadm settle - modprobe -r scsi_debug + _patient_rmmod scsi_debug }
tests/nvme/rc +7 -5
@@ -125,11 +125,13 @@ _cleanup_nvmet() { shopt -u nullglob trap SIGINT - modprobe -r nvme-"${nvme_trtype}" 2>/dev/null - if [[ "${nvme_trtype}" != "loop" ]]; then - modprobe -r nvmet-"${nvme_trtype}" 2>/dev/null - fi - modprobe -r nvmet 2>/dev/null + if [[ "${nvme_trtype}" == "loop" ]]; then + _patient_rmmod nvme_"${nvme_trtype}" + else + _patient_rmmod nvme-"${nvme_trtype}" + _patient_rmmod nvmet-"${nvme_trtype}" + fi + _patient_rmmod nvmet 2>/dev/null if [[ "${nvme_trtype}" == "rdma" ]]; then stop_soft_rdma fi
tests/nvmeof-mp/rc +8 -7
@@ -164,12 +164,13 @@ start_nvme_client() { } stop_nvme_client() { - unload_module nvme-rdma || return $? - unload_module nvme-fabrics || return $? + _patient_rmmod nvme-rdma || return 1 + _patient_rmmod nvme-fabrics || return 1 # Ignore nvme and nvme-core unload errors - this test may be run on a # system equipped with one or more NVMe SSDs. - unload_module nvme >&/dev/null - unload_module nvme-core >&/dev/null + export MODPROBE_PATIENT_RM_TIMEOUT_SECONDS=5 + _patient_rmmod nvme + _patient_rmmod nvme-core return 0 }
@@ -269,9 +270,9 @@ stop_nvme_target() { rmdir "$d" done ) - unload_module nvmet_rdma && - unload_module nvmet && - _exit_null_blk + _patient_rmmod nvmet_rdma || return 1 + _patient_rmmod nvmet || return 1 + _exit_null_blk } start_target() {
tests/srp/rc +5 -14
@@ -323,19 +323,10 @@ start_srp_ini() { # Unload the SRP initiator driver. stop_srp_ini() { - local i - log_out - for ((i=40;i>=0;i--)); do - remove_mpath_devs || return $? - unload_module ib_srp >/dev/null 2>&1 && break - sleep 1 - done - if [ -e /sys/module/ib_srp ]; then - echo "Error: unloading kernel module ib_srp failed" - return 1 - fi - unload_module scsi_transport_srp || return $? + remove_mpath_devs || return $? + _patient_rmmod ib_srp || return 1 + _patient_rmmod scsi_transport_srp || return $? } # Associate the LIO device with name $1/$2 with file $3 and SCSI serial $4.
@@ -494,7 +485,7 @@ start_lio_srpt() { if modinfo ib_srpt | grep -q '^parm:[[:blank:]]*rdma_cm_port:'; then opts+=("rdma_cm_port=${srp_rdma_cm_port}") fi - unload_module ib_srpt + _patient_rmmod ib_srpt modprobe ib_srpt "${opts[@]}" || return $? i=0 for r in "${vdev_path[@]}"; do
@@ -556,7 +547,7 @@ stop_lio_srpt() { target_core_file target_core_stgt target_core_user \ target_core_mod do - unload_module $m 10 || return $? + _patient_rmmod $m || return $? done }
Signature · 1 line
2.35.1
common/null_blk _init_null_blk() { · line 27
Quoted 5 lines · Luis Chamberlain
common/null_blk +2
-3- if ! modprobe -r null_blk || ! modprobe null_blk "$@" "${zoned}" ; then - return 1 - fi + _patient_rmmod null_blk || return 1 + modprobe null_blk "$@" "${zoned}" || 1
"1" is not a valid command. Should "|| 1" perhaps be changed into "|| return 1"?
common/rc _uptime_s() { · line 329
Quoted 6 lines
common/rc +6
-0+_has_modprobe_patient() +{ + modprobe --help >& /dev/null || return 1 + modprobe --help | grep -q -1 "remove-patiently" || return 1 + return 0 +}
I can't find the meaning of "-1" in the grep man page. Did I perhaps overlook something?
Quoted 8 lines
common/rc +8
-0+# checks the refcount and returns 0 if we can safely remove the module. rmmod +# does this check for us, but we can use this to also iterate checking for this +# refcount before we even try to remove the module. This is useful when using +# debug test modules which take a while to quiesce. +_patient_rmmod_check_refcnt() +{ + local module=$1 + local refcnt=0
common/rc _uptime_s() { · line 378
Quoted 10 lines
common/rc +10
-0+ + if [[ -f "/sys/module/$module/refcnt" ]]; then + refcnt=$(cat "/sys/module/$module/refcnt" 2>/dev/null) + if [[ $? -ne 0 || $refcnt -eq 0 ]]; then + return 0 + fi + return 1 + fi + return 0 +}
Hmm ... why is the check for existence of the refcnt separate from reading the refcnt? I think that just reading the refcnt should be sufficient. Additionally, that will avoid the race where the module is unloaded after the check and before the refcnt is read.
tests/nvme/rc _cleanup_nvmet() { · line 134
Quoted 12 lines
tests/nvme/rc +7
-5- modprobe -r nvme-"${nvme_trtype}" 2>/dev/null - if [[ "${nvme_trtype}" != "loop" ]]; then - modprobe -r nvmet-"${nvme_trtype}" 2>/dev/null - fi - modprobe -r nvmet 2>/dev/null + if [[ "${nvme_trtype}" == "loop" ]]; then + _patient_rmmod nvme_"${nvme_trtype}" + else + _patient_rmmod nvme-"${nvme_trtype}" + _patient_rmmod nvmet-"${nvme_trtype}" + fi + _patient_rmmod nvmet 2>/dev/null
The statement _patient_rmmod nvme-"${nvme_trtype}" occurs twice in the above code. How about preserving the structure of the existing code such that that statement only occurs once?
Quoted 9 lines
# Unload the SRP initiator driver. stop_srp_ini() {tests/srp/rc +0
-6- local i - log_out - for ((i=40;i>=0;i--)); do - remove_mpath_devs || return $? - unload_module ib_srp >/dev/null 2>&1 && break - sleep 1
tests/srp/rc start_srp_ini() { · line 330
Quoted 10 lines
tests/srp/rc +3
-6- done - if [ -e /sys/module/ib_srp ]; then - echo "Error: unloading kernel module ib_srp failed" - return 1 - fi - unload_module scsi_transport_srp || return $? + remove_mpath_devs || return $? + _patient_rmmod ib_srp || return 1 + _patient_rmmod scsi_transport_srp || return $? }
Removing the loop from around remove_mpath_devs is wrong. It is important that that loop is preserved.
Thanks,
Bart.
Quoted 9 lines · Bart Van Assche
common/null_blk _init_null_blk() { · line 27
Quoted 5 lines · Luis Chamberlain
common/null_blk +2
-3- if ! modprobe -r null_blk || ! modprobe null_blk "$@" "${zoned}" ; then - return 1 - fi + _patient_rmmod null_blk || return 1 + modprobe null_blk "$@" "${zoned}" || 1"1" is not a valid command. Should "|| 1" perhaps be changed into "|| return 1"?
That was a typo, fixed.
Quoted 9 lines
common/rc _uptime_s() { · line 329
Quoted 6 lines
common/rc +6
-0+_has_modprobe_patient() +{ + modprobe --help >& /dev/null || return 1 + modprobe --help | grep -q -1 "remove-patiently" || return 1 + return 0 +}I can't find the meaning of "-1" in the grep man page. Did I perhaps overlook something?
That's shorthand for -C 1, but we can remove it as it is not needed.
Quoted 15 lines
Quoted 15 lines
common/rc +15
-0+# checks the refcount and returns 0 if we can safely remove the module. rmmod +# does this check for us, but we can use this to also iterate checking for this +# refcount before we even try to remove the module. This is useful when using +# debug test modules which take a while to quiesce. +_patient_rmmod_check_refcnt() +{ + local module=$1 + local refcnt=0 + + if [[ -f "/sys/module/$module/refcnt" ]]; then + refcnt=$(cat "/sys/module/$module/refcnt" 2>/dev/null) + if [[ $? -ne 0 || $refcnt -eq 0 ]]; then + return 0 + fi + return 1
Quoted 8 lines
common/rc _uptime_s() { · line 378
Quoted 3 lines
common/rc +3
-0+ fi + return 0 +}Hmm ... why is the check for existence of the refcnt separate from reading the refcnt? I think that just reading the refcnt should be sufficient. Additionally, that will avoid the race where the module is unloaded after the check and before the refcnt is read.
We can certainly simplify it as follows:
_patient_rmmod_check_refcnt()
{
local module=$1
local refcnt=0
refcnt=$(cat "/sys/module/$module/refcnt" 2>/dev/null)
if [[ $? -ne 0 || $refcnt -eq 0 ]]; then
return 0
fi
return 1
}Quoted 8 lines
Quoted 8 lines
tests/nvme/rc +3
-5- modprobe -r nvme-"${nvme_trtype}" 2>/dev/null - if [[ "${nvme_trtype}" != "loop" ]]; then - modprobe -r nvmet-"${nvme_trtype}" 2>/dev/null - fi - modprobe -r nvmet 2>/dev/null + if [[ "${nvme_trtype}" == "loop" ]]; then + _patient_rmmod nvme_"${nvme_trtype}" + else
Quoted 8 lines
tests/nvme/rc _cleanup_nvmet() { · line 134
Quoted 4 lines
tests/nvme/rc +4
-0+ _patient_rmmod nvme-"${nvme_trtype}" + _patient_rmmod nvmet-"${nvme_trtype}" + fi + _patient_rmmod nvmet 2>/dev/nullThe statement
_patient_rmmodnvme-"${nvme_trtype}" occurs twice in the above code. How about preserving the structure of the existing code such that that statement only occurs once?
There is one call for nvme-"${nvme_trtype}", the other is for the underscore version, so there are no two calls.
Did I miss something?
Quoted 14 lines
Quoted 14 lines
# Unload the SRP initiator driver. stop_srp_ini() {tests/srp/rc +0
-11- local i - log_out - for ((i=40;i>=0;i--)); do - remove_mpath_devs || return $? - unload_module ib_srp >/dev/null 2>&1 && break - sleep 1 - done - if [ -e /sys/module/ib_srp ]; then - echo "Error: unloading kernel module ib_srp failed" - return 1 - fi
Quoted 8 lines
tests/srp/rc start_srp_ini() { · line 330
Quoted 5 lines
tests/srp/rc +3
-1- unload_module scsi_transport_srp || return $? + remove_mpath_devs || return $? + _patient_rmmod ib_srp || return 1 + _patient_rmmod scsi_transport_srp || return $? }Removing the loop from around
remove_mpath_devsis wrong. It is important that that loop is preserved.
Why ? Can you test and verify?
I can explain why I'm removing it. Code which typically used to try to insist on a module removal were just running into situations where the refcnt got bumped but they could not explain why, and the reason is that module refcnt's are finicky. *Anything* in userspace can easily trigger this, and this is module specific. While doing a module removal, if you are running into this, the only thing you can do is to patiently wait until userspace is done with whatever it may try to do. The old re-try attempts are buggy because of this.
A future mechansim for kmod will be to allow userspace to try to remove first the modules which hold a refcnt, but that does not do anything for whatever userspace might do. These entry points from userspace are completely module specific and cannot be generalized (nor do I think we want to add semantics to the kernelf or it).
So the only thing one can sensibly do, specially in test frameworks, is to wait and use a timeout for it.
Luis
Quoted 19 lines · Luis Chamberlain
Quoted 17 lines · Bart Van Assche
tests/nvme/rc _cleanup_nvmet() { · line 134
Quoted 12 lines · Luis Chamberlain
tests/nvme/rc +7
-5- modprobe -r nvme-"${nvme_trtype}" 2>/dev/null - if [[ "${nvme_trtype}" != "loop" ]]; then - modprobe -r nvmet-"${nvme_trtype}" 2>/dev/null - fi - modprobe -r nvmet 2>/dev/null + if [[ "${nvme_trtype}" == "loop" ]]; then + _patient_rmmod nvme_"${nvme_trtype}" + else + _patient_rmmod nvme-"${nvme_trtype}" + _patient_rmmod nvmet-"${nvme_trtype}" + fi + _patient_rmmod nvmet 2>/dev/nullThe statement
_patient_rmmodnvme-"${nvme_trtype}" occurs twice in the above code. How about preserving the structure of the existing code such that that statement only occurs once?
Quoted 4 lines
There is one call for nvme-"${nvme_trtype}", the other is for the underscore version, so there are no two calls.
Did I miss something?
It's only now that I see the underscore/hyphen difference in the if and else branches. It is not clear to me why this behavior change has been introduced? The following command produces no output on my test setup:
find /lib/modules -name 'nvme_*'
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
# Unload the SRP initiator driver. stop_srp_ini() {tests/srp/rc +0
-11- local i - log_out - for ((i=40;i>=0;i--)); do - remove_mpath_devs || return $? - unload_module ib_srp >/dev/null 2>&1 && break - sleep 1 - done - if [ -e /sys/module/ib_srp ]; then - echo "Error: unloading kernel module ib_srp failed" - return 1 - fiQuoted 8 lines
tests/srp/rc start_srp_ini() { · line 330
Quoted 5 lines
tests/srp/rc +3
-1- unload_module scsi_transport_srp || return $? + remove_mpath_devs || return $? + _patient_rmmod ib_srp || return 1 + _patient_rmmod scsi_transport_srp || return $? }Removing the loop from around
remove_mpath_devsis wrong. It is important that that loop is preserved.
Quoted 1 line
Why ? Can you test and verify?
If I remember correctly I put remove_mpath_devs call inside the loop because multipathd keeps running while the loop is ongoing and hence can modify paths while the loop is running.
Thanks,
Bart.
Quoted 23 lines · Bart Van Assche
Quoted 22 lines · Luis Chamberlain
Quoted 17 lines · Bart Van Assche
Review
Quoted 12 lines · Luis Chamberlain
(quoted diff) +7
-5- modprobe -r nvme-"${nvme_trtype}" 2>/dev/null - if [[ "${nvme_trtype}" != "loop" ]]; then - modprobe -r nvmet-"${nvme_trtype}" 2>/dev/null - fi - modprobe -r nvmet 2>/dev/null + if [[ "${nvme_trtype}" == "loop" ]]; then + _patient_rmmod nvme_"${nvme_trtype}" + else + _patient_rmmod nvme-"${nvme_trtype}" + _patient_rmmod nvmet-"${nvme_trtype}" + fi + _patient_rmmod nvmet 2>/dev/nullThe statement
_patient_rmmodnvme-"${nvme_trtype}" occurs twice in the above code. How about preserving the structure of the existing code such that that statement only occurs once?There is one call for nvme-"${nvme_trtype}", the other is for the underscore version, so there are no two calls.
Quoted 7 lines
Quoted 1 line
Did I miss something?
It's only now that I see the underscore/hyphen difference in the if and else branches. It is not clear to me why this behavior change has been introduced? The following command produces no output on my test setup:
find
/lib/modules-name 'nvme_*'
Try:
find /sys/module/ -maxdepth 1 -type d | grep loop
I get:/sys/module/nvme_loop
Blame scripts/Makefile.lib which does the s|-|_| for KBUILD_MODNAME:
name-fix-token = $(subst $(comma),_,$(subst -,_,$1)) name-fix = $(call stringify,$(call name-fix-token,$1)) basename_flags = -DKBUILD_BASENAME=$(call name-fix,$(basetarget))
modname_flags = -DKBUILD_MODNAME=$(call name-fix,$(modname)) \
-D__KBUILD_MODNAME=kmod_$(call name-fix-token,$(modname))I'm thinking now it is best to do this hack here:
common/rc +4 -1
@@ -390,7 +390,10 @@ _patient_rmmod_check_refcnt() # ourselves. _patient_rmmod() { - local module=$1 + # Since we are looking for a directory we must adopt the + # specific directory used by scripts/Makefile.lib for + # KBUILD_MODNAME + local module=$(echo $1 | sed 's|-|_|g') local max_tries_max=$MODPROBE_PATIENT_RM_TIMEOUT_SECONDS local max_tries=0 local mod_ret=0
That would allow us to keep the old order, and in fact would be correct for the other targets as well.
Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 9 lines
# Unload the SRP initiator driver. stop_srp_ini() {(quoted diff) +0
-6- local i - log_out - for ((i=40;i>=0;i--)); do - remove_mpath_devs || return $? - unload_module ib_srp >/dev/null 2>&1 && break - sleep 1Review
Quoted 10 lines
(quoted diff) +3
-6- done - if [ -e /sys/module/ib_srp ]; then - echo "Error: unloading kernel module ib_srp failed" - return 1 - fi - unload_module scsi_transport_srp || return $? + remove_mpath_devs || return $? + _patient_rmmod ib_srp || return 1 + _patient_rmmod scsi_transport_srp || return $? }Removing the loop from around
remove_mpath_devsis wrong. It is important that that loop is preserved.
Quoted 5 lines
Quoted 1 line
Why ? Can you test and verify?
If I remember correctly I put
remove_mpath_devscall inside the loop because multipathd keeps running while the loop is ongoing and hence can modify paths while the loop is running.
I suspect you ran into the issue of the refcnt being bumped by anything multipathd tried, and not being able to remove the module, but if it is adding *new* mpath devices that would seem like a bug which we'd need to address. The point to the patient module removal is to keep on trying until the recnt is 0 and if that fails wait and keep trying, until the the timeout. Anytihng userspace does, say multipathd does, like bdev_open(), would be yielded to.
So I'd like to keep this change as it is exactly the sort of hack I am chasing after with this crusade.
Let me know how you'd like to proceed so I can punt a v3.
Luis
Quoted 8 lines · Luis Chamberlain
I suspect you ran into the issue of the refcnt being bumped by anything multipathd tried, and not being able to remove the module, but if it is adding *new* mpath devices that would seem like a bug which we'd need to address. The point to the patient module removal is to keep on trying until the recnt is 0 and if that fails wait and keep trying, until the the timeout. Anytihng userspace does, say multipathd does, like
bdev_open(), would be yielded to.
Quoted 4 lines
So I'd like to keep this change as it is exactly the sort of hack I am chasing after with this crusade.
Let me know how you'd like to proceed so I can punt a v3.
Please implement the patient module removal and the stop_srp_ini() behavior changes as separate patches such that the stop_srp_ini() behavioral changes can be reverted easily in case these would trigger a regression.
Thanks,
Bart.
Quoted 9 lines · Bart Van Assche
Quoted 8 lines · Luis Chamberlain
I suspect you ran into the issue of the refcnt being bumped by anything multipathd tried, and not being able to remove the module, but if it is adding *new* mpath devices that would seem like a bug which we'd need to address. The point to the patient module removal is to keep on trying until the recnt is 0 and if that fails wait and keep trying, until the the timeout. Anytihng userspace does, say multipathd does, like
bdev_open(), would be yielded to.
Quoted 8 lines
Quoted 4 lines
So I'd like to keep this change as it is exactly the sort of hack I am chasing after with this crusade.
Let me know how you'd like to proceed so I can punt a v3.
Please implement the patient module removal and the
stop_srp_ini()behavior changes as separate patches such that thestop_srp_ini()behavioral changes can be reverted easily in case these would trigger a regression.
I had dropped the ball here, because well, it takes a bit of time to re-test everything. And also just around this time I was also ironing out how to properly automate testing for srp. That's all done now and I finally had time to re-test so will send a v3 out shortly.
Luis