selftests: livepatch: filter kobject debug messages in check_result()
Subsystems (2)
PATCH 1/1 selftests: livepatch: filter kobject debug messages in check_result()
CONFIG_DEBUG_KOBJECT makes kobject_add_internal(), kobject_uevent_env(), fill_kobj_path() and friends emit pr_debug() messages, and CONFIG_DEBUG_KOBJECT_RELEASE makes kobject_release() emit a pr_info() for every delayed kobject free. All of these carry the "kobject:" prefix via pr_fmt(), e.g.:
kobject: 'test_klp_livepatch' (ffff...): kobject_release, parent ... (delayed N)
kobject: 'test_klp_livepatch' (ffff...): kobject_add_internal: parent: 'module', set: 'module'
The livepatch test modules' kobjects are named "test_klp_*", so these lines match the check_result() grep for "test_klp" and leak into the result. The extra lines no longer match the expected output, so the selftests fail when either debug config is enabled.
Filter out the "kobject:" lines from the result and note in the README that both configs are safe to keep enabled.
--- tools/testing/selftests/livepatch/README | 3 +++ tools/testing/selftests/livepatch/functions.sh | 1 + 2 files changed, 4 insertions(+)
tools/testing/selftests/livepatch/README +3 -0
@@ -15,6 +15,9 @@ Config Set CONFIG_LIVEPATCH=y option and it's prerequisites. +Note: CONFIG_DEBUG_KOBJECT and CONFIG_DEBUG_KOBJECT_RELEASE emit "kobject:" +debug messages to the kernel log. The tests filter these out, so they are safe +to keep enabled. Building the tests ------------------
tools/testing/selftests/livepatch/functions.sh +1 -0
@@ -327,6 +327,7 @@ function check_result { result=$(dmesg | awk -v last_dmesg="$LAST_DMESG" 'p; $0 == last_dmesg { p=1 }' | \ grep -e 'livepatch:' -e 'test_klp' | \ grep -v '\(tainting\|taints\) kernel' | \ + grep -v 'kobject:' | \ sed 's/^\[[ 0-9.]*\] //' | \ sed 's/^\[[ ]*[CT][0-9]*\] //')
Signature · 1 line
2.52.0
Petr Mladek
maintainerQuoted 7 lines · Yafang Shao
CONFIG_DEBUG_KOBJECTmakeskobject_add_internal(),kobject_uevent_env(),fill_kobj_path()and friends emitpr_debug()messages, andCONFIG_DEBUG_KOBJECT_RELEASEmakeskobject_release()emit apr_info()for every delayed kobject free. All of these carry the "kobject:" prefix viapr_fmt(), e.g.:kobject: '
test_klp_livepatch' (ffff...):kobject_release, parent ... (delayed N)
Yeah, this is pr_info() printed only when CONFIG_DEBUG_KOBJECT_RELEASE enabled.
Quoted 1 line
kobject: '
test_klp_livepatch' (ffff...):kobject_add_internal: parent: 'module', set: 'module'
This one seems to be printed by pr_debug(). It is printed with CONFIG_DEBUG_KOBJECT but only when CONFIG_DYNAMIC_DEBUG is not enabled.
Quoted 7 lines
The livepatch test modules' kobjects are named "test_klp_*", so these lines match the
check_result()grep for "test_klp" and leak into the result. The extra lines no longer match the expected output, so the selftests fail when either debug config is enabled.Filter out the "kobject:" lines from the result and note in the README that both configs are safe to keep enabled.
Honestly, I do not feel comfortable with this ignore-all-kobject-messages approach. We might miss real problems reported by the kobject API.
It seems acceptable to ignore the quite specific "kobject: .*parent.* (delayed" pattern.
But I would prefer to ignore the other debug messages by detecting the KERN_DEBUG loglevel. It seems that dmesg even allows to filter the output on its own. It is a bit ugly because we want all other loglevels but:
dmesg --level=info,notice,warn,err,crit,alert,emerg
might do the job.
Best Regards,
Petr
Quoted 23 lines · Petr Mladek
Quoted 7 lines · Yafang Shao
CONFIG_DEBUG_KOBJECTmakeskobject_add_internal(),kobject_uevent_env(),fill_kobj_path()and friends emitpr_debug()messages, andCONFIG_DEBUG_KOBJECT_RELEASEmakeskobject_release()emit apr_info()for every delayed kobject free. All of these carry the "kobject:" prefix viapr_fmt(), e.g.:kobject: '
test_klp_livepatch' (ffff...):kobject_release, parent ... (delayed N)Yeah, this is
pr_info()printed only whenCONFIG_DEBUG_KOBJECT_RELEASEenabled.Quoted 1 line
kobject: '
test_klp_livepatch' (ffff...):kobject_add_internal: parent: 'module', set: 'module'This one seems to be printed by
pr_debug(). It is printed withCONFIG_DEBUG_KOBJECTbut only whenCONFIG_DYNAMIC_DEBUGis not enabled.Quoted 5 lines
The livepatch test modules' kobjects are named "test_klp_*", so these lines match the
check_result()grep for "test_klp" and leak into the result. The extra lines no longer match the expected output, so the selftests fail when either debug config is enabled.
Quoted 8 lines
Quoted 2 lines
Filter out the "kobject:" lines from the result and note in the README that both configs are safe to keep enabled.
Honestly, I do not feel comfortable with this ignore-all-kobject-messages approach. We might miss real problems reported by the kobject API.
It seems acceptable to ignore the quite specific "kobject: .*parent.* (delayed" pattern.
These messages are printed using pr_info:
pr_info("'%s' (%p): %s, parent %p (delayed %ld)\n",
kobject_name(kobj), kobj, __func__, kobj->parent, delay);
In addition to the specific "kobject: .*parent.*(delayed" pattern, several other pr_debug messages are also produced.
For example:
# --- expected
# +++ result
# @@ -1,7 +1,13 @@
# % insmod test_modules/test_klp_livepatch.ko
# +kobject: 'test_klp_livepatch' (000000009dbf565e):kobject_add_internal: parent: 'module', set: 'module' # +kobject: 'holders' (00000000d9fe84ca): kobject_add_internal:
parent: 'test_klp_livepatch', set: '<NULL>'
# +kobject: 'test_klp_livepatch' (00000000dcae1113):kobject_add_internal: parent: 'livepatch', set: '<NULL>' # +kobject: 'vmlinux' (000000007b8837e6): kobject_add_internal:
parent: 'test_klp_livepatch', set: '<NULL>'
# livepatch: enabling patch 'test_klp_livepatch' # livepatch: 'test_klp_livepatch': initializing patching transition # livepatch: 'test_klp_livepatch': starting patching transition # +kobject: 'test_klp_livepatch' (000000009dbf565e): kobject_uevent_env # +kobject: 'test_klp_livepatch' (000000009dbf565e): fill_kobj_path: path = '/module/test_klp_livepatch' # livepatch: 'test_klp_livepatch': completing patching transition # livepatch: 'test_klp_livepatch': patching complete # % echo 0 > /sys/kernel/livepatch/test_klp_livepatch/enabled # @@ -9,4 +15,15 @@ livepatch: 'test_klp_livepatch': initial # livepatch: 'test_klp_livepatch': starting unpatching transition # livepatch: 'test_klp_livepatch': completing unpatching transition # livepatch: 'test_klp_livepatch': unpatching complete
# +kobject: 'test_klp_livepatch' (00000000dcae1113): kobject_cleanup, parent 00000000f8785d63
# +kobject: 'test_klp_livepatch' (00000000dcae1113): auto cleanup kobject_del # +kobject: 'test_klp_livepatch' (00000000dcae1113): calling ktype release # +kobject: 'test_klp_livepatch': free name
# % rmmod test_klp_livepatch
# +kobject: 'test_klp_livepatch' (000000009dbf565e): kobject_cleanup, parent 0000000052e5c022
# +kobject: 'test_klp_livepatch' (000000009dbf565e): auto cleanup kobject_del # +kobject: 'test_klp_livepatch' (000000009dbf565e): auto cleanup 'remove' event # +kobject: 'test_klp_livepatch' (000000009dbf565e): kobject_uevent_env # +kobject: 'test_klp_livepatch' (000000009dbf565e): fill_kobj_path:
path = '/module/test_klp_livepatch'
# +kobject: 'test_klp_livepatch' (000000009dbf565e): calling ktype release # +kobject: 'test_klp_livepatch': free name
Quoted 9 lines
But I would prefer to ignore the other debug messages by detecting the
KERN_DEBUGloglevel. It seems that dmesg even allows to filter the output on its own. It is a bit ugly because we want all other loglevels but:dmesg --level=info,notice,warn,err,crit,alert,emergmight do the job.
The following issues will occur:
# --- expected
# +++ result
# @@ -1,12 +1,8 @@
# % insmod test_modules/test_klp_livepatch.ko
# livepatch: enabling patch 'test_klp_livepatch'
# -livepatch: 'test_klp_livepatch': initializing patching transition # livepatch: 'test_klp_livepatch': starting patching transition # -livepatch: 'test_klp_livepatch': completing patching transition # livepatch: 'test_klp_livepatch': patching complete
# % echo 0 > /sys/kernel/livepatch/test_klp_livepatch/enabled # -livepatch: 'test_klp_livepatch': initializing unpatching transition # livepatch: 'test_klp_livepatch': starting unpatching transition # -livepatch: 'test_klp_livepatch': completing unpatching transition # livepatch: 'test_klp_livepatch': unpatching complete
# % rmmod test_klp_livepatch
This is because these messages are emitted using pr_debug:
pr_debug("'%s': initializing %s transition\n", patch->mod->name,
klp_target_state == KLP_TRANSITION_PATCHED ? "patching" : "unpatching");
pr_debug("'%s': completing %s transition\n",
klp_transition_patch->mod->name,
klp_target_state == KLP_TRANSITION_PATCHED ? "patching" : "unpatching");In summary, we need to filter out the pr_info messages from kobject while retaining the pr_debug messages from livepatch. Therefore, the following change should address this:
(quoted diff) +0 -0
b/tools/testing/selftests/livepatch/functions.sh
index 46991a029f7c..54789fae8055 100644
tools/testing/selftests/livepatch/functions.sh +1 -2
@@ -300,9 +300,9 @@ function start_test { # find new kernel messages since the test started. local last_dmesg_msg="livepatch kselftest timestamp: $(date --rfc-3339=ns)" log "$last_dmesg_msg" - loop_until 'dmesg | grep -q "$last_dmesg_msg"' || + loop_until 'dmesg --raw | grep -q "$last_dmesg_msg"' || die "buffer busy? can't find canary dmesg message:
$last_dmesg_msg"
- LAST_DMESG=$(dmesg | grep "$last_dmesg_msg")
+ LAST_DMESG=$(dmesg --raw | grep "$last_dmesg_msg")
echo -n "TEST: $test ... "
log "===== TEST: $test ====="@@ -318,9 +318,12 @@ function check_result {
# - include lines matching keywords
# - exclude lines matching keywords
# - filter out dmesg timestamp prefixes
- result=$(dmesg | awk -v last_dmesg="$LAST_DMESG" 'p; $0 ==last_dmesg { p=1 }' | \
+ result=$(dmesg --raw | \
+ awk -v last_dmesg="$LAST_DMESG" 'p { if ($0 !~ /^<7>/
|| $0 ~ /livepatch:/) print; next } $0 == last_dmesg { p = 1 }' | \
grep -e 'livepatch:' -e 'test_klp' | \
grep -v '\(tainting\|taints\) kernel' | \
+ grep -v 'kobject: .*parent.*(delayed' | \
+ sed 's/^<[0-9]*>//' | \
sed 's/^\[[ 0-9.]*\] //' | \
sed 's/^\[[ ]*[CT][0-9]*\] //')Signature · 2 lines
Regards Yafang
Petr Mladek
maintainerQuoted 111 lines · Yafang Shao
Quoted 23 lines · Petr Mladek
Quoted 7 lines · Yafang Shao
CONFIG_DEBUG_KOBJECTmakeskobject_add_internal(),kobject_uevent_env(),fill_kobj_path()and friends emitpr_debug()messages, andCONFIG_DEBUG_KOBJECT_RELEASEmakeskobject_release()emit apr_info()for every delayed kobject free. All of these carry the "kobject:" prefix viapr_fmt(), e.g.:kobject: '
test_klp_livepatch' (ffff...):kobject_release, parent ... (delayed N)Yeah, this is
pr_info()printed only whenCONFIG_DEBUG_KOBJECT_RELEASEenabled.Quoted 1 line
kobject: '
test_klp_livepatch' (ffff...):kobject_add_internal: parent: 'module', set: 'module'This one seems to be printed by
pr_debug(). It is printed withCONFIG_DEBUG_KOBJECTbut only whenCONFIG_DYNAMIC_DEBUGis not enabled.Quoted 5 lines
The livepatch test modules' kobjects are named "test_klp_*", so these lines match the
check_result()grep for "test_klp" and leak into the result. The extra lines no longer match the expected output, so the selftests fail when either debug config is enabled.Quoted 8 lines
Quoted 2 lines
Filter out the "kobject:" lines from the result and note in the README that both configs are safe to keep enabled.
Honestly, I do not feel comfortable with this ignore-all-kobject-messages approach. We might miss real problems reported by the kobject API.
It seems acceptable to ignore the quite specific "kobject: .*parent.* (delayed" pattern.
These messages are printed using
pr_info:pr_info("'%s' (%p): %s, parent %p (delayed %ld)\n",
kobject_name(kobj), kobj, __func__, kobj->parent, delay);In addition to the specific "kobject: .*parent.*(delayed" pattern, several other
pr_debugmessages are also produced.
For example:# --- expected
# +++ result
# @@ -1,7 +1,13 @@
# % insmodtest_modules/test_klp_livepatch.ko
# +kobject: 'test_klp_livepatch' (000000009dbf565e):kobject_add_internal: parent: 'module', set: 'module' # +kobject: 'holders' (00000000d9fe84ca):kobject_add_internal:
parent: 'test_klp_livepatch', set: '<NULL>'
# +kobject: 'test_klp_livepatch' (00000000dcae1113):kobject_add_internal: parent: 'livepatch', set: '<NULL>' # +kobject: 'vmlinux' (000000007b8837e6):kobject_add_internal:
parent: 'test_klp_livepatch', set: '<NULL>'# livepatch: enabling patch 'test_klp_livepatch' # livepatch: 'test_klp_livepatch': initializing patching transition # livepatch: 'test_klp_livepatch': starting patching transition # +kobject: 'test_klp_livepatch' (000000009dbf565e): kobject_uevent_env # +kobject: 'test_klp_livepatch' (000000009dbf565e): fill_kobj_path: path = '/module/test_klp_livepatch' # livepatch: 'test_klp_livepatch': completing patching transition # livepatch: 'test_klp_livepatch': patching complete # % echo 0 > /sys/kernel/livepatch/test_klp_livepatch/enabled # @@ -9,4 +15,15 @@ livepatch: 'test_klp_livepatch': initial # livepatch: 'test_klp_livepatch': starting unpatching transition # livepatch: 'test_klp_livepatch': completing unpatching transition # livepatch: 'test_klp_livepatch': unpatching complete# +kobject: '
test_klp_livepatch' (00000000dcae1113):kobject_cleanup, parent00000000f8785d63
# +kobject: 'test_klp_livepatch' (00000000dcae1113): auto cleanupkobject_del# +kobject: 'test_klp_livepatch' (00000000dcae1113): calling ktype release # +kobject: 'test_klp_livepatch': free name
# % rmmodtest_klp_livepatch
# +kobject: 'test_klp_livepatch' (000000009dbf565e):kobject_cleanup, parent0000000052e5c022
# +kobject: 'test_klp_livepatch' (000000009dbf565e): auto cleanupkobject_del# +kobject: 'test_klp_livepatch' (000000009dbf565e): auto cleanup 'remove' event # +kobject: 'test_klp_livepatch' (000000009dbf565e):kobject_uevent_env# +kobject: 'test_klp_livepatch' (000000009dbf565e):fill_kobj_path:
path = '/module/test_klp_livepatch'
# +kobject: 'test_klp_livepatch' (000000009dbf565e): calling ktype release # +kobject: 'test_klp_livepatch': free nameQuoted 9 lines
But I would prefer to ignore the other debug messages by detecting the
KERN_DEBUGloglevel. It seems that dmesg even allows to filter the output on its own. It is a bit ugly because we want all other loglevels but:dmesg --level=info,notice,warn,err,crit,alert,emergmight do the job.
The following issues will occur:
# --- expected
# +++ result
# @@ -1,12 +1,8 @@
# % insmodtest_modules/test_klp_livepatch.ko
# livepatch: enabling patch 'test_klp_livepatch'
# -livepatch: 'test_klp_livepatch': initializing patching transition # livepatch: 'test_klp_livepatch': starting patching transition # -livepatch: 'test_klp_livepatch': completing patching transition # livepatch: 'test_klp_livepatch': patching complete
# % echo 0 >/sys/kernel/livepatch/test_klp_livepatch/enabled# -livepatch: 'test_klp_livepatch': initializing unpatching transition # livepatch: 'test_klp_livepatch': starting unpatching transition # -livepatch: 'test_klp_livepatch': completing unpatching transition # livepatch: 'test_klp_livepatch': unpatching complete
# % rmmodtest_klp_livepatch
Quoted 8 lines
This is because these messages are emitted using
pr_debug:pr_debug("'%s': initializing %s transition\n",
patch->mod->name,klp_target_state == KLP_TRANSITION_PATCHED ? "patching" : "unpatching");pr_debug("'%s': completing %s transition\n", klp_transition_patch->mod->name, klp_target_state == KLP_TRANSITION_PATCHED ? "patching" : "unpatching");
I see.
tools/testing/selftests/livepatch/functions.sh function check_result {
Quoted 24 lines
In summary, we need to filter out the
pr_infomessages from kobject while retaining thepr_debugmessages from livepatch. Therefore, the following change should address this:tools/testing/selftests/livepatch/functions.sh +1
-2@@ -300,9 +300,9 @@ function start_test { # find new kernel messages since the test started. local last_dmesg_msg="livepatch kselftest timestamp: $(date --rfc-3339=ns)" log "$last_dmesg_msg" - loop_until 'dmesg | grep -q "$last_dmesg_msg"' || + loop_until 'dmesg --raw | grep -q "$last_dmesg_msg"' || die "buffer busy? can't find canary dmesg message:$last_dmesg_msg"
tools/testing/selftests/livepatch/functions.sh +1
-1- LAST_DMESG=$(dmesg | grep "$last_dmesg_msg") + LAST_DMESG=$(dmesg --raw | grep "$last_dmesg_msg") echo -n "TEST: $test ... " log "===== TEST: $test ====="@@ -318,9 +318,12 @@ function check_result {
Also we should update the comment, something like:
+ # - exclude debug messages except with "livepatch:" prefix
Quoted 13 lines
tools/testing/selftests/livepatch/functions.sh +4
-1# - include lines matching keywords # - exclude lines matching keywords # - filter out dmesg timestamp prefixes - result=$(dmesg | awk -v last_dmesg="$LAST_DMESG" 'p; $0 == last_dmesg { p=1 }' | \ + result=$(dmesg --raw | \ + awk -v last_dmesg="$LAST_DMESG" 'p { if ($0 !~ /^<7>/ || $0 ~ /livepatch:/) print; next } $0 == last_dmesg { p = 1 }' | \ grep -e 'livepatch:' -e 'test_klp' | \ grep -v '\(tainting\|taints\) kernel' | \ + grep -v 'kobject: .*parent.*(delayed' | \ + sed 's/^<[0-9]*>//' | \ sed 's/^\[[ 0-9.]*\] //' | \
Quoted 1 line
sed 's/^\[[ ]*[CT][0-9]*\] //')
It is more tricky that I have expected. But I think that it is worth it because it covers many other debug messages.
Best Regards,
Petr