bpf: btf: don't log ignored BTF mismatches
PATCH 1/1 bpf: btf: don't log ignored BTF mismatches
Enabling CONFIG_MODULE_ALLOW_BTF_MISMATCH is an indication that BTF mismatches are expected and module loading should proceed anyway. Logging with pr_warn() on every one of these "benign" mismatches creates unnecessary noise when many such modules are loaded. Instead, limit logging to the case where a BTF mismatch actually prevents a module form loading.
--- kernel/bpf/btf.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/bpf/btf.c +4 -3
@@ -7190,11 +7190,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op, } btf =btf_parse_module (mod->name, mod->btf_data, mod->btf_data_size);if (IS_ERR (btf)) { -pr_warn ("failed to validate module [%s] BTF: %ld\n" , - mod->name,PTR_ERR (btf));kfree (btf_mod); -if (!IS_ENABLED (CONFIG_MODULE_ALLOW_BTF_MISMATCH )) +if (!IS_ENABLED (CONFIG_MODULE_ALLOW_BTF_MISMATCH )) { +pr_warn ("failed to validate module [%s] BTF: %ld\n" , + mod->name,PTR_ERR (btf)); err =PTR_ERR (btf); + } goto out; } err =btf_alloc_id (btf);
Signature · 1 line
2.38.1.431.g37b22c650d-goog
Quoted 17 lines · Connor O'Brien
Enabling
CONFIG_MODULE_ALLOW_BTF_MISMATCHis an indication that BTF mismatches are expected and module loading should proceed anyway. Logging withpr_warn()on every one of these "benign" mismatches creates unnecessary noise when many such modules are loaded. Instead, limit logging to the case where a BTF mismatch actually prevents a module form loading.Signed-off-by Connor O'Brien <[email protected]>--- kernel/bpf/btf.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)kernel/bpf/btf.c +0
-0
kernel/bpf/btf.c static int btf_module_notify(struct notifier_block *nb, unsigned long op, · line 7194
Quoted 5 lines
kernel/bpf/btf.c +0
-2@@ -7190,11 +7190,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op, } btf =btf_parse_module (mod->name, mod->btf_data, mod->btf_data_size);if (IS_ERR (btf)) { -pr_warn ("failed to validate module [%s] BTF: %ld\n" , - mod->name,PTR_ERR (btf));
I think such warning still useful even with CONFIG_MODULE_ALLOW_BTF_MISMATCH. Can we use pr_warn_ratelimited instead of pr_warn in the above to avoid excessive warnings?
Quoted 10 lines
kernel/bpf/btf.c +4
-1kfree (btf_mod); -if (!IS_ENABLED (CONFIG_MODULE_ALLOW_BTF_MISMATCH )) +if (!IS_ENABLED (CONFIG_MODULE_ALLOW_BTF_MISMATCH )) { +pr_warn ("failed to validate module [%s] BTF: %ld\n" , + mod->name,PTR_ERR (btf)); err =PTR_ERR (btf); + } goto out; } err =btf_alloc_id (btf);
Quoted 16 lines · Yonghong Song
Quoted 12 lines · Connor O'Brien
Enabling
CONFIG_MODULE_ALLOW_BTF_MISMATCHis an indication that BTF mismatches are expected and module loading should proceed anyway. Logging withpr_warn()on every one of these "benign" mismatches creates unnecessary noise when many such modules are loaded. Instead, limit logging to the case where a BTF mismatch actually prevents a module form loading.Signed-off-by Connor O'Brien <[email protected]>--- kernel/bpf/btf.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
Quoted 15 lines
kernel/bpf/btf.c static int btf_module_notify(struct notifier_block *nb, unsigned long op, · line 7194
Quoted 10 lines
kernel/bpf/btf.c +0
-2@@ -7190,11 +7190,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op, } btf =btf_parse_module (mod->name, mod->btf_data, mod->btf_data_size);if (IS_ERR (btf)) { -pr_warn ("failed to validate module [%s] BTF: %ld\n" , - mod->name,PTR_ERR (btf));I think such warning still useful even with
CONFIG_MODULE_ALLOW_BTF_MISMATCH.
Can we usepr_warn_ratelimitedinstead ofpr_warnin the above to avoid excessive warnings?
I gave this a try on a Pixel 6 but I'm not sure it quite addresses the issue. The amount of logging doesn't seem to decrease much, I think because the interval between loading one mismatched module and the next can be greater than the default rate limit. To my mind, the issue is the total volume of these messages more so than their rate.
For context, Android devices using the GKI may load hundreds of separately-built modules, and BTF mismatches are possible for any/all of these. It was pointed out to me that btf_verifier_log_type can also print several more lines per mismatched module. ~5 lines of logging for each mismatched module can start to add up, in terms of both overhead and the noise added to the kernel logs.
This is more subjective but I think the warnings also read as though this is a more serious failure that might prevent affected modules from working correctly; anecdotally, I've gotten multiple questions about them asking if something is broken. This can be a red herring for anyone unfamiliar with BTF who is reading the logs to debug unrelated issues. In the CONFIG_MODULE_ALLOW_BTF_MISMATCH=y case the flood of warnings seems out of proportion to the actual result (modules still load successfully, just without debug info) especially since the user has explicitly enabled a config saying they expect mismatches.
If there needs to be some logging in the "mismatch allowed" case, could an acceptable middle ground be to use pr_warn_once to send a single message reporting that mismatches were detected & module BTF debug info might be unavailable? Alternatively, if we could opt out of module BTF loading then that would also avoid this issue, but that's already been proposed before ([1], [2]) so I thought working with the existing config option might be preferred.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/
Quoted 57 lines · Connor O'Brien
Quoted 16 lines · Yonghong Song
Quoted 12 lines · Connor O'Brien
Enabling
CONFIG_MODULE_ALLOW_BTF_MISMATCHis an indication that BTF mismatches are expected and module loading should proceed anyway. Logging withpr_warn()on every one of these "benign" mismatches creates unnecessary noise when many such modules are loaded. Instead, limit logging to the case where a BTF mismatch actually prevents a module form loading.Signed-off-by Connor O'Brien <[email protected]>--- kernel/bpf/btf.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)Quoted 15 lines
kernel/bpf/btf.c static int btf_module_notify(struct notifier_block *nb, unsigned long op, · line 7194
Quoted 10 lines
kernel/bpf/btf.c +0
-2@@ -7190,11 +7190,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op, } btf =btf_parse_module (mod->name, mod->btf_data, mod->btf_data_size);if (IS_ERR (btf)) { -pr_warn ("failed to validate module [%s] BTF: %ld\n" , - mod->name,PTR_ERR (btf));I think such warning still useful even with
CONFIG_MODULE_ALLOW_BTF_MISMATCH.
Can we usepr_warn_ratelimitedinstead ofpr_warnin the above to avoid excessive warnings?I gave this a try on a Pixel 6 but I'm not sure it quite addresses the issue. The amount of logging doesn't seem to decrease much, I think because the interval between loading one mismatched module and the next can be greater than the default rate limit. To my mind, the issue is the total volume of these messages more so than their rate.
For context, Android devices using the GKI may load hundreds of separately-built modules, and BTF mismatches are possible for any/all of these. It was pointed out to me that
btf_verifier_log_typecan also print several more lines per mismatched module. ~5 lines of logging for each mismatched module can start to add up, in terms of both overhead and the noise added to the kernel logs.This is more subjective but I think the warnings also read as though this is a more serious failure that might prevent affected modules from working correctly; anecdotally, I've gotten multiple questions about them asking if something is broken. This can be a red herring for anyone unfamiliar with BTF who is reading the logs to debug unrelated issues. In the
CONFIG_MODULE_ALLOW_BTF_MISMATCH=ycase the flood of warnings seems out of proportion to the actual result (modules still load successfully, just without debug info) especially since the user has explicitly enabled a config saying they expect mismatches.
Quoted 2 lines
If there needs to be some logging in the "mismatch allowed" case, could an acceptable middle ground be to use
pr_warn_onceto send a
So it looks like pr_warn_ratelimited still produces a lot of warning. In this case, I guess pr_warn_once should be okay.
Quoted 8 lines
single message reporting that mismatches were detected & module BTF debug info might be unavailable? Alternatively, if we could opt out of module BTF loading then that would also avoid this issue, but that's already been proposed before ([1], [2]) so I thought working with the existing config option might be preferred.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/
Quoted 61 lines · Yonghong Song
Quoted 57 lines · Connor O'Brien
Quoted 16 lines · Yonghong Song
Quoted 12 lines · Connor O'Brien
Enabling
CONFIG_MODULE_ALLOW_BTF_MISMATCHis an indication that BTF mismatches are expected and module loading should proceed anyway. Logging withpr_warn()on every one of these "benign" mismatches creates unnecessary noise when many such modules are loaded. Instead, limit logging to the case where a BTF mismatch actually prevents a module form loading.Signed-off-by Connor O'Brien <[email protected]>--- kernel/bpf/btf.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)Quoted 15 lines
kernel/bpf/btf.c static int btf_module_notify(struct notifier_block *nb, unsigned long op, · line 7194
Quoted 10 lines
kernel/bpf/btf.c +0
-2@@ -7190,11 +7190,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op, } btf =btf_parse_module (mod->name, mod->btf_data, mod->btf_data_size);if (IS_ERR (btf)) { -pr_warn ("failed to validate module [%s] BTF: %ld\n" , - mod->name,PTR_ERR (btf));I think such warning still useful even with
CONFIG_MODULE_ALLOW_BTF_MISMATCH.
Can we usepr_warn_ratelimitedinstead ofpr_warnin the above to avoid excessive warnings?I gave this a try on a Pixel 6 but I'm not sure it quite addresses the issue. The amount of logging doesn't seem to decrease much, I think because the interval between loading one mismatched module and the next can be greater than the default rate limit. To my mind, the issue is the total volume of these messages more so than their rate.
For context, Android devices using the GKI may load hundreds of separately-built modules, and BTF mismatches are possible for any/all of these. It was pointed out to me that
btf_verifier_log_typecan also print several more lines per mismatched module. ~5 lines of logging for each mismatched module can start to add up, in terms of both overhead and the noise added to the kernel logs.This is more subjective but I think the warnings also read as though this is a more serious failure that might prevent affected modules from working correctly; anecdotally, I've gotten multiple questions about them asking if something is broken. This can be a red herring for anyone unfamiliar with BTF who is reading the logs to debug unrelated issues. In the
CONFIG_MODULE_ALLOW_BTF_MISMATCH=ycase the flood of warnings seems out of proportion to the actual result (modules still load successfully, just without debug info) especially since the user has explicitly enabled a config saying they expect mismatches.
Quoted 5 lines
Quoted 2 lines
If there needs to be some logging in the "mismatch allowed" case, could an acceptable middle ground be to use
pr_warn_onceto send aSo it looks like
pr_warn_ratelimitedstill produces a lot of warning. In this case, I guesspr_warn_onceshould be okay.
Maybe pr_warn_once generic "some kernel module BTF mismatched". And also warn per-module message with details if CONFIG_MODULE_ALLOW_BTF_MISMATCH is not set?
Quoted 9 lines
Quoted 9 lines
single message reporting that mismatches were detected & module BTF debug info might be unavailable? Alternatively, if we could opt out of module BTF loading then that would also avoid this issue, but that's already been proposed before ([1], [2]) so I thought working with the existing config option might be preferred.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/
Quoted 66 lines · Andrii Nakryiko
Quoted 64 lines · Yonghong Song
Quoted 59 lines · Connor O'Brien
Quoted 16 lines · Yonghong Song
Quoted 12 lines · Connor O'Brien
Enabling
CONFIG_MODULE_ALLOW_BTF_MISMATCHis an indication that BTF mismatches are expected and module loading should proceed anyway. Logging withpr_warn()on every one of these "benign" mismatches creates unnecessary noise when many such modules are loaded. Instead, limit logging to the case where a BTF mismatch actually prevents a module form loading.Signed-off-by Connor O'Brien <[email protected]>--- kernel/bpf/btf.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)Quoted 15 lines
kernel/bpf/btf.c static int btf_module_notify(struct notifier_block *nb, unsigned long op, · line 7194
Quoted 10 lines
kernel/bpf/btf.c +0
-2@@ -7190,11 +7190,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op, } btf =btf_parse_module (mod->name, mod->btf_data, mod->btf_data_size);if (IS_ERR (btf)) { -pr_warn ("failed to validate module [%s] BTF: %ld\n" , - mod->name,PTR_ERR (btf));I think such warning still useful even with
CONFIG_MODULE_ALLOW_BTF_MISMATCH.
Can we usepr_warn_ratelimitedinstead ofpr_warnin the above to avoid excessive warnings?I gave this a try on a Pixel 6 but I'm not sure it quite addresses the issue. The amount of logging doesn't seem to decrease much, I think because the interval between loading one mismatched module and the next can be greater than the default rate limit. To my mind, the issue is the total volume of these messages more so than their rate.
For context, Android devices using the GKI may load hundreds of separately-built modules, and BTF mismatches are possible for any/all of these. It was pointed out to me that
btf_verifier_log_typecan also print several more lines per mismatched module. ~5 lines of logging for each mismatched module can start to add up, in terms of both overhead and the noise added to the kernel logs.This is more subjective but I think the warnings also read as though this is a more serious failure that might prevent affected modules from working correctly; anecdotally, I've gotten multiple questions about them asking if something is broken. This can be a red herring for anyone unfamiliar with BTF who is reading the logs to debug unrelated issues. In the
CONFIG_MODULE_ALLOW_BTF_MISMATCH=ycase the flood of warnings seems out of proportion to the actual result (modules still load successfully, just without debug info) especially since the user has explicitly enabled a config saying they expect mismatches.If there needs to be some logging in the "mismatch allowed" case, could an acceptable middle ground be to use
pr_warn_onceto send a
Quoted 7 lines
Quoted 2 lines
So it looks like
pr_warn_ratelimitedstill produces a lot of warning. In this case, I guesspr_warn_onceshould be okay.Maybe
pr_warn_oncegeneric "some kernel module BTF mismatched". And also warn per-module message with details ifCONFIG_MODULE_ALLOW_BTF_MISMATCHis not set?
Yeah, I figured the logging behavior should remain unchanged when CONFIG_MODULE_ALLOW_BTF_MISMATCH is unset, the existing verbose logging makes sense when every mismatch stops a module from loading.
Then for the pr_warn_once case something like "Kernel module BTF mismatch detected, BTF debug info may be unavailable for some modules"
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
single message reporting that mismatches were detected & module BTF debug info might be unavailable? Alternatively, if we could opt out of module BTF loading then that would also avoid this issue, but that's already been proposed before ([1], [2]) so I thought working with the existing config option might be preferred.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/
Quoted 77 lines · Connor O'Brien
Quoted 66 lines · Andrii Nakryiko
Quoted 64 lines · Yonghong Song
Quoted 59 lines · Connor O'Brien
Quoted 16 lines · Yonghong Song
Quoted 12 lines · Connor O'Brien
Enabling
CONFIG_MODULE_ALLOW_BTF_MISMATCHis an indication that BTF mismatches are expected and module loading should proceed anyway. Logging withpr_warn()on every one of these "benign" mismatches creates unnecessary noise when many such modules are loaded. Instead, limit logging to the case where a BTF mismatch actually prevents a module form loading.Signed-off-by Connor O'Brien <[email protected]>--- kernel/bpf/btf.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)Quoted 15 lines
kernel/bpf/btf.c static int btf_module_notify(struct notifier_block *nb, unsigned long op, · line 7194
Quoted 10 lines
kernel/bpf/btf.c +0
-2@@ -7190,11 +7190,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op, } btf =btf_parse_module (mod->name, mod->btf_data, mod->btf_data_size);if (IS_ERR (btf)) { -pr_warn ("failed to validate module [%s] BTF: %ld\n" , - mod->name,PTR_ERR (btf));I think such warning still useful even with
CONFIG_MODULE_ALLOW_BTF_MISMATCH.
Can we usepr_warn_ratelimitedinstead ofpr_warnin the above to avoid excessive warnings?I gave this a try on a Pixel 6 but I'm not sure it quite addresses the issue. The amount of logging doesn't seem to decrease much, I think because the interval between loading one mismatched module and the next can be greater than the default rate limit. To my mind, the issue is the total volume of these messages more so than their rate.
For context, Android devices using the GKI may load hundreds of separately-built modules, and BTF mismatches are possible for any/all of these. It was pointed out to me that
btf_verifier_log_typecan also print several more lines per mismatched module. ~5 lines of logging for each mismatched module can start to add up, in terms of both overhead and the noise added to the kernel logs.This is more subjective but I think the warnings also read as though this is a more serious failure that might prevent affected modules from working correctly; anecdotally, I've gotten multiple questions about them asking if something is broken. This can be a red herring for anyone unfamiliar with BTF who is reading the logs to debug unrelated issues. In the
CONFIG_MODULE_ALLOW_BTF_MISMATCH=ycase the flood of warnings seems out of proportion to the actual result (modules still load successfully, just without debug info) especially since the user has explicitly enabled a config saying they expect mismatches.If there needs to be some logging in the "mismatch allowed" case, could an acceptable middle ground be to use
pr_warn_onceto send aQuoted 7 lines
Quoted 2 lines
So it looks like
pr_warn_ratelimitedstill produces a lot of warning. In this case, I guesspr_warn_onceshould be okay.Maybe
pr_warn_oncegeneric "some kernel module BTF mismatched". And also warn per-module message with details ifCONFIG_MODULE_ALLOW_BTF_MISMATCHis not set?
Quoted 6 lines
Yeah, I figured the logging behavior should remain unchanged when
CONFIG_MODULE_ALLOW_BTF_MISMATCHis unset, the existing verbose logging makes sense when every mismatch stops a module from loading.Then for the
pr_warn_oncecase something like "Kernel module BTF mismatch detected, BTF debug info may be unavailable for some modules"
yep, makes sense to me
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
single message reporting that mismatches were detected & module BTF debug info might be unavailable? Alternatively, if we could opt out of module BTF loading then that would also avoid this issue, but that's already been proposed before ([1], [2]) so I thought working with the existing config option might be preferred.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/