bpf: Release module BTF IDR before module unload
Kumar Kartikeya Dwivedi
maintainerPATCH 1/1 bpf: Release module BTF IDR before module unload
Gregory reported in [0] that the global_map_resize test when run in repeatedly ends up failing during program load. This stems from the fact that BTF reference has not dropped to zero after the previous run's module is unloaded, and the older module's BTF is still discoverable and visible. Later, in libbpf, load_module_btfs() will find the ID for this stale BTF, open its fd, and then it will be used during program load where later steps taking module reference using btf_try_get_module() fail since the underlying module for the BTF is gone.
Logically, once a module is unloaded, it's associated BTF artifacts should become hidden. The BTF object inside the kernel may still remain alive as long its reference counts are alive, but it should no longer be discoverable.
To fix this, let us call btf_free_id() from the MODULE_STATE_GOING case for the module unload to free the BTF associated IDR entry, and disable its discovery once module unload returns to user space. If a race happens during unload, the outcome is non-deterministic anyway. However, user space should be able to rely on the guarantee that once it has synchronously established a successful module unload, no more stale artifacts associated with this module can be obtained subsequently.
Note that we must be careful to not invoke btf_free_id() in btf_put() when btf_is_module() is true now. There could be a window where the module unload drops a non-terminal reference, frees the IDR, but the same ID gets reused and the second unconditional btf_free_id() ends up releasing an unrelated entry.
We can be sure that by the time we reach final btf_put() for btf_is_module() case, the btf_free_id() is already done, since the module itself holds the BTF reference, and it will call this function for the BTF before dropping its own reference.
[0]: https://lore.kernel.org/bpf/[email protected]
--- kernel/bpf/btf.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)
kernel/bpf/btf.c +14 -1
@@ -1883,10 +1883,17 @@ void btf_get(struct btf *btf)refcount_inc (&btf->refcnt); } +bool btf_is_module (const struct btf *btf); +void btf_put (struct btf *btf) {if (btf &&refcount_dec_and_test (&btf->refcnt)) { -btf_free_id (btf); +/* +* IDR for module BTFs is freed when module's BTF reference is +* dropped, which _will_ happen before the final put. +*/ +if (!btf_is_module (btf)) +btf_free_id (btf);call_rcu (&btf->rcu, btf_free_rcu); } }
@@ -8382,6 +8389,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,if (btf_mod->module != module)continue ; +/* +* For modules, we do the freeing of BTF IDR as soon as +* module goes away to disable BTF discovery, since the +* btf_try_get_module() on such BTFs will fail. +*/ +btf_free_id (btf_mod->btf);list_del (&btf_mod->list);if (btf_mod->sysfs_attr)sysfs_remove_bin_file (btf_kobj, btf_mod->sysfs_attr);
--
2.52.0
Quoted 33 lines · Kumar Kartikeya Dwivedi
Gregory reported in [0] that the
global_map_resizetest when run in repeatedly ends up failing during program load. This stems from the fact that BTF reference has not dropped to zero after the previous run's module is unloaded, and the older module's BTF is still discoverable and visible. Later, in libbpf,load_module_btfs()will find the ID for this stale BTF, open its fd, and then it will be used during program load where later steps taking module reference usingbtf_try_get_module()fail since the underlying module for the BTF is gone.Logically, once a module is unloaded, it's associated BTF artifacts should become hidden. The BTF object inside the kernel may still remain alive as long its reference counts are alive, but it should no longer be discoverable.
To fix this, let us call
btf_free_id()from theMODULE_STATE_GOINGcase for the module unload to free the BTF associated IDR entry, and disable its discovery once module unload returns to user space. If a race happens during unload, the outcome is non-deterministic anyway. However, user space should be able to rely on the guarantee that once it has synchronously established a successful module unload, no more stale artifacts associated with this module can be obtained subsequently.Note that we must be careful to not invoke
btf_free_id()inbtf_put()whenbtf_is_module()is true now. There could be a window where the module unload drops a non-terminal reference, frees the IDR, but the same ID gets reused and the second unconditionalbtf_free_id()ends up releasing an unrelated entry.We can be sure that by the time we reach final
btf_put()forbtf_is_module()case, thebtf_free_id()is already done, since the module itself holds the BTF reference, and it will call this function for the BTF before dropping its own reference.
Quoted 7 lines
[0]: https://lore.kernel.org/bpf/[email protected]
Fixes 36e68442d1af ("bpf: Load and verify kernel module BTFs")Suggested-by Martin KaFai Lau <[email protected]>Reported-by Gregory Bell <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>---
kernel/bpf/btf.c void btf_get(struct btf *btf) · line 1886
Quoted 12 lines
kernel/bpf/btf.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)kernel/bpf/btf.c +1
-0@@ -1883,10 +1883,17 @@ void btf_get(struct btf *btf)refcount_inc (&btf->refcnt); } +bool btf_is_module (const struct btf *btf);
Nit: Maybe just move the entire btf_is_module function to here? It's a one-liner anyway.
Quoted 29 lines
kernel/bpf/btf.c +13
-2+void btf_put (struct btf *btf) {if (btf &&refcount_dec_and_test (&btf->refcnt)) { -btf_free_id (btf); +/* +* IDR for module BTFs is freed when module's BTF reference is +* dropped, which _will_ happen before the final put. +*/ +if (!btf_is_module (btf)) +btf_free_id (btf);call_rcu (&btf->rcu, btf_free_rcu); } }@@ -8382,6 +8389,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,if (btf_mod->module != module)continue ; +/* +* For modules, we do the freeing of BTF IDR as soon as +* module goes away to disable BTF discovery, since the +* btf_try_get_module() on such BTFs will fail. +*/ +btf_free_id (btf_mod->btf);list_del (&btf_mod->list);if (btf_mod->sysfs_attr)sysfs_remove_bin_file (btf_kobj, btf_mod->sysfs_attr); --2.52.0
Quoted 23 lines · Kumar Kartikeya Dwivedi
Gregory reported in [0] that the
global_map_resizetest when run in repeatedly ends up failing during program load. This stems from the fact that BTF reference has not dropped to zero after the previous run's module is unloaded, and the older module's BTF is still discoverable and visible. Later, in libbpf,load_module_btfs()will find the ID for this stale BTF, open its fd, and then it will be used during program load where later steps taking module reference usingbtf_try_get_module()fail since the underlying module for the BTF is gone.Logically, once a module is unloaded, it's associated BTF artifacts should become hidden. The BTF object inside the kernel may still remain alive as long its reference counts are alive, but it should no longer be discoverable.
To fix this, let us call
btf_free_id()from theMODULE_STATE_GOINGcase for the module unload to free the BTF associated IDR entry, and disable its discovery once module unload returns to user space. If a race happens during unload, the outcome is non-deterministic anyway. However, user space should be able to rely on the guarantee that once it has synchronously established a successful module unload, no more stale artifacts associated with this module can be obtained subsequently.
Quoted 5 lines
Note that we must be careful to not invoke
btf_free_id()inbtf_put()whenbtf_is_module()is true now. There could be a window where the module unload drops a non-terminal reference, frees the IDR, but the same ID gets reused and the second unconditionalbtf_free_id()ends up releasing an unrelated entry.
what if we just check module_is_live() state for module BTFs during GET_FD_BY_ID() operation and return -ENOENT? We already have refcount_inc_not_zero(&btf->refcnt), it's just another "it's still there, but not valid to be taken ahold by user space" condition
Quoted 55 lines
We can be sure that by the time we reach final
btf_put()forbtf_is_module()case, thebtf_free_id()is already done, since the module itself holds the BTF reference, and it will call this function for the BTF before dropping its own reference.[0]: https://lore.kernel.org/bpf/[email protected]
Fixes 36e68442d1af ("bpf: Load and verify kernel module BTFs")Suggested-by Martin KaFai Lau <[email protected]>Reported-by Gregory Bell <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/btf.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)kernel/bpf/btf.c +14
-2@@ -1883,10 +1883,17 @@ void btf_get(struct btf *btf)refcount_inc (&btf->refcnt); } +bool btf_is_module (const struct btf *btf); +void btf_put (struct btf *btf) {if (btf &&refcount_dec_and_test (&btf->refcnt)) { -btf_free_id (btf); +/* +* IDR for module BTFs is freed when module's BTF reference is +* dropped, which _will_ happen before the final put. +*/ +if (!btf_is_module (btf)) +btf_free_id (btf);call_rcu (&btf->rcu, btf_free_rcu); } }@@ -8382,6 +8389,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,if (btf_mod->module != module)continue ; +/* +* For modules, we do the freeing of BTF IDR as soon as +* module goes away to disable BTF discovery, since the +* btf_try_get_module() on such BTFs will fail. +*/ +btf_free_id (btf_mod->btf);list_del (&btf_mod->list);if (btf_mod->sysfs_attr)sysfs_remove_bin_file (btf_kobj, btf_mod->sysfs_attr); --2.52.0
Kumar Kartikeya Dwivedi
maintainerQuoted 32 lines · Andrii Nakryiko
Quoted 23 lines · Kumar Kartikeya Dwivedi
Gregory reported in [0] that the
global_map_resizetest when run in repeatedly ends up failing during program load. This stems from the fact that BTF reference has not dropped to zero after the previous run's module is unloaded, and the older module's BTF is still discoverable and visible. Later, in libbpf,load_module_btfs()will find the ID for this stale BTF, open its fd, and then it will be used during program load where later steps taking module reference usingbtf_try_get_module()fail since the underlying module for the BTF is gone.Logically, once a module is unloaded, it's associated BTF artifacts should become hidden. The BTF object inside the kernel may still remain alive as long its reference counts are alive, but it should no longer be discoverable.
To fix this, let us call
btf_free_id()from theMODULE_STATE_GOINGcase for the module unload to free the BTF associated IDR entry, and disable its discovery once module unload returns to user space. If a race happens during unload, the outcome is non-deterministic anyway. However, user space should be able to rely on the guarantee that once it has synchronously established a successful module unload, no more stale artifacts associated with this module can be obtained subsequently.Quoted 5 lines
Note that we must be careful to not invoke
btf_free_id()inbtf_put()whenbtf_is_module()is true now. There could be a window where the module unload drops a non-terminal reference, frees the IDR, but the same ID gets reused and the second unconditionalbtf_free_id()ends up releasing an unrelated entry.
Quoted 4 lines
what if we just check
module_is_live()state for module BTFs duringGET_FD_BY_ID()operation and return-ENOENT? We already haverefcount_inc_not_zero(&btf->refcnt), it's just another "it's still there, but not valid to be taken ahold by user space" condition
We will need the module pointer to be able to do module_is_live(), but in this case the module is already gone.
I was achieving the same thing in my first proposal, I didn't share the diff, but TL;DR was setting a bool in btf->module_unloaded and checking for it after refcount_inc_not_zero.
Set btf->module_unloaded from MODULE_STATE_GOING.
I don't have much of a preference for the above or the current diff.
Quoted 56 lines
Quoted 56 lines
We can be sure that by the time we reach final
btf_put()forbtf_is_module()case, thebtf_free_id()is already done, since the module itself holds the BTF reference, and it will call this function for the BTF before dropping its own reference.[0]: https://lore.kernel.org/bpf/[email protected]
Fixes 36e68442d1af ("bpf: Load and verify kernel module BTFs")Suggested-by Martin KaFai Lau <[email protected]>Reported-by Gregory Bell <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/btf.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)kernel/bpf/btf.c +14
-2@@ -1883,10 +1883,17 @@ void btf_get(struct btf *btf)refcount_inc (&btf->refcnt); } +bool btf_is_module (const struct btf *btf); +void btf_put (struct btf *btf) {if (btf &&refcount_dec_and_test (&btf->refcnt)) { -btf_free_id (btf); +/* +* IDR for module BTFs is freed when module's BTF reference is +* dropped, which _will_ happen before the final put. +*/ +if (!btf_is_module (btf)) +btf_free_id (btf);call_rcu (&btf->rcu, btf_free_rcu); } }@@ -8382,6 +8389,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,if (btf_mod->module != module)continue ; +/* +* For modules, we do the freeing of BTF IDR as soon as +* module goes away to disable BTF discovery, since the +* btf_try_get_module() on such BTFs will fail. +*/ +btf_free_id (btf_mod->btf);list_del (&btf_mod->list);if (btf_mod->sysfs_attr)sysfs_remove_bin_file (btf_kobj, btf_mod->sysfs_attr); --2.52.0
Quoted 39 lines · Kumar Kartikeya Dwivedi
Quoted 32 lines · Andrii Nakryiko
Quoted 23 lines · Kumar Kartikeya Dwivedi
Gregory reported in [0] that the
global_map_resizetest when run in repeatedly ends up failing during program load. This stems from the fact that BTF reference has not dropped to zero after the previous run's module is unloaded, and the older module's BTF is still discoverable and visible. Later, in libbpf,load_module_btfs()will find the ID for this stale BTF, open its fd, and then it will be used during program load where later steps taking module reference usingbtf_try_get_module()fail since the underlying module for the BTF is gone.Logically, once a module is unloaded, it's associated BTF artifacts should become hidden. The BTF object inside the kernel may still remain alive as long its reference counts are alive, but it should no longer be discoverable.
To fix this, let us call
btf_free_id()from theMODULE_STATE_GOINGcase for the module unload to free the BTF associated IDR entry, and disable its discovery once module unload returns to user space. If a race happens during unload, the outcome is non-deterministic anyway. However, user space should be able to rely on the guarantee that once it has synchronously established a successful module unload, no more stale artifacts associated with this module can be obtained subsequently.Quoted 5 lines
Note that we must be careful to not invoke
btf_free_id()inbtf_put()whenbtf_is_module()is true now. There could be a window where the module unload drops a non-terminal reference, frees the IDR, but the same ID gets reused and the second unconditionalbtf_free_id()ends up releasing an unrelated entry.Quoted 4 lines
what if we just check
module_is_live()state for module BTFs duringGET_FD_BY_ID()operation and return-ENOENT? We already haverefcount_inc_not_zero(&btf->refcnt), it's just another "it's still there, but not valid to be taken ahold by user space" condition
Quoted 8 lines
We will need the module pointer to be able to do
module_is_live(), but in this case the module is already gone.
I was achieving the same thing in my first proposal, I didn't share the diff, but TL;DR was setting a bool inbtf->module_unloadedand checking for it afterrefcount_inc_not_zero.
Setbtf->module_unloadedfromMODULE_STATE_GOING.I don't have much of a preference for the above or the current diff.
The part I don't particularly like is that we have two different cases when ID is removed, based on whether BTF is module BTF or not. What if we add btf->id = 0; after idr_remove within spinlock area in btf_free_id() and then call btf_Free_id() unconditionally both on module unload and in normal btf_put path?
Quoted 57 lines
Quoted 57 lines
Quoted 57 lines
We can be sure that by the time we reach final
btf_put()forbtf_is_module()case, thebtf_free_id()is already done, since the module itself holds the BTF reference, and it will call this function for the BTF before dropping its own reference.[0]: https://lore.kernel.org/bpf/[email protected]
Fixes 36e68442d1af ("bpf: Load and verify kernel module BTFs")Suggested-by Martin KaFai Lau <[email protected]>Reported-by Gregory Bell <[email protected]>Signed-off-by Kumar Kartikeya Dwivedi <[email protected]>--- kernel/bpf/btf.c | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-)kernel/bpf/btf.c +14
-2@@ -1883,10 +1883,17 @@ void btf_get(struct btf *btf)refcount_inc (&btf->refcnt); } +bool btf_is_module (const struct btf *btf); +void btf_put (struct btf *btf) {if (btf &&refcount_dec_and_test (&btf->refcnt)) { -btf_free_id (btf); +/* +* IDR for module BTFs is freed when module's BTF reference is +* dropped, which _will_ happen before the final put. +*/ +if (!btf_is_module (btf)) +btf_free_id (btf);call_rcu (&btf->rcu, btf_free_rcu); } }@@ -8382,6 +8389,12 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,if (btf_mod->module != module)continue ; +/* +* For modules, we do the freeing of BTF IDR as soon as +* module goes away to disable BTF discovery, since the +* btf_try_get_module() on such BTFs will fail. +*/ +btf_free_id (btf_mod->btf);list_del (&btf_mod->list);if (btf_mod->sysfs_attr)sysfs_remove_bin_file (btf_kobj, btf_mod->sysfs_attr); --2.52.0
Kumar Kartikeya Dwivedi
maintainerQuoted 49 lines · Andrii Nakryiko
Quoted 46 lines · Kumar Kartikeya Dwivedi
Quoted 32 lines · Andrii Nakryiko
Quoted 23 lines · Kumar Kartikeya Dwivedi
Gregory reported in [0] that the
global_map_resizetest when run in repeatedly ends up failing during program load. This stems from the fact that BTF reference has not dropped to zero after the previous run's module is unloaded, and the older module's BTF is still discoverable and visible. Later, in libbpf,load_module_btfs()will find the ID for this stale BTF, open its fd, and then it will be used during program load where later steps taking module reference usingbtf_try_get_module()fail since the underlying module for the BTF is gone.Logically, once a module is unloaded, it's associated BTF artifacts should become hidden. The BTF object inside the kernel may still remain alive as long its reference counts are alive, but it should no longer be discoverable.
To fix this, let us call
btf_free_id()from theMODULE_STATE_GOINGcase for the module unload to free the BTF associated IDR entry, and disable its discovery once module unload returns to user space. If a race happens during unload, the outcome is non-deterministic anyway. However, user space should be able to rely on the guarantee that once it has synchronously established a successful module unload, no more stale artifacts associated with this module can be obtained subsequently.Quoted 5 lines
Note that we must be careful to not invoke
btf_free_id()inbtf_put()whenbtf_is_module()is true now. There could be a window where the module unload drops a non-terminal reference, frees the IDR, but the same ID gets reused and the second unconditionalbtf_free_id()ends up releasing an unrelated entry.Quoted 4 lines
what if we just check
module_is_live()state for module BTFs duringGET_FD_BY_ID()operation and return-ENOENT? We already haverefcount_inc_not_zero(&btf->refcnt), it's just another "it's still there, but not valid to be taken ahold by user space" conditionWe will need the module pointer to be able to do
module_is_live(), but in this case the module is already gone.
I was achieving the same thing in my first proposal, I didn't share the diff, but TL;DR was setting a bool inbtf->module_unloadedand checking for it afterrefcount_inc_not_zero.
Setbtf->module_unloadedfromMODULE_STATE_GOING.
Quoted 8 lines
Quoted 1 line
I don't have much of a preference for the above or the current diff.
The part I don't particularly like is that we have two different cases when ID is removed, based on whether BTF is module BTF or not. What if we add
btf->id = 0;afteridr_removewithin spinlock area inbtf_free_id()and then callbtf_Free_id()unconditionally both on module unload and in normalbtf_putpath?
Yeah, that should work too. id = 0 should make the function idempotent since 0 is not a valid IDR. Let me give it a spin and resend.
Quoted 1 line
Quoted 1 line
[...]