libbpf: count present CPUs, not theoretically possible
Subsystems (4)
PATCH 1/1 libbpf: count present CPUs, not theoretically possible
This patch switches libbpf_num_possible_cpus() from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing of PERF_EVENT_ARRAY map on HOTPLUG-enabled systems.
On HOTPLUG enabled systems, /sys/devices/system/cpu/possible is going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present). /sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.
On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.
--- tools/lib/bpf/libbpf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +1 -1
@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)int libbpf_num_possible_cpus (void ) { -static const char *fcpu ="/sys/devices/system/cpu/possible" ; +static const char *fcpu ="/sys/devices/system/cpu/present" ;int len =0 , n =0 , il =0 , ir =0 ;unsigned int start =0 , end =0 ;int tmp_cpus =0 ;
Signature · 1 line
2.17.1
Quoted 13 lines · Andrii Nakryiko
This patch switches
libbpf_num_possible_cpus()from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing ofPERF_EVENT_ARRAYmap on HOTPLUG-enabled systems.On HOTPLUG enabled systems,
/sys/devices/system/cpu/possibleis going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present)./sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.
Quoted 3 lines
On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.
Just curious - is there a reason for not adding a new libbpf_num_present_cpus() function to cover this case, and switching to using that in various places?
Looking at the places libbpf_num_possible_cpus() is called in libbpf
- __perf_buffer__new(): this could just change to use the number of present CPUs, since perf_buffer__new_raw() with a cpu_cnt in struct perf_buffer_raw_ops
- bpf_object__create_maps(), which is called via bpf_oject__load_xattr(). In this case it seems like switching to num present makes sense, though it might make sense to add a field to struct bpf_object_load_attr * to allow users to explicitly set another max value.
This would give the desired default behaviour, while still giving users a way of specifying the possible number. What do you think? Thanks!
Alan
Quoted 22 lines
Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/libbpf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tools/lib/bpf/libbpf.c +1
-1@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)int libbpf_num_possible_cpus (void ) { -static const char *fcpu ="/sys/devices/system/cpu/possible" ; +static const char *fcpu ="/sys/devices/system/cpu/present" ;int len =0 , n =0 , il =0 , ir =0 ;unsigned int start =0 , end =0 ;int tmp_cpus =0 ;Signature · 1 line
2.17.1
Quoted 16 lines · Alan Maguire
Quoted 13 lines · Andrii Nakryiko
This patch switches
libbpf_num_possible_cpus()from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing ofPERF_EVENT_ARRAYmap on HOTPLUG-enabled systems.On HOTPLUG enabled systems,
/sys/devices/system/cpu/possibleis going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present)./sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.
Quoted 6 lines
Quoted 3 lines
On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.
Just curious - is there a reason for not adding a new
libbpf_num_present_cpus()function to cover this case, and switching to using that in various places?
The reason is that libbpf_num_possible_cpus() is useless on HOTPLUG systems and never worked as intended. If you rely on this function to create perf_buffer and/or PERF_EVENT_ARRAY, it will simply fail due to specifying more CPUs than are present. I didn't want to keep adding new APIs for no good reason, while also leaving useless ones, so I fixed the existing API to behave as expected. It's unfortunate that name doesn't match sysfs file we are reading it from, of course, but having people to choose between libbpf_num_possible_cpus() vs libbpf_num_present_cpus() seems like even bigger problem, as differences are non-obvious.
The good thing, it won't break all the non-HOTPLUG systems for sure, which seems to be the only cases that are used right now (otherwise someone would already complain about broken libbpf_num_possible_cpus()).
Quoted 11 lines
Looking at the places
libbpf_num_possible_cpus()is called in libbpf-
__perf_buffer__new(): this could just change to use the number of present CPUs, sinceperf_buffer__new_raw()with acpu_cntin structperf_buffer_raw_ops-
bpf_object__create_maps(), which is called viabpf_oject__load_xattr(). In this case it seems like switching to num present makes sense, though it might make sense to add a field to structbpf_object_load_attr* to allow users to explicitly set another max value.
I believe more knobs is not always better for API. Plus, adding any field to those xxx_xattr structs is an ABI breakage and requires adding new APIs, so I don't think this is good enough reason to add new flag. See discussion in another thread about this whole API design w/ current attributes and ABI consequences of adding anything new to them.
Quoted 3 lines
This would give the desired default behaviour, while still giving users a way of specifying the possible number. What do you think? Thanks!
BTW, if user wants to override the size of maps, they can do it easily either in map definition or programmatically after bpf_object__open, but before bpf_object__load, so there is no need for flags, it's all easily achievable with existing API.
Quoted 25 lines
Alan
Quoted 22 lines
Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/libbpf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tools/lib/bpf/libbpf.c +1
-2@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)int libbpf_num_possible_cpus (void ) { -static const char *fcpu ="/sys/devices/system/cpu/possible" ; +static const char *fcpu ="/sys/devices/system/cpu/present" ;int len =0 , n =0 , il =0 , ir =0 ;unsigned int start =0 , end =0 ;int tmp_cpus =0 ; --2.17.1
Alan Maguire
Reviewed-byQuoted 35 lines · Andrii Nakryiko
Quoted 16 lines · Alan Maguire
Quoted 13 lines · Andrii Nakryiko
This patch switches
libbpf_num_possible_cpus()from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing ofPERF_EVENT_ARRAYmap on HOTPLUG-enabled systems.On HOTPLUG enabled systems,
/sys/devices/system/cpu/possibleis going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present)./sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.Quoted 6 lines
Quoted 3 lines
On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.
Just curious - is there a reason for not adding a new
libbpf_num_present_cpus()function to cover this case, and switching to using that in various places?The reason is that
libbpf_num_possible_cpus()is useless on HOTPLUG systems and never worked as intended. If you rely on this function to createperf_bufferand/orPERF_EVENT_ARRAY, it will simply fail due to specifying more CPUs than are present. I didn't want to keep adding new APIs for no good reason, while also leaving useless ones, so I fixed the existing API to behave as expected. It's unfortunate that name doesn't match sysfs file we are reading it from, of course, but having people to choose betweenlibbpf_num_possible_cpus()vslibbpf_num_present_cpus()seems like even bigger problem, as differences are non-obvious.
Quoted 5 lines
The good thing, it won't break all the non-HOTPLUG systems for sure, which seems to be the only cases that are used right now (otherwise someone would already complain about broken
libbpf_num_possible_cpus()).
Understood, thanks for the explanation.
Quoted 20 lines
Quoted 11 lines
Looking at the places
libbpf_num_possible_cpus()is called in libbpf-
__perf_buffer__new(): this could just change to use the number of present CPUs, sinceperf_buffer__new_raw()with acpu_cntin structperf_buffer_raw_ops-
bpf_object__create_maps(), which is called viabpf_oject__load_xattr(). In this case it seems like switching to num present makes sense, though it might make sense to add a field to structbpf_object_load_attr* to allow users to explicitly set another max value.I believe more knobs is not always better for API. Plus, adding any field to those
xxx_xattrstructs is an ABI breakage and requires adding new APIs, so I don't think this is good enough reason to add new flag. See discussion in another thread about this whole API design w/ current attributes and ABI consequences of adding anything new to them.Quoted 1 line
Quoted 8 lines
Quoted 2 lines
This would give the desired default behaviour, while still giving users a way of specifying the possible number. What do you think? Thanks!
BTW, if user wants to override the size of maps, they can do it easily either in map definition or programmatically after
bpf_object__open, but beforebpf_object__load, so there is no need for flags, it's all easily achievable with existing API.
Ah, I missed that. Thanks for clarifying!
Quoted 26 lines
Quoted 25 lines
Alan
Quoted 22 lines
Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/libbpf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tools/lib/bpf/libbpf.c +1
-2@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)int libbpf_num_possible_cpus (void ) { -static const char *fcpu ="/sys/devices/system/cpu/possible" ; +static const char *fcpu ="/sys/devices/system/cpu/present" ;int len =0 , n =0 , il =0 , ir =0 ;unsigned int start =0 , end =0 ;int tmp_cpus =0 ; --2.17.1
Quoted 30 lines · Andrii Nakryiko
This patch switches
libbpf_num_possible_cpus()from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing ofPERF_EVENT_ARRAYmap on HOTPLUG-enabled systems.On HOTPLUG enabled systems,
/sys/devices/system/cpu/possibleis going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present)./sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.
Signed-off-by Andrii Nakryiko <[email protected]>---
tools/lib/bpf/libbpf.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)tools/lib/bpf/libbpf.c +0
-0
tools/lib/bpf/libbpf.c void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear) · line 5902
Quoted 5 lines
tools/lib/bpf/libbpf.c +1
-1@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)int libbpf_num_possible_cpus (void ) { -static const char *fcpu ="/sys/devices/system/cpu/possible" ; +static const char *fcpu ="/sys/devices/system/cpu/present" ;
This is _very_ confusing. "possible cpus", "present cpus", and "online cpus" are existing terminologies. I don't think we should force people to remember something like "By possible cpus, libbpf actually means present cpus".
This change works if we call it "libbbpf_num_cpus()". However, libbpf_num_possible_cpus(), should mean possible CPUs.
Thanks,
Song
Quoted 42 lines · Song Liu
Quoted 31 lines · Andrii Nakryiko
This patch switches
libbpf_num_possible_cpus()from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing ofPERF_EVENT_ARRAYmap on HOTPLUG-enabled systems.On HOTPLUG enabled systems,
/sys/devices/system/cpu/possibleis going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present)./sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.
Signed-off-by Andrii Nakryiko <[email protected]>---
tools/lib/bpf/libbpf.c| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)tools/lib/bpf/libbpf.c +0
-0tools/lib/bpf/libbpf.c void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear) · line 5902
Quoted 5 lines
tools/lib/bpf/libbpf.c +1
-1@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)int libbpf_num_possible_cpus (void ) { -static const char *fcpu ="/sys/devices/system/cpu/possible" ; +static const char *fcpu ="/sys/devices/system/cpu/present" ;This is _very_ confusing. "possible cpus", "present cpus", and "online cpus" are existing terminologies. I don't think we should force people to remember something like "By possible cpus, libbpf actually means present cpus".
Quoted 2 lines
This change works if we call it "
libbbpf_num_cpus()". However,libbpf_num_possible_cpus(), should mean possible CPUs.
Ok, then if we really need to (I'll play again with my VM to recall all the details of original problem with this that I had before), I'll just add libbpf_num_present_cpus().
Quoted 4 lines
Thanks,
Song
Daniel Borkmann
maintainerQuoted 3 lines · Andrii Nakryiko
This patch switches
libbpf_num_possible_cpus()from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing ofPERF_EVENT_ARRAYmap on HOTPLUG-enabled systems.
Those issues should be described in more detail here in the changelog, otherwise noone knows what is meant exactly when glancing at the git log.
Quoted 22 lines
On HOTPLUG enabled systems,
/sys/devices/system/cpu/possibleis going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present)./sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.
Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/libbpf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tools/lib/bpf/libbpf.c +0
-0
tools/lib/bpf/libbpf.c void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear) · line 5902
Quoted 5 lines
tools/lib/bpf/libbpf.c +1
-1@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)int libbpf_num_possible_cpus (void ) { -static const char *fcpu ="/sys/devices/system/cpu/possible" ; +static const char *fcpu ="/sys/devices/system/cpu/present" ;
Problem is that this is going to break things *badly* for per-cpu maps as BPF_DECLARE_PERCPU() relies on possible CPUs, not present ones. And given present<=possible you'll end up corrupting user space when you do a lookup on the map since kernel side operates on possible as well.
Quoted 4 lines
tools/lib/bpf/libbpf.c +0
-0int len =0 , n =0 , il =0 , ir =0 ;unsigned int start =0 , end =0 ;int tmp_cpus =0 ;
Quoted 8 lines · Daniel Borkmann
Quoted 3 lines · Andrii Nakryiko
This patch switches
libbpf_num_possible_cpus()from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing ofPERF_EVENT_ARRAYmap on HOTPLUG-enabled systems.Those issues should be described in more detail here in the changelog, otherwise noone knows what is meant exactly when glancing at the git log.
Sure, I can add more details.
Quoted 24 lines
Quoted 24 lines
On HOTPLUG enabled systems,
/sys/devices/system/cpu/possibleis going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present)./sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.
Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/libbpf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tools/lib/bpf/libbpf.c +0
-0@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)
Quoted 9 lines
tools/lib/bpf/libbpf.c void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear) · line 5902
Quoted 4 lines
tools/lib/bpf/libbpf.c +1
-1int libbpf_num_possible_cpus (void ) { -static const char *fcpu ="/sys/devices/system/cpu/possible" ; +static const char *fcpu ="/sys/devices/system/cpu/present" ;Problem is that this is going to break things *badly* for per-cpu maps as
BPF_DECLARE_PERCPU()relies on possible CPUs, not present ones. And given present<=possible you'll end up corrupting user space when you do a lookup on the map since kernel side operates on possible as well.
Yeah, you are right. Ok, let me go back to my VM and repro original issue I had and see what and why is causing that. I'll see maybe I don't need this fix at all.
Quoted 6 lines
Quoted 5 lines
tools/lib/bpf/libbpf.c +0
-0int len =0 , n =0 , il =0 , ir =0 ;unsigned int start =0 , end =0 ;int tmp_cpus =0 ;