Skip to contentSkip to conversation

libbpf: count present CPUs, not theoretically possible

PATCH In review Andrii Nakryiko 5 participants 8 messages Sep 28, 2019 – Sep 30, 2019

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 libbpf: count present CPUs, not theoretically possible

This patch switches libbpf_num_possible_cpus() from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing of PERF_EVENT_ARRAY map on HOTPLUG-enabled systems.

On HOTPLUG enabled systems, /sys/devices/system/cpu/possible is going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present). /sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.

On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.

---
 tools/lib/bpf/libbpf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +1 -1
index e0276520171b..45351c074e45 100644
@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)
 
 int libbpf_num_possible_cpus(void)
 {
-	static const char *fcpu = "/sys/devices/system/cpu/possible";
+	static const char *fcpu = "/sys/devices/system/cpu/present";
 	int len = 0, n = 0, il = 0, ir = 0;
 	unsigned int start = 0, end = 0;
 	int tmp_cpus = 0;
Signature · 1 line
2.17.1
#1 lore raw
Quoted 13 lines · Andrii Nakryiko

This patch switches libbpf_num_possible_cpus() from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing of PERF_EVENT_ARRAY map on HOTPLUG-enabled systems.

On HOTPLUG enabled systems, /sys/devices/system/cpu/possible is going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present). /sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.

Quoted 3 lines

On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.

Just curious - is there a reason for not adding a new libbpf_num_present_cpus() function to cover this case, and switching to using that in various places?

Looking at the places libbpf_num_possible_cpus() is called in libbpf

- __perf_buffer__new(): this could just change to use the number of present CPUs, since perf_buffer__new_raw() with a cpu_cnt in struct perf_buffer_raw_ops

- bpf_object__create_maps(), which is called via bpf_oject__load_xattr(). In this case it seems like switching to num present makes sense, though it might make sense to add a field to struct bpf_object_load_attr * to allow users to explicitly set another max value.

This would give the desired default behaviour, while still giving users a way of specifying the possible number. What do you think? Thanks!

Alan

Quoted 22 lines
---
 tools/lib/bpf/libbpf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +1 -1
index e0276520171b..45351c074e45 100644
@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)
 
 int libbpf_num_possible_cpus(void)
 {
-	static const char *fcpu = "/sys/devices/system/cpu/possible";
+	static const char *fcpu = "/sys/devices/system/cpu/present";
 	int len = 0, n = 0, il = 0, ir = 0;
 	unsigned int start = 0, end = 0;
 	int tmp_cpus = 0;
Signature · 1 line
2.17.1
#2 lore raw
Quoted 16 lines · Alan Maguire
Quoted 13 lines · Andrii Nakryiko

This patch switches libbpf_num_possible_cpus() from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing of PERF_EVENT_ARRAY map on HOTPLUG-enabled systems.

On HOTPLUG enabled systems, /sys/devices/system/cpu/possible is going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present). /sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.

Quoted 6 lines
Quoted 3 lines

On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.

Just curious - is there a reason for not adding a new libbpf_num_present_cpus() function to cover this case, and switching to using that in various places?

The reason is that libbpf_num_possible_cpus() is useless on HOTPLUG systems and never worked as intended. If you rely on this function to create perf_buffer and/or PERF_EVENT_ARRAY, it will simply fail due to specifying more CPUs than are present. I didn't want to keep adding new APIs for no good reason, while also leaving useless ones, so I fixed the existing API to behave as expected. It's unfortunate that name doesn't match sysfs file we are reading it from, of course, but having people to choose between libbpf_num_possible_cpus() vs libbpf_num_present_cpus() seems like even bigger problem, as differences are non-obvious.

The good thing, it won't break all the non-HOTPLUG systems for sure, which seems to be the only cases that are used right now (otherwise someone would already complain about broken libbpf_num_possible_cpus()).

Quoted 11 lines

Looking at the places libbpf_num_possible_cpus() is called in libbpf

- __perf_buffer__new(): this could just change to use the number of present CPUs, since perf_buffer__new_raw() with a cpu_cnt in struct perf_buffer_raw_ops

- bpf_object__create_maps(), which is called via bpf_oject__load_xattr(). In this case it seems like switching to num present makes sense, though it might make sense to add a field to struct bpf_object_load_attr * to allow users to explicitly set another max value.

I believe more knobs is not always better for API. Plus, adding any field to those xxx_xattr structs is an ABI breakage and requires adding new APIs, so I don't think this is good enough reason to add new flag. See discussion in another thread about this whole API design w/ current attributes and ABI consequences of adding anything new to them.

Quoted 3 lines

This would give the desired default behaviour, while still giving users a way of specifying the possible number. What do you think? Thanks!

BTW, if user wants to override the size of maps, they can do it easily either in map definition or programmatically after bpf_object__open, but before bpf_object__load, so there is no need for flags, it's all easily achievable with existing API.

Quoted 25 lines

Alan

Quoted 22 lines
---
 tools/lib/bpf/libbpf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +1 -2
index e0276520171b..45351c074e45 100644
@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)
 
 int libbpf_num_possible_cpus(void)
 {
-     static const char *fcpu = "/sys/devices/system/cpu/possible";
+     static const char *fcpu = "/sys/devices/system/cpu/present";
      int len = 0, n = 0, il = 0, ir = 0;
      unsigned int start = 0, end = 0;
      int tmp_cpus = 0;
--

2.17.1

Reviewed-by
#3 lore raw
Quoted 35 lines · Andrii Nakryiko
Quoted 16 lines · Alan Maguire
Quoted 13 lines · Andrii Nakryiko

This patch switches libbpf_num_possible_cpus() from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing of PERF_EVENT_ARRAY map on HOTPLUG-enabled systems.

On HOTPLUG enabled systems, /sys/devices/system/cpu/possible is going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present). /sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.

Quoted 6 lines
Quoted 3 lines

On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.

Just curious - is there a reason for not adding a new libbpf_num_present_cpus() function to cover this case, and switching to using that in various places?

The reason is that libbpf_num_possible_cpus() is useless on HOTPLUG systems and never worked as intended. If you rely on this function to create perf_buffer and/or PERF_EVENT_ARRAY, it will simply fail due to specifying more CPUs than are present. I didn't want to keep adding new APIs for no good reason, while also leaving useless ones, so I fixed the existing API to behave as expected. It's unfortunate that name doesn't match sysfs file we are reading it from, of course, but having people to choose between libbpf_num_possible_cpus() vs libbpf_num_present_cpus() seems like even bigger problem, as differences are non-obvious.

Quoted 5 lines

The good thing, it won't break all the non-HOTPLUG systems for sure, which seems to be the only cases that are used right now (otherwise someone would already complain about broken libbpf_num_possible_cpus()).

Understood, thanks for the explanation.

Quoted 20 lines
Quoted 11 lines

Looking at the places libbpf_num_possible_cpus() is called in libbpf

- __perf_buffer__new(): this could just change to use the number of present CPUs, since perf_buffer__new_raw() with a cpu_cnt in struct perf_buffer_raw_ops

- bpf_object__create_maps(), which is called via bpf_oject__load_xattr(). In this case it seems like switching to num present makes sense, though it might make sense to add a field to struct bpf_object_load_attr * to allow users to explicitly set another max value.

I believe more knobs is not always better for API. Plus, adding any field to those xxx_xattr structs is an ABI breakage and requires adding new APIs, so I don't think this is good enough reason to add new flag. See discussion in another thread about this whole API design w/ current attributes and ABI consequences of adding anything new to them.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

This would give the desired default behaviour, while still giving users a way of specifying the possible number. What do you think? Thanks!

BTW, if user wants to override the size of maps, they can do it easily either in map definition or programmatically after bpf_object__open, but before bpf_object__load, so there is no need for flags, it's all easily achievable with existing API.

Ah, I missed that. Thanks for clarifying!

Quoted 26 lines
Quoted 25 lines

Alan

Quoted 22 lines
---
 tools/lib/bpf/libbpf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +1 -2
index e0276520171b..45351c074e45 100644
@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)
 
 int libbpf_num_possible_cpus(void)
 {
-     static const char *fcpu = "/sys/devices/system/cpu/possible";
+     static const char *fcpu = "/sys/devices/system/cpu/present";
      int len = 0, n = 0, il = 0, ir = 0;
      unsigned int start = 0, end = 0;
      int tmp_cpus = 0;
--

2.17.1

#4 lore raw
In reply to Andrii Nakryiko
Quoted 30 lines · Andrii Nakryiko

This patch switches libbpf_num_possible_cpus() from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing of PERF_EVENT_ARRAY map on HOTPLUG-enabled systems.

On HOTPLUG enabled systems, /sys/devices/system/cpu/possible is going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present). /sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.

On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.

---
tools/lib/bpf/libbpf.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +0 -0
index e0276520171b..45351c074e45 100644

tools/lib/bpf/libbpf.c void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear) · line 5902

Quoted 5 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)
 
 int libbpf_num_possible_cpus(void)
 {
-	static const char *fcpu = "/sys/devices/system/cpu/possible";
+	static const char *fcpu = "/sys/devices/system/cpu/present";

This is _very_ confusing. "possible cpus", "present cpus", and "online cpus" are existing terminologies. I don't think we should force people to remember something like "By possible cpus, libbpf actually means present cpus".

This change works if we call it "libbbpf_num_cpus()". However, libbpf_num_possible_cpus(), should mean possible CPUs.

Thanks,
Song

#5 lore raw
Quoted 42 lines · Song Liu
Quoted 31 lines · Andrii Nakryiko

This patch switches libbpf_num_possible_cpus() from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing of PERF_EVENT_ARRAY map on HOTPLUG-enabled systems.

On HOTPLUG enabled systems, /sys/devices/system/cpu/possible is going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present). /sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.

On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.

---
tools/lib/bpf/libbpf.c | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +0 -0
index e0276520171b..45351c074e45 100644

tools/lib/bpf/libbpf.c void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear) · line 5902

Quoted 5 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)
 
 int libbpf_num_possible_cpus(void)
 {
-     static const char *fcpu = "/sys/devices/system/cpu/possible";
+     static const char *fcpu = "/sys/devices/system/cpu/present";

This is _very_ confusing. "possible cpus", "present cpus", and "online cpus" are existing terminologies. I don't think we should force people to remember something like "By possible cpus, libbpf actually means present cpus".

Quoted 2 lines

This change works if we call it "libbbpf_num_cpus()". However, libbpf_num_possible_cpus(), should mean possible CPUs.

Ok, then if we really need to (I'll play again with my VM to recall all the details of original problem with this that I had before), I'll just add libbpf_num_present_cpus().

Quoted 4 lines

Thanks,
Song

maintainer
#6 lore raw
In reply to Andrii Nakryiko
Quoted 3 lines · Andrii Nakryiko

This patch switches libbpf_num_possible_cpus() from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing of PERF_EVENT_ARRAY map on HOTPLUG-enabled systems.

Those issues should be described in more detail here in the changelog, otherwise noone knows what is meant exactly when glancing at the git log.

Quoted 22 lines

On HOTPLUG enabled systems, /sys/devices/system/cpu/possible is going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present). /sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.

On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.

---
  tools/lib/bpf/libbpf.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +0 -0
index e0276520171b..45351c074e45 100644

tools/lib/bpf/libbpf.c void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear) · line 5902

Quoted 5 lines
tools/lib/bpf/libbpf.c +1 -1
@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)
 
  int libbpf_num_possible_cpus(void)
  {
-	static const char *fcpu = "/sys/devices/system/cpu/possible";
+	static const char *fcpu = "/sys/devices/system/cpu/present";

Problem is that this is going to break things *badly* for per-cpu maps as BPF_DECLARE_PERCPU() relies on possible CPUs, not present ones. And given present<=possible you'll end up corrupting user space when you do a lookup on the map since kernel side operates on possible as well.

Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
  	int len = 0, n = 0, il = 0, ir = 0;
  	unsigned int start = 0, end = 0;
  	int tmp_cpus = 0;
#7 lore raw
Quoted 8 lines · Daniel Borkmann
Quoted 3 lines · Andrii Nakryiko

This patch switches libbpf_num_possible_cpus() from using possible CPU set to present CPU set. This fixes issues with incorrect auto-sizing of PERF_EVENT_ARRAY map on HOTPLUG-enabled systems.

Those issues should be described in more detail here in the changelog, otherwise noone knows what is meant exactly when glancing at the git log.

Sure, I can add more details.

Quoted 24 lines
Quoted 24 lines

On HOTPLUG enabled systems, /sys/devices/system/cpu/possible is going to be a set of any representable (i.e., potentially possible) CPU, which is normally way higher than real amount of CPUs (e.g., 0-127 on VM I've tested on, while there were just two CPU cores actually present). /sys/devices/system/cpu/present, on the other hand, will only contain CPUs that are physically present in the system (even if not online yet), which is what we really want, especially when creating per-CPU maps or perf events.

On systems with HOTPLUG disabled, present and possible are identical, so there is no change of behavior there.

---
  tools/lib/bpf/libbpf.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
tools/lib/bpf/libbpf.c +0 -0
index e0276520171b..45351c074e45 100644
@@ -5899,7 +5899,7 @@ void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear)
 
Quoted 9 lines

tools/lib/bpf/libbpf.c void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear) · line 5902

Quoted 4 lines
tools/lib/bpf/libbpf.c +1 -1
  int libbpf_num_possible_cpus(void)
  {
-     static const char *fcpu = "/sys/devices/system/cpu/possible";
+     static const char *fcpu = "/sys/devices/system/cpu/present";

Problem is that this is going to break things *badly* for per-cpu maps as BPF_DECLARE_PERCPU() relies on possible CPUs, not present ones. And given present<=possible you'll end up corrupting user space when you do a lookup on the map since kernel side operates on possible as well.

Yeah, you are right. Ok, let me go back to my VM and repro original issue I had and see what and why is causing that. I'll see maybe I don't need this fix at all.

Quoted 6 lines
Quoted 5 lines
tools/lib/bpf/libbpf.c +0 -0
      int len = 0, n = 0, il = 0, ir = 0;
      unsigned int start = 0, end = 0;
      int tmp_cpus = 0;