Skip to contentSkip to conversation

drivers/base/node.c: fix userspace break from using bin_attributes for cpumap and cpulist

PATCH Superseded Phil Auld 2 participants 4 messages Jul 13, 2022 – Jul 14, 2022 Driver core

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Phil Auld

author
#0 lore raw

PATCH v3 1/1 drivers/base/node.c: fix userspace break from using bin_attributes for cpumap and cpulist

Using bin_attributes with a 0 size causes fstat and friends to return that 0 size. This breaks userspace code that retrieves the size before reading the file. Rather than reverting 75bd50fa841 ("drivers/base/node.c: use bin_attribute to break the size limitation of cpumap ABI") let's put in a size value at compile time. Use direct comparison and a worst-case maximum to ensure compile time constants. For cpulist the max is on the order of NR_CPUS * (ceil(log10(NR_CPUS)) + 1) which for 8192 is 40960 (8192 * 5). In order to get near that you'd need a system with every other CPU on one node or something similar. e.g. (0,2,4,... 1024,1026...). To simplify the math and support larger NR_CPUS we are using NR_CPUS * 7 to support a future with much larger NR_CPUS. We also set it to a min of PAGE_SIZE to retain the older behavior for smaller NR_CPUS. The cpumap file wants to be something like NR_CPUS/4 + NR_CPUS/32, for the ","s so for simplicity we are using NR_CPUS/2.

On an 80 cpu 4-node sytem (NR_CPUS == 8192)

before:

-r--r--r--. 1 root root 0 Jul 12 14:08 /sys/devices/system/node/node0/cpulist -r--r--r--. 1 root root 0 Jul 11 17:25 /sys/devices/system/node/node0/cpumap

after:

-r--r--r--. 1 root root 57344 Jul 13 11:32 /sys/devices/system/node/node0/cpulist -r--r--r--. 1 root root 4096 Jul 13 11:31 /sys/devices/system/node/node0/cpumap

NR_CPUS = 16384
-r--r--r--. 1 root root 114688 Jul 13 14:03 /sys/devices/system/node/node0/cpulist
-r--r--r--. 1 root root 8192 Jul 13 14:02 /sys/devices/system/node/node0/cpumap

Fixes 75bd50fa841 ("drivers/base/node.c: use bin_attribute to break the size limitation of cpumap ABI")
Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
---
 drivers/base/node.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
drivers/base/node.c +14 -2
index 0ac6376ef7a1..89c932a1d8ca 100644
@@ -45,7 +45,11 @@ static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj,
 	return n;
 }
 
-static BIN_ATTR_RO(cpumap, 0);
+/* Report a valid max size for this file to avoid breaking userspace. We use NR_CPUS/2 as 
+ * a simplification of NR_CPUS/8 + NR_CPUS/32.  Use PAGE_SIZE as a minimum for smaller 
+ * configurations. 
+ */ 
+static BIN_ATTR_RO(cpumap, (((NR_CPUS >> 1) > PAGE_SIZE) ? NR_CPUS >> 1 : PAGE_SIZE));
 
 static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj,
 				   struct bin_attribute *attr, char *buf,
@@ -66,7 +70,15 @@ static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj,
 	return n;
 }
 
-static BIN_ATTR_RO(cpulist, 0);
+/* Report a valid maximum size for this file since 0 breaks userspace, which
+ * may use the size from fstat to allocate a read buffer. 
+ * The value 7 is a hardcoded version of ceil(log10(NR_CPUS)) + 1 for future values
+ * of NR_CPUS that may be upto 2 orders of magnitude larger than 8192.
+ * In a worst case system every other cpu is on one of two nodes. This leads to 
+ * a file like "0,2,4,6,8...1024,...8190,...". Use PAGE_SIZE as a minimum for smaller
+ * NR_CPUS.  
+*/
+static BIN_ATTR_RO(cpulist, (((NR_CPUS * 7) > PAGE_SIZE) ? NR_CPUS * 7 : PAGE_SIZE));
 
 /**
  * struct node_access_nodes - Access class device to hold user visible
Signature · 1 line
2.31.1
#1 lore raw
Quoted 43 lines · Phil Auld

Using bin_attributes with a 0 size causes fstat and friends to return that 0 size. This breaks userspace code that retrieves the size before reading the file. Rather than reverting 75bd50fa841 ("drivers/base/node.c: use bin_attribute to break the size limitation of cpumap ABI") let's put in a size value at compile time. Use direct comparison and a worst-case maximum to ensure compile time constants. For cpulist the max is on the order of NR_CPUS * (ceil(log10(NR_CPUS)) + 1) which for 8192 is 40960 (8192 * 5). In order to get near that you'd need a system with every other CPU on one node or something similar. e.g. (0,2,4,... 1024,1026...). To simplify the math and support larger NR_CPUS we are using NR_CPUS * 7 to support a future with much larger NR_CPUS. We also set it to a min of PAGE_SIZE to retain the older behavior for smaller NR_CPUS. The cpumap file wants to be something like NR_CPUS/4 + NR_CPUS/32, for the ","s so for simplicity we are using NR_CPUS/2.

On an 80 cpu 4-node sytem (NR_CPUS == 8192)

before:

(quoted diff) +0 -2
-r--r--r--. 1 root root 0 Jul 12 14:08 /sys/devices/system/node/node0/cpulist
-r--r--r--. 1 root root 0 Jul 11 17:25 /sys/devices/system/node/node0/cpumap

after:

(quoted diff) +0 -2
-r--r--r--. 1 root root 57344 Jul 13 11:32 /sys/devices/system/node/node0/cpulist
-r--r--r--. 1 root root  4096 Jul 13 11:31 /sys/devices/system/node/node0/cpumap

NR_CPUS = 16384

(quoted diff) +0 -2
-r--r--r--. 1 root root 114688 Jul 13 14:03 /sys/devices/system/node/node0/cpulist
-r--r--r--. 1 root root   8192 Jul 13 14:02 /sys/devices/system/node/node0/cpumap
Fixes 75bd50fa841 ("drivers/base/node.c: use bin_attribute to break the size limitation of cpumap ABI")
Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
---
 drivers/base/node.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
drivers/base/node.c +0 -0
index 0ac6376ef7a1..89c932a1d8ca 100644

drivers/base/node.c static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj, · line 52

Quoted 9 lines
drivers/base/node.c +5 -1
@@ -45,7 +45,11 @@ static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj,
        return n;
 }
 
-static BIN_ATTR_RO(cpumap, 0);
+/* Report a valid max size for this file to avoid breaking userspace. We use NR_CPUS/2 as
+ * a simplification of NR_CPUS/8 + NR_CPUS/32.  Use PAGE_SIZE as a minimum for smaller
+ * configurations.
+ */
+static BIN_ATTR_RO(cpumap, (((NR_CPUS >> 1) > PAGE_SIZE) ? NR_CPUS >> 1 : PAGE_SIZE));

the code should be fine. but the comment seems to be wrong?

/$ cat /sys/devices/system/node/node0/cpumap
00000000,00000000,00000000,000000ff

4 cpus need one byte in hex, 32 cpus need a comma. for 32cpus, we totally need 9 bytes.

Based on your comment, you get 32/8+32/32=5.
should be NR_CPUS/4 ?

Quoted 8 lines
drivers/base/node.c +0 -1
 static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj,
                                   struct bin_attribute *attr, char *buf,
@@ -66,7 +70,15 @@ static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj,
        return n;
 }
 
-static BIN_ATTR_RO(cpulist, 0);

drivers/base/node.c static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj, · line 82

Quoted 10 lines
drivers/base/node.c +9 -0
+/* Report a valid maximum size for this file since 0 breaks userspace, which
+ * may use the size from fstat to allocate a read buffer.
+ * The value 7 is a hardcoded version of ceil(log10(NR_CPUS)) + 1 for future values
+ * of NR_CPUS that may be upto 2 orders of magnitude larger than 8192.
+ * In a worst case system every other cpu is on one of two nodes. This leads to
+ * a file like "0,2,4,6,8...1024,...8190,...". Use PAGE_SIZE as a minimum for smaller
+ * NR_CPUS.
+*/
+static BIN_ATTR_RO(cpulist, (((NR_CPUS * 7) > PAGE_SIZE) ? NR_CPUS * 7 : PAGE_SIZE));
 

It seems to be very sufficient. At least, my poor math tells me 7 bytes can describe cpu id like
"100000," and up to "999999,"
but it is still hard for me to understand the comments :-)

btw, we have a lot of other places which might need this, such as drivers/base/topology.c

so perhaps we can move them to some common place,

#define cpu_bitmap_bytes (((NR_CPUS >> 1) > PAGE_SIZE) ? NR_CPUS >> 1 : PAGE_SIZE)
#define cpu_list_bytes (((NR_CPUS * 7) > PAGE_SIZE) ? NR_CPUS * 7 : PAGE_SIZE)

is include/linux/cpumask.h a good place for it?

Quoted 5 lines
drivers/base/node.c +0 -1
 /**
  * struct node_access_nodes - Access class device to hold user visible
--

2.31.1

Thanks
Barry

author
#2 lore raw
Quoted 56 lines · On Thu, Jul 14, 2022 at 12:23:01PM +1200 Barry Song wrote:
Quoted 43 lines · Phil Auld

Using bin_attributes with a 0 size causes fstat and friends to return that 0 size. This breaks userspace code that retrieves the size before reading the file. Rather than reverting 75bd50fa841 ("drivers/base/node.c: use bin_attribute to break the size limitation of cpumap ABI") let's put in a size value at compile time. Use direct comparison and a worst-case maximum to ensure compile time constants. For cpulist the max is on the order of NR_CPUS * (ceil(log10(NR_CPUS)) + 1) which for 8192 is 40960 (8192 * 5). In order to get near that you'd need a system with every other CPU on one node or something similar. e.g. (0,2,4,... 1024,1026...). To simplify the math and support larger NR_CPUS we are using NR_CPUS * 7 to support a future with much larger NR_CPUS. We also set it to a min of PAGE_SIZE to retain the older behavior for smaller NR_CPUS. The cpumap file wants to be something like NR_CPUS/4 + NR_CPUS/32, for the ","s so for simplicity we are using NR_CPUS/2.

On an 80 cpu 4-node sytem (NR_CPUS == 8192)

before:

(quoted diff) +0 -2
-r--r--r--. 1 root root 0 Jul 12 14:08 /sys/devices/system/node/node0/cpulist
-r--r--r--. 1 root root 0 Jul 11 17:25 /sys/devices/system/node/node0/cpumap

after:

(quoted diff) +0 -2
-r--r--r--. 1 root root 57344 Jul 13 11:32 /sys/devices/system/node/node0/cpulist
-r--r--r--. 1 root root  4096 Jul 13 11:31 /sys/devices/system/node/node0/cpumap

NR_CPUS = 16384

(quoted diff) +0 -2
-r--r--r--. 1 root root 114688 Jul 13 14:03 /sys/devices/system/node/node0/cpulist
-r--r--r--. 1 root root   8192 Jul 13 14:02 /sys/devices/system/node/node0/cpumap
Fixes 75bd50fa841 ("drivers/base/node.c: use bin_attribute to break the size limitation of cpumap ABI")
Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
---
 drivers/base/node.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)
drivers/base/node.c +0 -0
index 0ac6376ef7a1..89c932a1d8ca 100644

drivers/base/node.c static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj, · line 52

Quoted 9 lines
drivers/base/node.c +5 -1
@@ -45,7 +45,11 @@ static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj,
        return n;
 }
 
-static BIN_ATTR_RO(cpumap, 0);
+/* Report a valid max size for this file to avoid breaking userspace. We use NR_CPUS/2 as
+ * a simplification of NR_CPUS/8 + NR_CPUS/32.  Use PAGE_SIZE as a minimum for smaller
+ * configurations.
+ */
+static BIN_ATTR_RO(cpumap, (((NR_CPUS >> 1) > PAGE_SIZE) ? NR_CPUS >> 1 : PAGE_SIZE));

the code should be fine. but the comment seems to be wrong?

Quoted 9 lines

/$ cat /sys/devices/system/node/node0/cpumap
00000000,00000000,00000000,000000ff

4 cpus need one byte in hex, 32 cpus need a comma. for 32cpus, we totally need 9 bytes.

Based on your comment, you get 32/8+32/32=5.
should be NR_CPUS/4 ?

Yes, sorry. Meant /4 as in the commit message. I'll fix that.

Quoted 7 lines
Quoted 7 lines
drivers/base/node.c +0 -0
 static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj,
                                   struct bin_attribute *attr, char *buf,
@@ -66,7 +70,15 @@ static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj,
        return n;
 }
 
Quoted 17 lines

drivers/base/node.c static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj, · line 82

Quoted 11 lines
drivers/base/node.c +9 -1
-static BIN_ATTR_RO(cpulist, 0);
+/* Report a valid maximum size for this file since 0 breaks userspace, which
+ * may use the size from fstat to allocate a read buffer.
+ * The value 7 is a hardcoded version of ceil(log10(NR_CPUS)) + 1 for future values
+ * of NR_CPUS that may be upto 2 orders of magnitude larger than 8192.
+ * In a worst case system every other cpu is on one of two nodes. This leads to
+ * a file like "0,2,4,6,8...1024,...8190,...". Use PAGE_SIZE as a minimum for smaller
+ * NR_CPUS.
+*/
+static BIN_ATTR_RO(cpulist, (((NR_CPUS * 7) > PAGE_SIZE) ? NR_CPUS * 7 : PAGE_SIZE));
 

It seems to be very sufficient. At least, my poor math tells me 7 bytes can describe cpu id like
"100000," and up to "999999,"
but it is still hard for me to understand the comments :-)

I picked 7 based on Greg saying there might be systems with 2 orders of magnitude more than 8192 cpus. Personally I think lock contention and percpu data will start to be a problem before that. I couldn't get x86 to build with more than NR_CPUS=16k. But it allows for future expansion.

What would you like the comment to say that makes more sense to you? Should I put some of those really large cpuids in the worst case example? Take that out completely?

Quoted 10 lines

btw, we have a lot of other places which might need this, such as drivers/base/topology.c

so perhaps we can move them to some common place,

#define cpu_bitmap_bytes (((NR_CPUS >> 1) > PAGE_SIZE) ? NR_CPUS >> 1 : PAGE_SIZE)
#define cpu_list_bytes (((NR_CPUS * 7) > PAGE_SIZE) ? NR_CPUS * 7 : PAGE_SIZE)

is include/linux/cpumask.h a good place for it?

My concern is the ones that are breaking actual userspace code. But yes, those otherwise have the same 0 size.

It seems somewhat specific to drivers/base. Maybe there's a less global place to put those closer. I can look and do it this way if that will help get it fixed.

Cheers,
Phil

Quoted 10 lines
Quoted 6 lines
drivers/base/node.c +0 -1
 /**
  * struct node_access_nodes - Access class device to hold user visible
--

2.31.1

Thanks
Barry

author
#3 lore raw
Quoted 11 lines · On Thu, Jul 14, 2022 at 08:59:25AM -0400 Phil Auld wrote:
Quoted 10 lines · On Thu, Jul 14, 2022 at 12:23:01PM +1200 Barry Song wrote:

btw, we have a lot of other places which might need this, such as drivers/base/topology.c

so perhaps we can move them to some common place,

#define cpu_bitmap_bytes (((NR_CPUS >> 1) > PAGE_SIZE) ? NR_CPUS >> 1 : PAGE_SIZE)
#define cpu_list_bytes (((NR_CPUS * 7) > PAGE_SIZE) ? NR_CPUS * 7 : PAGE_SIZE)

is include/linux/cpumask.h a good place for it?

drivers/base/base.h does not look like the right place, so I think your cpumask.h idea is better. I'll put in there and update the topology.c BIN_ATTRs.

Thanks,
Phil

Quoted 23 lines

My concern is the ones that are breaking actual userspace code. But yes, those otherwise have the same 0 size.

It seems somewhat specific to drivers/base. Maybe there's a less global place to put those closer. I can look and do it this way if that will help get it fixed.

Cheers,
Phil

Quoted 10 lines
Quoted 6 lines

/**
* struct node_access_nodes - Access class device to hold user visible
--
2.31.1

Thanks
Barry