drivers/base/node.c: fix userspace break from using bin_attributes for cpumap and cpulist
PATCH v3 1/1 drivers/base/node.c: fix userspace break from using bin_attributes for cpumap and cpulist
Using bin_attributes with a 0 size causes fstat and friends to return that 0 size. This breaks userspace code that retrieves the size before reading the file. Rather than reverting 75bd50fa841 ("drivers/base/node.c: use bin_attribute to break the size limitation of cpumap ABI") let's put in a size value at compile time. Use direct comparison and a worst-case maximum to ensure compile time constants. For cpulist the max is on the order of NR_CPUS * (ceil(log10(NR_CPUS)) + 1) which for 8192 is 40960 (8192 * 5). In order to get near that you'd need a system with every other CPU on one node or something similar. e.g. (0,2,4,... 1024,1026...). To simplify the math and support larger NR_CPUS we are using NR_CPUS * 7 to support a future with much larger NR_CPUS. We also set it to a min of PAGE_SIZE to retain the older behavior for smaller NR_CPUS. The cpumap file wants to be something like NR_CPUS/4 + NR_CPUS/32, for the ","s so for simplicity we are using NR_CPUS/2.
On an 80 cpu 4-node sytem (NR_CPUS == 8192)
before:
-r--r--r--. 1 root root 0 Jul 12 14:08 /sys/devices/system/node/node0/cpulist -r--r--r--. 1 root root 0 Jul 11 17:25 /sys/devices/system/node/node0/cpumap
after:
-r--r--r--. 1 root root 57344 Jul 13 11:32 /sys/devices/system/node/node0/cpulist -r--r--r--. 1 root root 4096 Jul 13 11:31 /sys/devices/system/node/node0/cpumap
NR_CPUS = 16384
-r--r--r--. 1 root root 114688 Jul 13 14:03 /sys/devices/system/node/node0/cpulist
-r--r--r--. 1 root root 8192 Jul 13 14:02 /sys/devices/system/node/node0/cpumap
--- drivers/base/node.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
drivers/base/node.c +14 -2
@@ -45,7 +45,11 @@ static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj,return n; } -static BIN_ATTR_RO(cpumap, 0); +/* Report a valid max size for this file to avoid breaking userspace. We use NR_CPUS/2 as +* a simplification of NR_CPUS/8 + NR_CPUS/32. Use PAGE_SIZE as a minimum for smaller +* configurations. +*/ +static BIN_ATTR_RO(cpumap, (((NR_CPUS >> 1) > PAGE_SIZE) ? NR_CPUS >> 1 : PAGE_SIZE)); static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj, struct bin_attribute *attr, char *buf,
@@ -66,7 +70,15 @@ static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj,return n; } -static BIN_ATTR_RO(cpulist, 0); +/* Report a valid maximum size for this file since 0 breaks userspace, which +* may use the size from fstat to allocate a read buffer. +* The value 7 is a hardcoded version of ceil(log10(NR_CPUS)) + 1 for future values +* of NR_CPUS that may be upto 2 orders of magnitude larger than 8192. +* In a worst case system every other cpu is on one of two nodes. This leads to +* a file like "0,2,4,6,8...1024,...8190,...". Use PAGE_SIZE as a minimum for smaller +* NR_CPUS. +*/ +static BIN_ATTR_RO(cpulist, (((NR_CPUS * 7) > PAGE_SIZE) ? NR_CPUS * 7 : PAGE_SIZE));/** * struct node_access_nodes - Access class device to hold user visible
Signature · 1 line
2.31.1
Quoted 43 lines · Phil Auld
Using
bin_attributeswith a 0 size causes fstat and friends to return that 0 size. This breaks userspace code that retrieves the size before reading the file. Rather than reverting75bd50fa841("drivers/base/node.c: usebin_attributeto break the size limitation of cpumap ABI") let's put in a size value at compile time. Use direct comparison and a worst-case maximum to ensure compile time constants. For cpulist the max is on the order ofNR_CPUS* (ceil(log10(NR_CPUS)) + 1) which for 8192 is 40960 (8192 * 5). In order to get near that you'd need a system with every other CPU on one node or something similar. e.g. (0,2,4,... 1024,1026...). To simplify the math and support largerNR_CPUSwe are usingNR_CPUS* 7 to support a future with much largerNR_CPUS. We also set it to a min ofPAGE_SIZEto retain the older behavior for smallerNR_CPUS. The cpumap file wants to be something like NR_CPUS/4 + NR_CPUS/32, for the ","s so for simplicity we are using NR_CPUS/2.On an 80 cpu 4-node sytem (
NR_CPUS== 8192)before:
(quoted diff) +0
-2-r--r--r--. 1 root root 0 Jul 12 14:08 /sys/devices/system/node/node0/cpulist -r--r--r--. 1 root root 0 Jul 11 17:25 /sys/devices/system/node/node0/cpumapafter:
(quoted diff) +0
-2-r--r--r--. 1 root root 57344 Jul 13 11:32 /sys/devices/system/node/node0/cpulist -r--r--r--. 1 root root 4096 Jul 13 11:31 /sys/devices/system/node/node0/cpumap
NR_CPUS= 16384(quoted diff) +0
-2-r--r--r--. 1 root root 114688 Jul 13 14:03 /sys/devices/system/node/node0/cpulist -r--r--r--. 1 root root 8192 Jul 13 14:02 /sys/devices/system/node/node0/cpumapFixes 75bd50fa841 ("drivers/base/node.c: use bin_attribute to break the size limitation of cpumap ABI")Cc Greg Kroah-Hartman <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Signed-off-by Phil Auld <[email protected]>--- drivers/base/node.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)drivers/base/node.c +0
-0
drivers/base/node.c static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj, · line 52
Quoted 9 lines
drivers/base/node.c +5
-1@@ -45,7 +45,11 @@ static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj,return n; } -static BIN_ATTR_RO(cpumap, 0); +/* Report a valid max size for this file to avoid breaking userspace. We use NR_CPUS/2 as +* a simplification of NR_CPUS/8 + NR_CPUS/32. Use PAGE_SIZE as a minimum for smaller +* configurations. +*/ +static BIN_ATTR_RO(cpumap, (((NR_CPUS >> 1) > PAGE_SIZE) ? NR_CPUS >> 1 : PAGE_SIZE));
the code should be fine. but the comment seems to be wrong?
/$ cat /sys/devices/system/node/node0/cpumap
00000000,00000000,00000000,000000ff
4 cpus need one byte in hex, 32 cpus need a comma. for 32cpus, we totally need 9 bytes.
Based on your comment, you get 32/8+32/32=5.
should be NR_CPUS/4 ?
Quoted 8 lines
drivers/base/node.c +0
-1static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj, struct bin_attribute *attr, char *buf,@@ -66,7 +70,15 @@ static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj,return n; } -static BIN_ATTR_RO(cpulist, 0);
drivers/base/node.c static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj, · line 82
Quoted 10 lines
drivers/base/node.c +9
-0+/* Report a valid maximum size for this file since 0 breaks userspace, which +* may use the size from fstat to allocate a read buffer. +* The value 7 is a hardcoded version of ceil(log10(NR_CPUS)) + 1 for future values +* of NR_CPUS that may be upto 2 orders of magnitude larger than 8192. +* In a worst case system every other cpu is on one of two nodes. This leads to +* a file like "0,2,4,6,8...1024,...8190,...". Use PAGE_SIZE as a minimum for smaller +* NR_CPUS. +*/ +static BIN_ATTR_RO(cpulist, (((NR_CPUS * 7) > PAGE_SIZE) ? NR_CPUS * 7 : PAGE_SIZE));
It seems to be very sufficient. At least, my poor math tells me 7 bytes can describe cpu id like
"100000," and up to "999999,"
but it is still hard for me to understand the comments :-)
btw, we have a lot of other places which might need this, such as drivers/base/topology.c
so perhaps we can move them to some common place,
#define cpu_bitmap_bytes (((NR_CPUS >> 1) > PAGE_SIZE) ? NR_CPUS >> 1 : PAGE_SIZE)
#define cpu_list_bytes (((NR_CPUS * 7) > PAGE_SIZE) ? NR_CPUS * 7 : PAGE_SIZE)
is include/linux/cpumask.h a good place for it?
Quoted 5 lines
drivers/base/node.c +0
-1/** * struct node_access_nodes - Access class device to hold user visible -- 2.31.1
Thanks
Barry
Quoted 56 lines · On Thu, Jul 14, 2022 at 12:23:01PM +1200 Barry Song wrote:
Quoted 43 lines · Phil Auld
Using
bin_attributeswith a 0 size causes fstat and friends to return that 0 size. This breaks userspace code that retrieves the size before reading the file. Rather than reverting75bd50fa841("drivers/base/node.c: usebin_attributeto break the size limitation of cpumap ABI") let's put in a size value at compile time. Use direct comparison and a worst-case maximum to ensure compile time constants. For cpulist the max is on the order ofNR_CPUS* (ceil(log10(NR_CPUS)) + 1) which for 8192 is 40960 (8192 * 5). In order to get near that you'd need a system with every other CPU on one node or something similar. e.g. (0,2,4,... 1024,1026...). To simplify the math and support largerNR_CPUSwe are usingNR_CPUS* 7 to support a future with much largerNR_CPUS. We also set it to a min ofPAGE_SIZEto retain the older behavior for smallerNR_CPUS. The cpumap file wants to be something like NR_CPUS/4 + NR_CPUS/32, for the ","s so for simplicity we are using NR_CPUS/2.On an 80 cpu 4-node sytem (
NR_CPUS== 8192)before:
(quoted diff) +0
-2-r--r--r--. 1 root root 0 Jul 12 14:08 /sys/devices/system/node/node0/cpulist -r--r--r--. 1 root root 0 Jul 11 17:25 /sys/devices/system/node/node0/cpumapafter:
(quoted diff) +0
-2-r--r--r--. 1 root root 57344 Jul 13 11:32 /sys/devices/system/node/node0/cpulist -r--r--r--. 1 root root 4096 Jul 13 11:31 /sys/devices/system/node/node0/cpumap
NR_CPUS= 16384(quoted diff) +0
-2-r--r--r--. 1 root root 114688 Jul 13 14:03 /sys/devices/system/node/node0/cpulist -r--r--r--. 1 root root 8192 Jul 13 14:02 /sys/devices/system/node/node0/cpumapFixes 75bd50fa841 ("drivers/base/node.c: use bin_attribute to break the size limitation of cpumap ABI")Cc Greg Kroah-Hartman <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Signed-off-by Phil Auld <[email protected]>--- drivers/base/node.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)drivers/base/node.c +0
-0drivers/base/node.c static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj, · line 52
Quoted 9 lines
drivers/base/node.c +5
-1@@ -45,7 +45,11 @@ static inline ssize_t cpumap_read(struct file *file, struct kobject *kobj,return n; } -static BIN_ATTR_RO(cpumap, 0); +/* Report a valid max size for this file to avoid breaking userspace. We use NR_CPUS/2 as +* a simplification of NR_CPUS/8 + NR_CPUS/32. Use PAGE_SIZE as a minimum for smaller +* configurations. +*/ +static BIN_ATTR_RO(cpumap, (((NR_CPUS >> 1) > PAGE_SIZE) ? NR_CPUS >> 1 : PAGE_SIZE));the code should be fine. but the comment seems to be wrong?
Quoted 9 lines
/$ cat
/sys/devices/system/node/node0/cpumap
00000000,00000000,00000000,000000ff4 cpus need one byte in hex, 32 cpus need a comma. for 32cpus, we totally need 9 bytes.
Based on your comment, you get 32/8+32/32=5.
should be NR_CPUS/4 ?
Yes, sorry. Meant /4 as in the commit message. I'll fix that.
Quoted 7 lines
Quoted 7 lines
drivers/base/node.c +0
-0static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj, struct bin_attribute *attr, char *buf,@@ -66,7 +70,15 @@ static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj,return n; }
Quoted 17 lines
drivers/base/node.c static inline ssize_t cpulist_read(struct file *file, struct kobject *kobj, · line 82
Quoted 11 lines
drivers/base/node.c +9
-1-static BIN_ATTR_RO(cpulist, 0); +/* Report a valid maximum size for this file since 0 breaks userspace, which +* may use the size from fstat to allocate a read buffer. +* The value 7 is a hardcoded version of ceil(log10(NR_CPUS)) + 1 for future values +* of NR_CPUS that may be upto 2 orders of magnitude larger than 8192. +* In a worst case system every other cpu is on one of two nodes. This leads to +* a file like "0,2,4,6,8...1024,...8190,...". Use PAGE_SIZE as a minimum for smaller +* NR_CPUS. +*/ +static BIN_ATTR_RO(cpulist, (((NR_CPUS * 7) > PAGE_SIZE) ? NR_CPUS * 7 : PAGE_SIZE));It seems to be very sufficient. At least, my poor math tells me 7 bytes can describe cpu id like
"100000," and up to "999999,"
but it is still hard for me to understand the comments :-)
I picked 7 based on Greg saying there might be systems with 2 orders of magnitude more than 8192 cpus. Personally I think lock contention and percpu data will start to be a problem before that. I couldn't get x86 to build with more than NR_CPUS=16k. But it allows for future expansion.
What would you like the comment to say that makes more sense to you? Should I put some of those really large cpuids in the worst case example? Take that out completely?
Quoted 10 lines
btw, we have a lot of other places which might need this, such as
drivers/base/topology.cso perhaps we can move them to some common place,
#define
cpu_bitmap_bytes(((NR_CPUS>> 1) >PAGE_SIZE) ?NR_CPUS>> 1 :PAGE_SIZE)
#definecpu_list_bytes(((NR_CPUS* 7) >PAGE_SIZE) ?NR_CPUS* 7 :PAGE_SIZE)is
include/linux/cpumask.ha good place for it?
My concern is the ones that are breaking actual userspace code. But yes, those otherwise have the same 0 size.
It seems somewhat specific to drivers/base. Maybe there's a less global place to put those closer. I can look and do it this way if that will help get it fixed.
Cheers,
Phil
Quoted 10 lines
Quoted 6 lines
drivers/base/node.c +0
-1/** * struct node_access_nodes - Access class device to hold user visible -- 2.31.1
Thanks
Barry
Quoted 11 lines · On Thu, Jul 14, 2022 at 08:59:25AM -0400 Phil Auld wrote:
Quoted 10 lines · On Thu, Jul 14, 2022 at 12:23:01PM +1200 Barry Song wrote:
btw, we have a lot of other places which might need this, such as
drivers/base/topology.cso perhaps we can move them to some common place,
#define
cpu_bitmap_bytes(((NR_CPUS>> 1) >PAGE_SIZE) ?NR_CPUS>> 1 :PAGE_SIZE)
#definecpu_list_bytes(((NR_CPUS* 7) >PAGE_SIZE) ?NR_CPUS* 7 :PAGE_SIZE)is
include/linux/cpumask.ha good place for it?
drivers/base/base.h does not look like the right place, so I think your cpumask.h idea is better. I'll put in there and update the topology.c BIN_ATTRs.
Thanks,
Phil
Quoted 23 lines
My concern is the ones that are breaking actual userspace code. But yes, those otherwise have the same 0 size.
It seems somewhat specific to
drivers/base. Maybe there's a less global place to put those closer. I can look and do it this way if that will help get it fixed.Cheers,
PhilQuoted 10 lines
Quoted 6 lines
/**
* structnode_access_nodes- Access class device to hold user visible
--
2.31.1Thanks
Barry