Skip to contentSkip to conversation

cpumask: fix compat getaffinity

PATCH Arnd Bergmann 6 participants 22 messages May 7, 2010 – Dec 15, 2010 Bitmap API

lore
Share & tools
lore
#0 lore raw

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

On NUMA systems with less than 32 possibly CPUs, the current compat_sys_sched_getaffinity now returns '4' instead of the actual NR_CPUS/8, which makes libnuma bail out when parsing the cpumap.

This restores the original return value for now. If we ever get around to changing cpumask_size to return only possibly CPUs, we will also need to make the format of the cpumap file.

We should probably also make libnuma able to deal with the modified kernel interface, so it can operate on all kernels.

kernel/compat.c +1 -1
@@ -497,7 +497,7 @@ asmlinkage long compat_sys_sched_getaffinity(compat_pid_t pid, unsigned int len,
 	unsigned long *k;
 	unsigned int min_length = cpumask_size();
 
-	if (nr_cpu_ids <= BITS_PER_COMPAT_LONG)
+	if (NR_CPUS <= BITS_PER_COMPAT_LONG)
 		min_length = sizeof(compat_ulong_t);
 
 	if (len < min_length)
#1 lore raw
Quoted 8 lines · Arnd Bergmann

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

On NUMA systems with less than 32 possibly CPUs, the current compat_sys_sched_getaffinity now returns '4' instead of the actual NR_CPUS/8, which makes libnuma bail out when parsing the cpumap.

Really? AFAICT the cpumap is printed using nr_cpu_ids too. Can you give an example of what cpumap is on this system?

Confused,
Rusty.

#2 lore raw
Quoted 6 lines · On Saturday 08 May 2010 10:30:47 Rusty Russell wrote:
Quoted 4 lines · Arnd Bergmann

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

Quoted 7 lines
Quoted 4 lines

On NUMA systems with less than 32 possibly CPUs, the current compat_sys_sched_getaffinity now returns '4' instead of the actual NR_CPUS/8, which makes libnuma bail out when parsing the cpumap.

Really? AFAICT the cpumap is printed using nr_cpu_ids too. Can you give an example of what cpumap is on this system?

On Ken's PS3 running the Fedora 12 kernel (2.6.32-something), the output from my memory is 00000000,00000000,00000000,00000003\n. NR_CPUs is 128, nr_cpu_ids is most likely 2.

Arnd

#3 lore raw
Quoted 12 lines · Arnd Bergmann
Quoted 11 lines · On Saturday 08 May 2010 10:30:47 Rusty Russell wrote:
Quoted 8 lines · Arnd Bergmann

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

On NUMA systems with less than 32 possibly CPUs, the current compat_sys_sched_getaffinity now returns '4' instead of the actual NR_CPUS/8, which makes libnuma bail out when parsing the cpumap.

Quoted 6 lines
Quoted 2 lines

Really? AFAICT the cpumap is printed using nr_cpu_ids too. Can you give an example of what cpumap is on this system?

On Ken's PS3 running the Fedora 12 kernel (2.6.32-something), the output from my memory is 00000000,00000000,00000000,00000003\n. NR_CPUs is 128, nr_cpu_ids is most likely 2.

Ah, I see. CONFIG_CPUMASK_OFFSTACK=n.

The nr_cpumask_bits is really an optimization for cpumask_first etc: making it a constant (NR_CPUS) rather than a variable (nr_cpu_ids) for small NR_CPUS makes it slightly faster.

However, we also use this for scnprintf et al, revealing this inconsistency. Your patch would break libnuma on CONFIG_CPUMASK_OFFSTACK=y.

How's this?

cpumask: use nr_cpu_ids for printing and parsing cpumasks

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

However, the result now returned reflects nr_cpu_ids, and cpumask_scnprintf et al. use nr_cpumask_bits which is NR_CPUS (for CONFIG_CPUMASK_OFFSTACK=n) or nr_cpu_ids (for CONFIG_CPUMASK_OFFSTACK=y).

We should use nr_cpu_ids consistently.

include/linux/cpumask.h +4 -5
@@ -529,7 +529,7 @@ static inline void cpumask_copy(struct c
 static inline int cpumask_scnprintf(char *buf, int len,
 				    const struct cpumask *srcp)
 {
-	return bitmap_scnprintf(buf, len, cpumask_bits(srcp), nr_cpumask_bits);
+	return bitmap_scnprintf(buf, len, cpumask_bits(srcp), nr_cpu_ids);
 }
 
 /**
@@ -543,7 +543,7 @@ static inline int cpumask_scnprintf(char
 static inline int cpumask_parse_user(const char __user *buf, int len,
 				     struct cpumask *dstp)
 {
-	return bitmap_parse_user(buf, len, cpumask_bits(dstp), nr_cpumask_bits);
+	return bitmap_parse_user(buf, len, cpumask_bits(dstp), nr_cpu_ids);
 }
 
 /**
@@ -558,8 +558,7 @@ static inline int cpumask_parse_user(con
 static inline int cpulist_scnprintf(char *buf, int len,
 				    const struct cpumask *srcp)
 {
-	return bitmap_scnlistprintf(buf, len, cpumask_bits(srcp),
-				    nr_cpumask_bits);
+	return bitmap_scnlistprintf(buf, len, cpumask_bits(srcp), nr_cpu_ids);
 }
 
 /**
@@ -572,7 +571,7 @@ static inline int cpulist_scnprintf(char
  */
 static inline int cpulist_parse(const char *buf, struct cpumask *dstp)
 {
-	return bitmap_parselist(buf, cpumask_bits(dstp), nr_cpumask_bits);
+	return bitmap_parselist(buf, cpumask_bits(dstp), nr_cpu_ids);
 }
 
 /**
#4 lore raw
Quoted 13 lines

How's this?

cpumask: use nr_cpu_ids for printing and parsing cpumasks

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

However, the result now returned reflects nr_cpu_ids, and cpumask_scnprintf et al. use nr_cpumask_bits which is NR_CPUS (for CONFIG_CPUMASK_OFFSTACK=n) or nr_cpu_ids (for CONFIG_CPUMASK_OFFSTACK=y).

Quoted 5 lines

We should use nr_cpu_ids consistently.

Well, This patch seems to have ABI change. please don't send abi-change to -stable.

note: I'm not against this patch itself.

Quoted 46 lines
include/linux/cpumask.h +4 -6
@@ -529,7 +529,7 @@ static inline void cpumask_copy(struct c
 static inline int cpumask_scnprintf(char *buf, int len,
 				    const struct cpumask *srcp)
 {
-	return bitmap_scnprintf(buf, len, cpumask_bits(srcp), nr_cpumask_bits);
+	return bitmap_scnprintf(buf, len, cpumask_bits(srcp), nr_cpu_ids);
 }
 
 /**
@@ -543,7 +543,7 @@ static inline int cpumask_scnprintf(char
 static inline int cpumask_parse_user(const char __user *buf, int len,
 				     struct cpumask *dstp)
 {
-	return bitmap_parse_user(buf, len, cpumask_bits(dstp), nr_cpumask_bits);
+	return bitmap_parse_user(buf, len, cpumask_bits(dstp), nr_cpu_ids);
 }
 
 /**
@@ -558,8 +558,7 @@ static inline int cpumask_parse_user(con
 static inline int cpulist_scnprintf(char *buf, int len,
 				    const struct cpumask *srcp)
 {
-	return bitmap_scnlistprintf(buf, len, cpumask_bits(srcp),
-				    nr_cpumask_bits);
+	return bitmap_scnlistprintf(buf, len, cpumask_bits(srcp), nr_cpu_ids);
 }
 
 /**
@@ -572,7 +571,7 @@ static inline int cpulist_scnprintf(char
  */
 static inline int cpulist_parse(const char *buf, struct cpumask *dstp)
 {
-	return bitmap_parselist(buf, cpumask_bits(dstp), nr_cpumask_bits);
+	return bitmap_parselist(buf, cpumask_bits(dstp), nr_cpu_ids);
 }
 
 /**
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#5 lore raw
Quoted 13 lines · KOSAKI Motohiro
Quoted 13 lines

How's this?

cpumask: use nr_cpu_ids for printing and parsing cpumasks

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

However, the result now returned reflects nr_cpu_ids, and cpumask_scnprintf et al. use nr_cpumask_bits which is NR_CPUS (for CONFIG_CPUMASK_OFFSTACK=n) or nr_cpu_ids (for CONFIG_CPUMASK_OFFSTACK=y).

Quoted 7 lines
Quoted 5 lines

We should use nr_cpu_ids consistently.

Well, This patch seems to have ABI change. please don't send abi-change to -stable.

Why? There is no such thing as a "stable" internal abi in the kernel, and that includes the -stable kernel releases.

If it fixes a bug, that's all the requirement is.

Quoted 1 line

note: I'm not against this patch itself.

Great, then it should be just fine, right?

thanks,

greg k-h

#6 lore raw
Quoted 16 lines · Greg KH
Quoted 15 lines · KOSAKI Motohiro
Quoted 15 lines

How's this?

cpumask: use nr_cpu_ids for printing and parsing cpumasks

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

However, the result now returned reflects nr_cpu_ids, and cpumask_scnprintf et al. use nr_cpumask_bits which is NR_CPUS (for CONFIG_CPUMASK_OFFSTACK=n) or nr_cpu_ids (for CONFIG_CPUMASK_OFFSTACK=y).

We should use nr_cpu_ids consistently.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Well, This patch seems to have ABI change. please don't send abi-change to -stable.

Why? There is no such thing as a "stable" internal abi in the kernel, and that includes the -stable kernel releases.

He's referring to the change in sysfs output. However, the ABI involved is already defined to be robust against change of NR_CPUS, so changing it to nr_cpu_ids is OK.

Of course, if libnuma weren't abusing the ABI, this change wouldn't be necessary :(

Cheers,
Rusty.

#7 lore raw
Quoted 21 lines · Greg KH
Quoted 20 lines · KOSAKI Motohiro
Quoted 19 lines
Quoted 18 lines

How's this?

cpumask: use nr_cpu_ids for printing and parsing cpumasks

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

However, the result now returned reflects nr_cpu_ids, and cpumask_scnprintf et al. use nr_cpumask_bits which is NR_CPUS (for CONFIG_CPUMASK_OFFSTACK=n) or nr_cpu_ids (for CONFIG_CPUMASK_OFFSTACK=y).

We should use nr_cpu_ids consistently.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Well, This patch seems to have ABI change. please don't send abi-change to -stable.

Why? There is no such thing as a "stable" internal abi in the kernel, and that includes the -stable kernel releases.

He's referring to the change in sysfs output. However, the ABI involved is already defined to be robust against change of NR_CPUS, so changing it to nr_cpu_ids is OK.

hmm...

ok, I'm cancel my previous claim. generically I think I'm right. but in this case I agree the patch's risk is very low.

sorry for noise.

Quoted 2 lines

Of course, if libnuma weren't abusing the ABI, this change wouldn't be necessary :(

yes yes...

#8 lore raw
In reply to Greg KH
Quoted 20 lines · KOSAKI Motohiro
Quoted 19 lines
Quoted 18 lines

How's this?

cpumask: use nr_cpu_ids for printing and parsing cpumasks

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

However, the result now returned reflects nr_cpu_ids, and cpumask_scnprintf et al. use nr_cpumask_bits which is NR_CPUS (for CONFIG_CPUMASK_OFFSTACK=n) or nr_cpu_ids (for CONFIG_CPUMASK_OFFSTACK=y).

We should use nr_cpu_ids consistently.

Quoted 6 lines
Quoted 1 line

Well, This patch seems to have ABI change. please don't send abi-change to -stable.

Why? There is no such thing as a "stable" internal abi in the kernel, and that includes the -stable kernel releases.

If it fixes a bug, that's all the requirement is.

AFAIK, -stable is mainly used for distro and they have many and many packages. we can't assume no app read /sys files directly. IOW, To change /sys printing format is a bit risky change. frequently compatibility breaking natually break code stability.

Quoted 3 lines
Quoted 1 line

note: I'm not against this patch itself.

Great, then it should be just fine, right?

Yup, fine.

Thanks.

#9 lore raw
Quoted 25 lines · KOSAKI Motohiro
Quoted 25 lines · KOSAKI Motohiro
Quoted 13 lines
Quoted 13 lines

How's this?

cpumask: use nr_cpu_ids for printing and parsing cpumasks

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

However, the result now returned reflects nr_cpu_ids, and cpumask_scnprintf et al. use nr_cpumask_bits which is NR_CPUS (for CONFIG_CPUMASK_OFFSTACK=n) or nr_cpu_ids (for CONFIG_CPUMASK_OFFSTACK=y).

Quoted 7 lines
Quoted 5 lines

We should use nr_cpu_ids consistently.

Well, This patch seems to have ABI change. please don't send abi-change to -stable.

Why? There is no such thing as a "stable" internal abi in the kernel, and that includes the -stable kernel releases.

Quoted 6 lines
Quoted 1 line

If it fixes a bug, that's all the requirement is.

AFAIK, -stable is mainly used for distro and they have many and many packages. we can't assume no app read /sys files directly. IOW, To change /sys printing format is a bit risky change. frequently compatibility breaking natually break code stability.

Ah, the sysfs user/kernel API is what you are referring to here, right? If so, remember, any changes need to be documented in Documentation/API :)

And as for distros using -stable for their releases, that's fine, they well know the risks/benefits for that and handle changes on their own.

thanks,

greg k-h

#10 lore raw
Quoted 10 lines · On Tuesday 11 May 2010 17:20:02 Greg KH wrote:
Quoted 4 lines

AFAIK, -stable is mainly used for distro and they have many and many packages. we can't assume no app read /sys files directly. IOW, To change /sys printing format is a bit risky change. frequently compatibility breaking natually break code stability.

Ah, the sysfs user/kernel API is what you are referring to here, right? If so, remember, any changes need to be documented in Documentation/API :)
And as for distros using -stable for their releases, that's fine, they well know the risks/benefits for that and handle changes on their own.

Would it make sense to use my initial patch for -stable, which reverts the ABI back to before the change that caused the problem, but apply the correct fix (changing the ABI throughout) for future releases?

Arnd

#11 lore raw
Quoted 12 lines · Arnd Bergmann
Quoted 10 lines · On Tuesday 11 May 2010 17:20:02 Greg KH wrote:
Quoted 4 lines

AFAIK, -stable is mainly used for distro and they have many and many packages. we can't assume no app read /sys files directly. IOW, To change /sys printing format is a bit risky change. frequently compatibility breaking natually break code stability.

Ah, the sysfs user/kernel API is what you are referring to here, right? If so, remember, any changes need to be documented in Documentation/API :)
And as for distros using -stable for their releases, that's fine, they well know the risks/benefits for that and handle changes on their own.

Quoted 3 lines

Would it make sense to use my initial patch for -stable, which reverts the ABI back to before the change that caused the problem, but apply the correct fix (changing the ABI throughout) for future releases?

I do not know, as I really don't know what the issue is here.

I trust that you, and the others, can come up with a fix for mainline properly, and if it applies to the stable tree, feel free to forward it to [email protected] for submission then.

thanks,

greg k-h

#12 lore raw
In reply to Arnd Bergmann

At least for parsing, we need to allocate and parse NR_CPUS until all places like arch/powerpc/platforms/pseries/xics.c that compare a user-supplied mask to CPUMASK_ALL are eliminated.

Quoted 3 lines

Would it make sense to use my initial patch for -stable, which reverts the ABI back to before the change that caused the problem, but apply the correct fix (changing the ABI throughout) for future releases?

This would definitly be the conservative fix.

milton

#13 lore raw
Quoted 4 lines · Milton Miller

At least for parsing, we need to allocate and parse NR_CPUS until all places like arch/powerpc/platforms/pseries/xics.c that compare a user-supplied mask to CPUMASK_ALL are eliminated.

Good point. Anton will want to fix those anyway for CONFIG_CPUMASK_OFFSTACK, too, but that's the reason the parsing uses nr_cpumask_bits.

Quoted 5 lines
Quoted 3 lines

Would it make sense to use my initial patch for -stable, which reverts the ABI back to before the change that caused the problem, but apply the correct fix (changing the ABI throughout) for future releases?

This would definitly be the conservative fix.

Instead of changing back to NR_CPUS which will break libnuma for CPUMASK_OFFSTACK, how about changing it to nr_cpumask_bits and having an explicit comment above it:

	/* libnuma assumes we match scnprintf for /sys/.../node/node*/cpumap */

Thanks,
Rusty.

#14 lore raw
Quoted 13 lines · Milton Miller
Quoted 4 lines

At least for parsing, we need to allocate and parse NR_CPUS until all places like arch/powerpc/platforms/pseries/xics.c that compare a user-supplied mask to CPUMASK_ALL are eliminated.

Good point. Anton will want to fix those anyway for CONFIG_CPUMASK_OFFSTACK, too, but that's the reason the parsing uses nr_cpumask_bits.

Quoted 4 lines
Quoted 3 lines

Would it make sense to use my initial patch for -stable, which reverts the ABI back to before the change that caused the problem, but apply the correct fix (changing the ABI throughout) for future releases?

Quoted 5 lines
Quoted 1 line

This would definitly be the conservative fix.

Instead of changing back to NR_CPUS which will break libnuma for CPUMASK_OFFSTACK, how about changing it to nr_cpumask_bits and having an explicit comment above it:

Yes and No.

1) sched_getaffinity syscall is used from glibc and libnuma.
2) glibc doesn't use the return value almostly. glibc emulate it as NR_CPUS=1024.
3) Now, both sched_getaffinity() and compat_sys_sched_getaffinity() have nr_cpu_ids thing.
4) But only compat_sys_sched_getaffinity() hit libnuma problem.

I think It mean compat_sys_sched_getaffinity() should behave as sched_getaffinity(). IOW, libnuma assume compat_sys_sched_getaffinity() return len args or NR_CPUS. then, following patch do it. I confirmed the patch works with or without CPUMASK_OFFSTACK.

So, My proposal is
1) merge both mine and yours to linus tree
2) but backport only mine

How do you think?

==========================================================
From: KOSAKI Motohiro <[email protected]>
Subject: [PATCH] cpumask: fix compat getaffinity

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

On NUMA systems with less than 32 possibly CPUs, the current compat_sys_sched_getaffinity now returns '4' instead of the actual NR_CPUS/8, which makes libnuma bail out when parsing the cpumap.

The libnuma call sched_getaffinity(0, bitmap, 4096) at first. It mean the libnuma expect the return value of sched_getaffinity() is either len argument or NR_CPUS. But it doesn't expect to return nr_cpu_ids.

Strictly speaking, userland requirement are

1) Glibc assume the return value mean the lengh of initialized of mask argument. E.g. if sched_getaffinity(1024) return 128, glibc make zero fill rest 896 byte.
2) Libnuma assume the return value can be used to guess NR_CPUS in kernel. It assume len-arg<NR_CPUS makes -EINVAL. But it try len=4096 at first and 4096 is always bigger than NR_CPUS. Then, if we remove strange min_length normalization, we never hit -EINVAL case.

sched_getaffinity() already solved this issue. This patch adapt compat_sys_sched_getaffinity() as it.

---
 kernel/compat.c |   25 +++++++++++--------------
 1 files changed, 11 insertions(+), 14 deletions(-)
kernel/compat.c +11 -14
index 7f40e92..5adab05 100644
@@ -495,29 +495,26 @@ asmlinkage long compat_sys_sched_getaffinity(compat_pid_t pid, unsigned int len,
 {
 	int ret;
 	cpumask_var_t mask;
-	unsigned long *k;
-	unsigned int min_length = cpumask_size();
-
-	if (nr_cpu_ids <= BITS_PER_COMPAT_LONG)
-		min_length = sizeof(compat_ulong_t);
 
-	if (len < min_length)
+	if ((len * BITS_PER_BYTE) < nr_cpu_ids)
+		return -EINVAL;
+	if (len & (sizeof(compat_ulong_t)-1))
 		return -EINVAL;
 
 	if (!alloc_cpumask_var(&mask, GFP_KERNEL))
 		return -ENOMEM;
 
 	ret = sched_getaffinity(pid, mask);
-	if (ret < 0)
-		goto out;
+	if (ret == 0) {
+		size_t retlen = min_t(size_t, len, cpumask_size());
 
-	k = cpumask_bits(mask);
-	ret = compat_put_bitmap(user_mask_ptr, k, min_length * 8);
-	if (ret == 0)
-		ret = min_length;
-
-out:
+		if (compat_put_bitmap(user_mask_ptr, cpumask_bits(mask), retlen * 8))
+			ret = -EFAULT;
+		else
+			ret = retlen;
+	}
 	free_cpumask_var(mask);
+
 	return ret;
 }
 
Signature · 1 line
1.6.5.2
author Acked-by
#15 lore raw
Quoted 28 lines · KOSAKI Motohiro

From: KOSAKI Motohiro <[email protected]>
Subject: [PATCH] cpumask: fix compat getaffinity

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

On NUMA systems with less than 32 possibly CPUs, the current compat_sys_sched_getaffinity now returns '4' instead of the actual NR_CPUS/8, which makes libnuma bail out when parsing the cpumap.

The libnuma call sched_getaffinity(0, bitmap, 4096) at first. It mean the libnuma expect the return value of sched_getaffinity() is either len argument or NR_CPUS. But it doesn't expect to return nr_cpu_ids.

Strictly speaking, userland requirement are

1) Glibc assume the return value mean the lengh of initialized of mask argument. E.g. if sched_getaffinity(1024) return 128, glibc make zero fill rest 896 byte.
2) Libnuma assume the return value can be used to guess NR_CPUS in kernel. It assume len-arg<NR_CPUS makes -EINVAL. But it try len=4096 at first and 4096 is always bigger than NR_CPUS. Then, if we remove strange min_length normalization, we never hit -EINVAL case.

Quoted 4 lines

sched_getaffinity() already solved this issue. This patch adapt compat_sys_sched_getaffinity() as it.

Looks good to me,

You should also add a Cc:[email protected] line to the description so Greg's scripts catch it when it gets merged upstream.

Arnd

Acked-by
#16 lore raw
In reply to KOSAKI Motohiro
Quoted 26 lines · KOSAKI Motohiro
Quoted 13 lines · Milton Miller
Quoted 4 lines

At least for parsing, we need to allocate and parse NR_CPUS until all places like arch/powerpc/platforms/pseries/xics.c that compare a user-supplied mask to CPUMASK_ALL are eliminated.

Good point. Anton will want to fix those anyway for CONFIG_CPUMASK_OFFSTACK, too, but that's the reason the parsing uses nr_cpumask_bits.

Quoted 4 lines
Quoted 3 lines

Would it make sense to use my initial patch for -stable, which reverts the ABI back to before the change that caused the problem, but apply the correct fix (changing the ABI throughout) for future releases?

Quoted 5 lines
Quoted 1 line

This would definitly be the conservative fix.

Instead of changing back to NR_CPUS which will break libnuma for CPUMASK_OFFSTACK, how about changing it to nr_cpumask_bits and having an explicit comment above it:

Yes and No.

1) sched_getaffinity syscall is used from glibc and libnuma.
2) glibc doesn't use the return value almostly. glibc emulate it as NR_CPUS=1024.
3) Now, both sched_getaffinity() and compat_sys_sched_getaffinity() have nr_cpu_ids thing.
4) But only compat_sys_sched_getaffinity() hit libnuma problem.

Quoted 7 lines

I think It mean compat_sys_sched_getaffinity() should behave as sched_getaffinity(). IOW, libnuma assume compat_sys_sched_getaffinity() return len args or NR_CPUS. then, following patch do it. I confirmed the patch works with or without CPUMASK_OFFSTACK.

So, My proposal is
1) merge both mine and yours to linus tree
2) but backport only mine

I think we should just take yours; it seems sufficient.

We're going to break at > 4096 cpus whatever we do; I argued with Ingo about it when the affinity syscalls were added but noone thought we'd hit it so soon.

Thanks,
Rusty.

#17 lore raw
In reply to KOSAKI Motohiro

I realize this thread is fairly old, but I have a question about this statement in the changelog:

Quoted 4 lines

On NUMA systems with less than 32 possibly CPUs, the current compat_sys_sched_getaffinity now returns '4' instead of the actual NR_CPUS/8, which makes libnuma bail out when parsing the cpumap.

This does not seem accurate, at least not on my system with NR_CPUS=32 and nr_cpu_ids=8. The smallest value retlen will ever be set to is sizeof(long) which is 8 on most modern systems. This breaks the statement that this function will now return NR_CPUS/8.

Thanks
Josh

Quoted 135 lines · KOSAKI Motohiro
Quoted 13 lines · Milton Miller
Quoted 4 lines

At least for parsing, we need to allocate and parse NR_CPUS until all places like arch/powerpc/platforms/pseries/xics.c that compare a user-supplied mask to CPUMASK_ALL are eliminated.

Good point.  Anton will want to fix those anyway for CONFIG_CPUMASK_OFFSTACK, too, but that's the reason the parsing uses nr_cpumask_bits.

Quoted 4 lines
Quoted 3 lines

Would it make sense to use my initial patch for -stable, which reverts the ABI back to before the change that caused the problem, but apply the correct fix (changing the ABI throughout) for future releases?

Quoted 5 lines
Quoted 1 line

This would definitly be the conservative fix.

Instead of changing back to NR_CPUS which will break libnuma for CPUMASK_OFFSTACK, how about changing it to nr_cpumask_bits and having an explicit comment above it:

Yes and No.

1) sched_getaffinity syscall is used from glibc and libnuma.
2) glibc doesn't use the return value almostly. glibc emulate it as NR_CPUS=1024.
3) Now, both sched_getaffinity() and compat_sys_sched_getaffinity() have nr_cpu_ids thing.
4) But only compat_sys_sched_getaffinity() hit libnuma problem.

I think It mean compat_sys_sched_getaffinity() should behave as sched_getaffinity(). IOW, libnuma assume compat_sys_sched_getaffinity() return len args or NR_CPUS. then, following patch do it. I confirmed the patch works with or without CPUMASK_OFFSTACK.

So, My proposal is
       1) merge both mine and yours to linus tree
       2) but backport only mine

How do you think?

==========================================================
From: KOSAKI Motohiro <[email protected]>
Subject: [PATCH] cpumask: fix compat getaffinity

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

On NUMA systems with less than 32 possibly CPUs, the current compat_sys_sched_getaffinity now returns '4' instead of the actual NR_CPUS/8, which makes libnuma bail out when parsing the cpumap.

The libnuma call sched_getaffinity(0, bitmap, 4096) at first. It mean the libnuma expect the return value of sched_getaffinity() is either len argument or NR_CPUS. But it doesn't expect to return nr_cpu_ids.

Strictly speaking, userland requirement are

1) Glibc assume the return value mean the lengh of initialized of mask argument. E.g. if sched_getaffinity(1024) return 128, glibc make zero fill rest 896 byte.
2) Libnuma assume the return value can be used to guess NR_CPUS in kernel. It assume len-arg<NR_CPUS makes -EINVAL. But it try len=4096 at first and 4096 is always bigger than NR_CPUS. Then, if we remove strange min_length normalization, we never hit -EINVAL case.

sched_getaffinity() already solved this issue. This patch adapt compat_sys_sched_getaffinity() as it.

---
 kernel/compat.c |   25 +++++++++++--------------

 1 files changed, 11 insertions(+), 14 deletions(-)
kernel/compat.c +11 -15
index 7f40e92..5adab05 100644
@@ -495,29 +495,26 @@ asmlinkage long compat_sys_sched_getaffinity(compat_pid_t pid, unsigned int len,
  {
        int ret;
        cpumask_var_t mask;
-       unsigned long *k;
-       unsigned int min_length = cpumask_size();
-
-       if (nr_cpu_ids <= BITS_PER_COMPAT_LONG)
-               min_length = sizeof(compat_ulong_t);
 
-       if (len < min_length)
+       if ((len * BITS_PER_BYTE) < nr_cpu_ids)
+               return -EINVAL;
+       if (len & (sizeof(compat_ulong_t)-1))
                return -EINVAL;
 
        if (!alloc_cpumask_var(&mask, GFP_KERNEL))
                return -ENOMEM;
 
        ret = sched_getaffinity(pid, mask);
-       if (ret < 0)
-               goto out;
+       if (ret == 0) {
+               size_t retlen = min_t(size_t, len, cpumask_size());
 
-       k = cpumask_bits(mask);
-       ret = compat_put_bitmap(user_mask_ptr, k, min_length * 8);
-       if (ret == 0)
-               ret = min_length;
-
-out:
+               if (compat_put_bitmap(user_mask_ptr, cpumask_bits(mask), retlen * 8))
+                       ret = -EFAULT;
+               else
+                       ret = retlen;
+       }
        free_cpumask_var(mask);
+
        return ret;
  }
 
--

1.6.5.2

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

#18 lore raw
Quoted 9 lines · Josh Hunt

I realize this thread is fairly old, but I have a question about this statement in the changelog:

Quoted 4 lines

On NUMA systems with less than 32 possibly CPUs, the current compat_sys_sched_getaffinity now returns '4' instead of the actual NR_CPUS/8, which makes libnuma bail out when parsing the cpumap.

Quoted 4 lines

This does not seem accurate, at least not on my system with NR_CPUS=32 and nr_cpu_ids=8. The smallest value retlen will ever be set to is sizeof(long) which is 8 on most modern systems. This breaks the statement that this function will now return NR_CPUS/8.

Yes, I think the description is a bit misleading there. The point was that it should return the number that the sysfs files are based on, instead of the smallest multiple of sizeof(compat_long) that can hold the installed CPUs.

Arnd

#19 lore raw
In reply to Milton Miller
Quoted 4 lines · Milton Miller

At least for parsing, we need to allocate and parse NR_CPUS until all places like arch/powerpc/platforms/pseries/xics.c that compare a user-supplied mask to CPUMASK_ALL are eliminated.

Hi, this has been bugging me...

We're OK in this case: cpumask_equal() will only iterate to nr_cpus. That's one reason for all the cpumask churn...

I'm about to submit the cpumask shrinkage to Ingo. I'll cc you...

Cheers,
Rusty.

#20 lore raw
In reply to Rusty Russell
Quoted 11 lines · Rusty Russell

cpumask: use nr_cpu_ids for printing and parsing cpumasks

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

However, the result now returned reflects nr_cpu_ids, and cpumask_scnprintf et al. use nr_cpumask_bits which is NR_CPUS (for CONFIG_CPUMASK_OFFSTACK=n) or nr_cpu_ids (for CONFIG_CPUMASK_OFFSTACK=y).

Quoted 5 lines

We should use nr_cpu_ids consistently.

It looks like this changes fixes the compat case, but now the native system call would be inconsistent with the sysfs output.

Wouldn't you also need something like the change below to make the return value of sched_getaffinity() independent of NR_CPUS?

kernel/compat.c +4 -7
@@ -495,12 +495,9 @@ asmlinkage long compat_sys_sched_getaffinity(compat_pid_t pid, unsigned int len,
 	int ret;
 	cpumask_var_t mask;
 	unsigned long *k;
-	unsigned int min_length = cpumask_size();
+	size_t size = BITS_TO_COMPAT_LONGS(nr_cpu_ids) * sizeof(compat_ulong_t);
 
-	if (nr_cpu_ids <= BITS_PER_COMPAT_LONG)
-		min_length = sizeof(compat_ulong_t);
-
-	if (len < min_length)
+	if (len < size)
 		return -EINVAL;
 
 	if (!alloc_cpumask_var(&mask, GFP_KERNEL))
@@ -511,9 +508,9 @@ asmlinkage long compat_sys_sched_getaffinity(compat_pid_t pid, unsigned int len,
 		goto out;
 
 	k = cpumask_bits(mask);
-	ret = compat_put_bitmap(user_mask_ptr, k, min_length * 8);
+	ret = compat_put_bitmap(user_mask_ptr, k, size * 8);
 	if (ret == 0)
-		ret = min_length;
+		ret = size;
 
 out:
 	free_cpumask_var(mask);
kernel/sched.c +4 -3
@@ -6693,8 +6693,9 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
 {
 	int ret;
 	cpumask_var_t mask;
+	size_t size = BITS_TO_LONGS(nr_cpu_ids) * sizeof(long);
 
-	if (len < cpumask_size())
+	if (len < size)
 		return -EINVAL;
 
 	if (!alloc_cpumask_var(&mask, GFP_KERNEL))
@@ -6702,10 +6703,10 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
 
 	ret = sched_getaffinity(pid, mask);
 	if (ret == 0) {
-		if (copy_to_user(user_mask_ptr, mask, cpumask_size()))
+		if (copy_to_user(user_mask_ptr, mask, size))
 			ret = -EFAULT;
 		else
-			ret = cpumask_size();
+			ret = size;
 	}
 	free_cpumask_var(mask);
 
#21 lore raw
Quoted 22 lines · Rusty Russell
Quoted 11 lines

cpumask: use nr_cpu_ids for printing and parsing cpumasks

Commit a45185d2d "cpumask: convert kernel/compat.c" broke libnuma, which abuses sched_getaffinity to find out NR_CPUS in order to parse /sys/devices/system/node/node*/cpumap.

However, the result now returned reflects nr_cpu_ids, and cpumask_scnprintf et al. use nr_cpumask_bits which is NR_CPUS (for CONFIG_CPUMASK_OFFSTACK=n) or nr_cpu_ids (for CONFIG_CPUMASK_OFFSTACK=y).

Quoted 5 lines

We should use nr_cpu_ids consistently.

It looks like this changes fixes the compat case, but now the native system call would be inconsistent with the sysfs output.

Quoted 5 lines

Wouldn't you also need something like the change below to make the return value of sched_getaffinity() independent of NR_CPUS?

Looks good. but please rebase it on latest linus-tree. in linus tree, we are already using nr_cpu_ids instead NR_CPUS.

And I'm sorry. at making such change, I've forgot to care compat syscall, your patch fixes it ;)