Skip to contentSkip to conversation

sched: sched_getaffinity() allow less than NR_CPUS length

PATCH In review KOSAKI Motohiro 4 participants 8 messages Mar 12, 2010 – Mar 17, 2010

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sched: sched_getaffinity() allow less than NR_CPUS length

Recently, some distro decided to use NR_CPUS=4096 for mysterious reason. Unfortunately, glibc sched interface have following definition.

	# define __CPU_SETSIZE  1024
	# define __NCPUBITS     (8 * sizeof (__cpu_mask))
	typedef unsigned long int __cpu_mask;
	typedef struct
	{
	  __cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
	} cpu_set_t;

It mean, if NR_CPUS is bigger than 1024, cpu_set_t can makes ABI issue.

More recently, Sharyathi Nagesh reported following test program makes misterious syscall failure.

-----------------------------------------------------------------------
#define _GNU_SOURCE
#include<stdio.h>
#include<errno.h>
#include<sched.h>

int main()

{
     cpu_set_t set;
     if (sched_getaffinity(0, sizeof(cpu_set_t), &set) < 0)
         printf("\n Call is failing with:%d", errno);
}

-----------------------------------------------------------------------

Because the kernel assue len argument of sched_getaffinity() is bigger than NR_CPUS. but now it is not correct.

Now we faced annoying dilemma.
(1) if we change glibc's __CPU_SETSIZE definition, we lost binary compatibility of _all_ application.
(2) if we don't change it, we also lost binary compatibility of Sharyathi's use case.

Then, I would propse to change the rule of the len argument of sched_getaffinity().

Old:
len should be bigger than NR_CPUS
New:
len should be bigger than maximum possible cpu id

It makes following behavior
(A) In the real 4096 cpus machine, the above test program still return -EINVAL.
(B) NR_CPUS=4096 but the machine have less than 1024 cpus (almost machines in the world), the above can run successfully.

Fortunatelly, BIG SGI machine is mainly used for HPC use case. It mean they can rewrite their program by themself. IOW we can think they don't annoy this issue.

Cc Ulrich Drepper <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/sched.c |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)
kernel/sched.c +7 -3
index 150b698..34217f1 100644
@@ -4902,7 +4902,9 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
 	int ret;
 	cpumask_var_t mask;
 
-	if (len < cpumask_size())
+	if (len < nr_cpu_ids)
+		return -EINVAL;
+	if (len & (sizeof(unsigned long)-1))
 		return -EINVAL;
 
 	if (!alloc_cpumask_var(&mask, GFP_KERNEL))
@@ -4910,10 +4912,12 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
 
 	ret = sched_getaffinity(pid, mask);
 	if (ret == 0) {
-		if (copy_to_user(user_mask_ptr, mask, cpumask_size()))
+		int retlen = min(len, cpumask_size());
+
+		if (copy_to_user(user_mask_ptr, mask, retlen))
 			ret = -EFAULT;
 		else
-			ret = cpumask_size();
+			ret = retlen;
 	}
 	free_cpumask_var(mask);
 
Signature · 1 line
1.6.5.2
Acked-by on v1 1/1
#1 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines · KOSAKI Motohiro
Cc Ulrich Drepper <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 kernel/sched.c |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)

If this isn't creating problems with hotplug CPUs:

- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖ -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAkuaZukACgkQ2ijCOnn/RHRu9ACeNyLYw8q4ltCKxVOZme2at4tm 6DwAoJ6wz/weZnO5T3GJag/XsnfytCTu
=k2Dv
-----END PGP SIGNATURE-----

#2 lore raw

[tip:sched/urgent] sched: sched_getaffinity(): Allow less than NR_CPUS length

In reply to KOSAKI Motohiro
Commit-ID:  cd3d8031eb4311e516329aee03c79a08333141f1
Gitweb:     http://git.kernel.org/tip/cd3d8031eb4311e516329aee03c79a08333141f1
Author:     KOSAKI Motohiro <[email protected]>
AuthorDate: Fri, 12 Mar 2010 16:15:36 +0900
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 15 Mar 2010 08:28:44 +0100

sched: sched_getaffinity(): Allow less than NR_CPUS length

[ Note, this commit changes the syscall ABI for > 1024 CPUs systems. ]

Recently, some distro decided to use NR_CPUS=4096 for mysterious reasons. Unfortunately, glibc sched interface has the following definition:

	# define __CPU_SETSIZE  1024
	# define __NCPUBITS     (8 * sizeof (__cpu_mask))
	typedef unsigned long int __cpu_mask;
	typedef struct
	{
	  __cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
	} cpu_set_t;

It mean, if NR_CPUS is bigger than 1024, cpu_set_t makes an ABI issue ...

More recently, Sharyathi Nagesh reported following test program makes misterious syscall failure:

-----------------------------------------------------------------------
#define _GNU_SOURCE
#include<stdio.h>
#include<errno.h>
#include<sched.h>

int main()

 {
     cpu_set_t set;
     if (sched_getaffinity(0, sizeof(cpu_set_t), &set) < 0)
         printf("\n Call is failing with:%d", errno);

}
-----------------------------------------------------------------------

Because the kernel assumes len argument of sched_getaffinity() is bigger than NR_CPUS. But now it is not correct.

Now we are faced with the following annoying dilemma, due to the limitations of the glibc interface built in years ago:

(1) if we change glibc's __CPU_SETSIZE definition, we lost binary compatibility of _all_ application.

(2) if we don't change it, we also lost binary compatibility of Sharyathi's use case.

Then, I would propse to change the rule of the len argument of sched_getaffinity().

Old:
len should be bigger than NR_CPUS
New:
len should be bigger than maximum possible cpu id

This creates the following behavior:

(A) In the real 4096 cpus machine, the above test program still return -EINVAL.

(B) NR_CPUS=4096 but the machine have less than 1024 cpus (almost all machines in the world), the above can run successfully.

Fortunatelly, BIG SGI machine is mainly used for HPC use case. It means they can rebuild their programs.

IOW we hope they are not annoyed by this issue ...

LKML-Reference: <[email protected]>

---
 kernel/sched.c |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)
kernel/sched.c +7 -3
index 9ab3cd7..6eaef3d 100644
@@ -4902,7 +4902,9 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
 	int ret;
 	cpumask_var_t mask;
 
-	if (len < cpumask_size())
+	if (len < nr_cpu_ids)
+		return -EINVAL;
+	if (len & (sizeof(unsigned long)-1))
 		return -EINVAL;
 
 	if (!alloc_cpumask_var(&mask, GFP_KERNEL))
@@ -4910,10 +4912,12 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
 
 	ret = sched_getaffinity(pid, mask);
 	if (ret == 0) {
-		if (copy_to_user(user_mask_ptr, mask, cpumask_size()))
+		int retlen = min(len, cpumask_size());
+
+		if (copy_to_user(user_mask_ptr, mask, retlen))
 			ret = -EFAULT;
 		else
-			ret = cpumask_size();
+			ret = retlen;
 	}
 	free_cpumask_var(mask);
 
#3 lore raw

Re: [tip:sched/urgent] sched: sched_getaffinity(): Allow less than NR_CPUS length

Quoted 6 lines · tip-bot for KOSAKI Motohiro
Commit-ID:  cd3d8031eb4311e516329aee03c79a08333141f1
Gitweb:     http://git.kernel.org/tip/cd3d8031eb4311e516329aee03c79a08333141f1
Author:     KOSAKI Motohiro <[email protected]>
AuthorDate: Fri, 12 Mar 2010 16:15:36 +0900
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 15 Mar 2010 08:28:44 +0100

...

Quoted 1 line

IOW we hope they are not annoyed by this issue ...

The change looks ok but I can't reproduce the problem.

I'm running on a distro kernel that has NR_CPUS=4096. Glibc has also has a definition of __CPU_SETSIZE (I assume this change was made by the distro but am not certain):

    sched.c
    	...
	#if defined _SCHED_H && !defined __cpu_set_t_defined
	# define __cpu_set_t_defined
	/* Size definition for CPU sets.  */
	# define __CPU_SETSIZE  4096
	# define __NCPUBITS     (8 * sizeof (__cpu_mask))

Your test program runs ok:

	% strace t
	...
	sched_getaffinity(0, 512,  { ffff, 0, 0, 0, 0, 0, 0, 0 }) = 64

Also note that we've run on IA64 systems with NR_CPUS=4096 for several years w/o hitting any problems.

Bottom line. I don't think the change will affect us.

Quoted 114 lines

sched: sched_getaffinity(): Allow less than NR_CPUS length

[ Note, this commit changes the syscall ABI for > 1024 CPUs systems. ]

Recently, some distro decided to use NR_CPUS=4096 for mysterious reasons. Unfortunately, glibc sched interface has the following definition:

	# define __CPU_SETSIZE  1024
	# define __NCPUBITS     (8 * sizeof (__cpu_mask))
	typedef unsigned long int __cpu_mask;
	typedef struct
	{
	  __cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
	} cpu_set_t;

It mean, if NR_CPUS is bigger than 1024, cpu_set_t makes an ABI issue ...

More recently, Sharyathi Nagesh reported following test program makes misterious syscall failure:

-----------------------------------------------------------------------
#define _GNU_SOURCE
#include<stdio.h>
#include<errno.h>
#include<sched.h>

int main()

 {
     cpu_set_t set;
     if (sched_getaffinity(0, sizeof(cpu_set_t), &set) < 0)
         printf("\n Call is failing with:%d", errno);

}
-----------------------------------------------------------------------

Because the kernel assumes len argument of sched_getaffinity() is bigger than NR_CPUS. But now it is not correct.

Now we are faced with the following annoying dilemma, due to the limitations of the glibc interface built in years ago:

(1) if we change glibc's __CPU_SETSIZE definition, we lost binary compatibility of _all_ application.

(2) if we don't change it, we also lost binary compatibility of Sharyathi's use case.

Then, I would propse to change the rule of the len argument of sched_getaffinity().

Old:
len should be bigger than NR_CPUS
New:
len should be bigger than maximum possible cpu id

This creates the following behavior:

(A) In the real 4096 cpus machine, the above test program still return -EINVAL.

(B) NR_CPUS=4096 but the machine have less than 1024 cpus (almost all machines in the world), the above can run successfully.

Fortunatelly, BIG SGI machine is mainly used for HPC use case. It means they can rebuild their programs.

IOW we hope they are not annoyed by this issue ...

LKML-Reference: <[email protected]>

---
 kernel/sched.c |   10 +++++++---
 1 files changed, 7 insertions(+), 3 deletions(-)
kernel/sched.c +7 -3
index 9ab3cd7..6eaef3d 100644
@@ -4902,7 +4902,9 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
 	int ret;
 	cpumask_var_t mask;
 
-	if (len < cpumask_size())
+	if (len < nr_cpu_ids)
+		return -EINVAL;
+	if (len & (sizeof(unsigned long)-1))
 		return -EINVAL;
 
 	if (!alloc_cpumask_var(&mask, GFP_KERNEL))
@@ -4910,10 +4912,12 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
 
 	ret = sched_getaffinity(pid, mask);
 	if (ret == 0) {
-		if (copy_to_user(user_mask_ptr, mask, cpumask_size()))
+		int retlen = min(len, cpumask_size());
+
+		if (copy_to_user(user_mask_ptr, mask, retlen))
 			ret = -EFAULT;
 		else
-			ret = cpumask_size();
+			ret = retlen;
 	}
 	free_cpumask_var(mask);
 
#4 lore raw

Re: [tip:sched/urgent] sched: sched_getaffinity(): Allow less than NR_CPUS length

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 3 lines · Jack Steiner

I'm running on a distro kernel that has NR_CPUS=4096. Glibc has also has a definition of __CPU_SETSIZE (I assume this change was made by the distro but am not certain):

This isn't upstream and it cannot. Whoever make this change doesn't know what binary compatibility means. The size has always been 1024 bits.

- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖ -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/

iEYEARECAAYFAkueXYIACgkQ2ijCOnn/RHQJPQCbBvh+Mi7ZrHaQcVxEkglIj+Pz j7UAoJaBCpD2a5KbAR1VGkM7orp+GBhY
=T78A
-----END PGP SIGNATURE-----

#5 lore raw

Re: [tip:sched/urgent] sched: sched_getaffinity(): Allow less than NR_CPUS length

In reply to tip-bot for KOSAKI Motohiro

kernel/sched.c SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len, · line 4915

Quoted 10 lines · tip-bot for KOSAKI Motohiro
Commit-ID:  cd3d8031eb4311e516329aee03c79a08333141f1
Gitweb:     http://git.kernel.org/tip/cd3d8031eb4311e516329aee03c79a08333141f1
Author:     KOSAKI Motohiro <[email protected]>
AuthorDate: Fri, 12 Mar 2010 16:15:36 +0900
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 15 Mar 2010 08:28:44 +0100

sched: sched_getaffinity(): Allow less than NR_CPUS length

kernel/sched.c +1 -0
+		int retlen = min(len, cpumask_size());

this one causes:

kernel/sched.c:4850: warning: comparison of distinct pointer types lacks a cast

Ingo

#6 lore raw

Re: [tip:sched/urgent] sched: sched_getaffinity(): Allow less than NR_CPUS length

Quoted 10 lines · tip-bot for KOSAKI Motohiro
Quoted 8 lines
Commit-ID:  cd3d8031eb4311e516329aee03c79a08333141f1
Gitweb:     http://git.kernel.org/tip/cd3d8031eb4311e516329aee03c79a08333141f1
Author:     KOSAKI Motohiro <[email protected]>
AuthorDate: Fri, 12 Mar 2010 16:15:36 +0900
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 15 Mar 2010 08:28:44 +0100
Quoted 7 lines

Review

Quoted 3 lines

sched: sched_getaffinity(): Allow less than NR_CPUS length

(quoted diff) +1 -0
+		int retlen = min(len, cpumask_size());

this one causes:

kernel/sched.c:4850: warning: comparison of distinct pointer types lacks a cast

Ahh, I'm sorry. attached fixing patch.

thanks.

============================
Subject: [PATCH] fix warning

fix following warning.

kernel/sched.c:4850: warning: comparison of distinct pointer types lacks a cast

---
 kernel/sched.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
index 34217f1..d1d5acd 100644
@@ -4912,7 +4912,7 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
 
 	ret = sched_getaffinity(pid, mask);
 	if (ret == 0) {
-		int retlen = min(len, cpumask_size());
+		size_t retlen = min_t(size_t, len, cpumask_size());
 
 		if (copy_to_user(user_mask_ptr, mask, retlen))
 			ret = -EFAULT;
Signature · 1 line
1.6.5.2
#7 lore raw

[tip:sched/urgent] sched: Use proper type in sched_getaffinity()

Commit-ID:  8bc037fb89bb3104b9ae290d18c877624cd7d9cc
Gitweb:     http://git.kernel.org/tip/8bc037fb89bb3104b9ae290d18c877624cd7d9cc
Author:     KOSAKI Motohiro <[email protected]>
AuthorDate: Wed, 17 Mar 2010 09:36:58 +0900
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 17 Mar 2010 10:48:49 +0100

sched: Use proper type in sched_getaffinity()

Using the proper type fixes the following compiler warning:

kernel/sched.c:4850: warning: comparison of distinct pointer types lacks a cast

LKML-Reference: <[email protected]>

---
 kernel/sched.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
index 82975b5..49d2fa7 100644
@@ -4912,7 +4912,7 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
 
 	ret = sched_getaffinity(pid, mask);
 	if (ret == 0) {
-		int retlen = min(len, cpumask_size());
+		size_t retlen = min_t(size_t, len, cpumask_size());
 
 		if (copy_to_user(user_mask_ptr, mask, retlen))
 			ret = -EFAULT;