sched: sched_getaffinity() allow less than NR_CPUS length
PATCH 1/1 sched: sched_getaffinity() allow less than NR_CPUS length
Recently, some distro decided to use NR_CPUS=4096 for mysterious reason. Unfortunately, glibc sched interface have following definition.
# define __CPU_SETSIZE 1024
# define __NCPUBITS (8 * sizeof (__cpu_mask))
typedef unsigned long int __cpu_mask;
typedef struct
{
__cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
} cpu_set_t;It mean, if NR_CPUS is bigger than 1024, cpu_set_t can makes ABI issue.
More recently, Sharyathi Nagesh reported following test program makes misterious syscall failure.
-----------------------------------------------------------------------
#define _GNU_SOURCE
#include<stdio.h>
#include<errno.h>
#include<sched.h>
int main()
{
cpu_set_t set;
if (sched_getaffinity(0, sizeof(cpu_set_t), &set) < 0)
printf("\n Call is failing with:%d", errno);
}-----------------------------------------------------------------------
Because the kernel assue len argument of sched_getaffinity() is bigger than NR_CPUS. but now it is not correct.
Now we faced annoying dilemma.
(1) if we change glibc's __CPU_SETSIZE definition, we lost binary compatibility of _all_ application.
(2) if we don't change it, we also lost binary compatibility of Sharyathi's use case.
Then, I would propse to change the rule of the len argument of sched_getaffinity().
Old:
len should be bigger than NR_CPUS
New:
len should be bigger than maximum possible cpu id
It makes following behavior
(A) In the real 4096 cpus machine, the above test program still return -EINVAL.
(B) NR_CPUS=4096 but the machine have less than 1024 cpus (almost machines in the world), the above can run successfully.
Fortunatelly, BIG SGI machine is mainly used for HPC use case. It mean they can rewrite their program by themself. IOW we can think they don't annoy this issue.
--- kernel/sched.c | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-)
kernel/sched.c +7 -3
@@ -4902,7 +4902,9 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,int ret;cpumask_var_t mask; -if (len <cpumask_size ()) +if (len < nr_cpu_ids) +return -EINVAL ; +if (len & (sizeof (unsigned long )-1 ))return -EINVAL ;if (!alloc_cpumask_var (&mask,GFP_KERNEL ))
@@ -4910,10 +4912,12 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len, ret =sched_getaffinity (pid, mask);if (ret ==0 ) { -if (copy_to_user (user_mask_ptr, mask,cpumask_size ())) +int retlen =min (len,cpumask_size ()); + +if (copy_to_user (user_mask_ptr, mask, retlen)) ret = -EFAULT ;else - ret =cpumask_size (); + ret = retlen; }free_cpumask_var (mask);
Signature · 1 line
1.6.5.2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 9 lines · KOSAKI Motohiro
Reported-by Sharyathi Nagesh <[email protected]>Signed-off-by KOSAKI Motohiro <[email protected]>Cc Ulrich Drepper <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Ingo Molnar <[email protected]>--- kernel/sched.c | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-)
If this isn't creating problems with hotplug CPUs:
- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖ -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEARECAAYFAkuaZukACgkQ2ijCOnn/RHRu9ACeNyLYw8q4ltCKxVOZme2at4tm 6DwAoJ6wz/weZnO5T3GJag/XsnfytCTu
=k2Dv
-----END PGP SIGNATURE-----
[tip:sched/urgent] sched: sched_getaffinity(): Allow less than NR_CPUS length
In reply to KOSAKI MotohiroCommit-ID: cd3d8031eb4311e516329aee03c79a08333141f1 Gitweb: http://git.kernel.org/tip/cd3d8031eb4311e516329aee03c79a08333141f1 Author: KOSAKI Motohiro <[email protected]> AuthorDate: Fri, 12 Mar 2010 16:15:36 +0900 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 15 Mar 2010 08:28:44 +0100
sched: sched_getaffinity(): Allow less than NR_CPUS length
[ Note, this commit changes the syscall ABI for > 1024 CPUs systems. ]
Recently, some distro decided to use NR_CPUS=4096 for mysterious reasons. Unfortunately, glibc sched interface has the following definition:
# define __CPU_SETSIZE 1024
# define __NCPUBITS (8 * sizeof (__cpu_mask))
typedef unsigned long int __cpu_mask;
typedef struct
{
__cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
} cpu_set_t;It mean, if NR_CPUS is bigger than 1024, cpu_set_t makes an ABI issue ...
More recently, Sharyathi Nagesh reported following test program makes misterious syscall failure:
-----------------------------------------------------------------------
#define _GNU_SOURCE
#include<stdio.h>
#include<errno.h>
#include<sched.h>
int main()
{
cpu_set_t set;
if (sched_getaffinity(0, sizeof(cpu_set_t), &set) < 0)
printf("\n Call is failing with:%d", errno); }
-----------------------------------------------------------------------
Because the kernel assumes len argument of sched_getaffinity() is bigger than NR_CPUS. But now it is not correct.
Now we are faced with the following annoying dilemma, due to the limitations of the glibc interface built in years ago:
(1) if we change glibc's __CPU_SETSIZE definition, we lost binary compatibility of _all_ application.
(2) if we don't change it, we also lost binary compatibility of Sharyathi's use case.
Then, I would propse to change the rule of the len argument of sched_getaffinity().
Old:
len should be bigger than NR_CPUS
New:
len should be bigger than maximum possible cpu id
This creates the following behavior:
(A) In the real 4096 cpus machine, the above test program still return -EINVAL.
(B) NR_CPUS=4096 but the machine have less than 1024 cpus (almost all machines in the world), the above can run successfully.
Fortunatelly, BIG SGI machine is mainly used for HPC use case. It means they can rebuild their programs.
IOW we hope they are not annoyed by this issue ...
LKML-Reference: <[email protected]>
--- kernel/sched.c | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-)
kernel/sched.c +7 -3
@@ -4902,7 +4902,9 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,int ret;cpumask_var_t mask; -if (len <cpumask_size ()) +if (len < nr_cpu_ids) +return -EINVAL ; +if (len & (sizeof (unsigned long )-1 ))return -EINVAL ;if (!alloc_cpumask_var (&mask,GFP_KERNEL ))
@@ -4910,10 +4912,12 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len, ret =sched_getaffinity (pid, mask);if (ret ==0 ) { -if (copy_to_user (user_mask_ptr, mask,cpumask_size ())) +int retlen =min (len,cpumask_size ()); + +if (copy_to_user (user_mask_ptr, mask, retlen)) ret = -EFAULT ;else - ret =cpumask_size (); + ret = retlen; }free_cpumask_var (mask);
Re: [tip:sched/urgent] sched: sched_getaffinity(): Allow less than NR_CPUS length
Quoted 6 lines · tip-bot for KOSAKI Motohiro
Commit-ID: cd3d8031eb4311e516329aee03c79a08333141f1 Gitweb: http://git.kernel.org/tip/cd3d8031eb4311e516329aee03c79a08333141f1 Author: KOSAKI Motohiro <[email protected]> AuthorDate: Fri, 12 Mar 2010 16:15:36 +0900 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 15 Mar 2010 08:28:44 +0100
...
Quoted 1 line
IOW we hope they are not annoyed by this issue ...
The change looks ok but I can't reproduce the problem.
I'm running on a distro kernel that has NR_CPUS=4096. Glibc has also has a definition of __CPU_SETSIZE (I assume this change was made by the distro but am not certain):
sched.c
...
#if defined _SCHED_H && !defined __cpu_set_t_defined
# define __cpu_set_t_defined
/* Size definition for CPU sets. */
# define __CPU_SETSIZE 4096
# define __NCPUBITS (8 * sizeof (__cpu_mask))Your test program runs ok:
% strace t
...
sched_getaffinity(0, 512, { ffff, 0, 0, 0, 0, 0, 0, 0 }) = 64Also note that we've run on IA64 systems with NR_CPUS=4096 for several years w/o hitting any problems.
Bottom line. I don't think the change will affect us.
Quoted 114 lines
sched:
sched_getaffinity(): Allow less thanNR_CPUSlength[ Note, this commit changes the syscall ABI for > 1024 CPUs systems. ]
Recently, some distro decided to use NR_CPUS=4096 for mysterious reasons. Unfortunately, glibc sched interface has the following definition:
# define __CPU_SETSIZE 1024 # define __NCPUBITS (8 * sizeof (__cpu_mask)) typedef unsigned long int __cpu_mask; typedef struct { __cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS]; } cpu_set_t;It mean, if
NR_CPUSis bigger than 1024,cpu_set_tmakes an ABI issue ...More recently, Sharyathi Nagesh reported following test program makes misterious syscall failure:
-----------------------------------------------------------------------
#define_GNU_SOURCE
#include<stdio.h>
#include<errno.h>
#include<sched.h>int
main(){ cpu_set_t set; if (sched_getaffinity(0, sizeof(cpu_set_t), &set) < 0) printf("\n Call is failing with:%d", errno);}
-----------------------------------------------------------------------Because the kernel assumes len argument of
sched_getaffinity()is bigger thanNR_CPUS. But now it is not correct.Now we are faced with the following annoying dilemma, due to the limitations of the glibc interface built in years ago:
(1) if we change glibc's
__CPU_SETSIZEdefinition, we lost binary compatibility of _all_ application.(2) if we don't change it, we also lost binary compatibility of Sharyathi's use case.
Then, I would propse to change the rule of the len argument of
sched_getaffinity().Old:
len should be bigger thanNR_CPUS
New:
len should be bigger than maximum possible cpu idThis creates the following behavior:
(A) In the real 4096 cpus machine, the above test program still return
-EINVAL.(B) NR_CPUS=4096 but the machine have less than 1024 cpus (almost all machines in the world), the above can run successfully.
Fortunatelly, BIG SGI machine is mainly used for HPC use case. It means they can rebuild their programs.
IOW we hope they are not annoyed by this issue ...
Reported-by Sharyathi Nagesh <[email protected]>Signed-off-by KOSAKI Motohiro <[email protected]>Acked-by Ulrich Drepper <[email protected]>Acked-by Peter Zijlstra <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>Cc Jack Steiner <[email protected]>Cc Russ Anderson <[email protected]>Cc Mike Travis <[email protected]>LKML-Reference: <[email protected]>
Signed-off-by Ingo Molnar <[email protected]>--- kernel/sched.c | 10 +++++++--- 1 files changed, 7 insertions(+), 3 deletions(-)kernel/sched.c +7
-3@@ -4902,7 +4902,9 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,int ret;cpumask_var_t mask; -if (len <cpumask_size ()) +if (len < nr_cpu_ids) +return -EINVAL ; +if (len & (sizeof (unsigned long )-1 ))return -EINVAL ;if (!alloc_cpumask_var (&mask,GFP_KERNEL ))@@ -4910,10 +4912,12 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len, ret =sched_getaffinity (pid, mask);if (ret ==0 ) { -if (copy_to_user (user_mask_ptr, mask,cpumask_size ())) +int retlen =min (len,cpumask_size ()); + +if (copy_to_user (user_mask_ptr, mask, retlen)) ret = -EFAULT ;else - ret =cpumask_size (); + ret = retlen; }free_cpumask_var (mask);
Re: [tip:sched/urgent] sched: sched_getaffinity(): Allow less than NR_CPUS length
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 3 lines · Jack Steiner
I'm running on a distro kernel that has NR_CPUS=4096. Glibc has also has a definition of
__CPU_SETSIZE(I assume this change was made by the distro but am not certain):
This isn't upstream and it cannot. Whoever make this change doesn't know what binary compatibility means. The size has always been 1024 bits.
- --
➧ Ulrich Drepper ➧ Red Hat, Inc. ➧ 444 Castro St ➧ Mountain View, CA ❖ -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAkueXYIACgkQ2ijCOnn/RHQJPQCbBvh+Mi7ZrHaQcVxEkglIj+Pz j7UAoJaBCpD2a5KbAR1VGkM7orp+GBhY
=T78A
-----END PGP SIGNATURE-----
Re: [tip:sched/urgent] sched: sched_getaffinity(): Allow less than NR_CPUS length
In reply to tip-bot for KOSAKI Motohirokernel/sched.c SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len, · line 4915
Quoted 10 lines · tip-bot for KOSAKI Motohiro
Commit-ID: cd3d8031eb4311e516329aee03c79a08333141f1 Gitweb: http://git.kernel.org/tip/cd3d8031eb4311e516329aee03c79a08333141f1 Author: KOSAKI Motohiro <[email protected]> AuthorDate: Fri, 12 Mar 2010 16:15:36 +0900 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 15 Mar 2010 08:28:44 +0100sched:
sched_getaffinity(): Allow less thanNR_CPUSlengthkernel/sched.c +1
-0+int retlen =min (len,cpumask_size ());
this one causes:
kernel/sched.c:4850: warning: comparison of distinct pointer types lacks a cast
Ingo
Re: [tip:sched/urgent] sched: sched_getaffinity(): Allow less than NR_CPUS length
Quoted 10 lines · tip-bot for KOSAKI Motohiro
Quoted 8 lines
Commit-ID: cd3d8031eb4311e516329aee03c79a08333141f1 Gitweb: http://git.kernel.org/tip/cd3d8031eb4311e516329aee03c79a08333141f1 Author: KOSAKI Motohiro <[email protected]> AuthorDate: Fri, 12 Mar 2010 16:15:36 +0900 Committer: Ingo Molnar <[email protected]> CommitDate: Mon, 15 Mar 2010 08:28:44 +0100
Quoted 7 lines
Review
Quoted 3 lines
sched:
sched_getaffinity(): Allow less thanNR_CPUSlength(quoted diff) +1
-0+ int retlen = min(len, cpumask_size());this one causes:
kernel/sched.c:4850: warning: comparison of distinct pointer types lacks a cast
Ahh, I'm sorry. attached fixing patch.
thanks.
============================
Subject: [PATCH] fix warning
fix following warning.
kernel/sched.c:4850: warning: comparison of distinct pointer types lacks a cast
--- kernel/sched.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
@@ -4912,7 +4912,7 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len, ret =sched_getaffinity (pid, mask);if (ret ==0 ) { -int retlen =min (len,cpumask_size ()); +size_t retlen =min_t (size_t, len,cpumask_size ());if (copy_to_user (user_mask_ptr, mask, retlen)) ret = -EFAULT ;
Signature · 1 line
1.6.5.2
[tip:sched/urgent] sched: Use proper type in sched_getaffinity()
Commit-ID: 8bc037fb89bb3104b9ae290d18c877624cd7d9cc Gitweb: http://git.kernel.org/tip/8bc037fb89bb3104b9ae290d18c877624cd7d9cc Author: KOSAKI Motohiro <[email protected]> AuthorDate: Wed, 17 Mar 2010 09:36:58 +0900 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 17 Mar 2010 10:48:49 +0100
sched: Use proper type in sched_getaffinity()
Using the proper type fixes the following compiler warning:
kernel/sched.c:4850: warning: comparison of distinct pointer types lacks a cast
LKML-Reference: <[email protected]>
--- kernel/sched.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
kernel/sched.c +1 -1
@@ -4912,7 +4912,7 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len, ret =sched_getaffinity (pid, mask);if (ret ==0 ) { -int retlen =min (len,cpumask_size ()); +size_t retlen =min_t (size_t, len,cpumask_size ());if (copy_to_user (user_mask_ptr, mask, retlen)) ret = -EFAULT ;