Segmentation fault when running sched_setaffinity01 in RHEL5.10GA

LTP Xiaoguang Wang 3 participants 6 messages Jul 25, 2014

lore

Share & tools

lore

Xiaoguang Wang

author

2014-07-25 05:38 UTC #0 lore raw

Hi,

When we run sched_setaffinity01 in RHEL5.10GA, it occurs a segmentation fault. Below is the possible reason.

Hi Jan, would you please help to confirm this problem. I'm afraid RHEL5.10GA is an old distribution, which many people may not use it now, thanks!

Glibc provides a encapsulation for the raw kernel sched_setaffinity(2) system call, the corresponding code is below(The version of glibc I used is glibc-2.5-20061008T1257-RHEL5.11Beta):
I delete some code just for simple.

#######################################################################################

/* Size definition for CPU sets. */

# define __CPU_SETSIZE  1024
# define __NCPUBITS     (8 * sizeof (__cpu_mask))

/* Type for array elements in 'cpu_set'. */
typedef unsigned long int __cpu_mask;

/* Basic access functions. */

# define __CPUELT(cpu)  ((cpu) / __NCPUBITS)

/* Data structure to describe CPU mask. */
typedef struct
{
__cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
} cpu_set_t;

int __sched_setaffinity_new (pid_t pid, size_t cpusetsize, const cpu_set_t *cpuset) {
if (__builtin_expect (__kernel_cpumask_size == 0, 0))

    {
      int res;

      while (res = INTERNAL_SYSCALL (sched_getaffinity, err, 3, getpid (),
                                     psize, p),
             INTERNAL_SYSCALL_ERROR_P (res, err)
             && INTERNAL_SYSCALL_ERRNO (res, err) == EINVAL)
     	....

      __kernel_cpumask_size = res;
    }

/* We now know the size of the kernel cpumask_t. Make sure the user
does not request to set a bit beyond that. */
for (size_t cnt = __kernel_cpumask_size; cnt < cpusetsize; ++cnt)

    if (((char *) cpuset)[cnt] != '\0')
      {
        /* Found a nonzero byte.  This means the user request cannot be
           fulfilled.  */
        __set_errno (EINVAL);
        return -1;
      }

return INLINE_SYSCALL (sched_setaffinity, 3, pid, cpusetsize, cpuset);
}
#######################################################################################

Glibc in RHEL5.10GA does not provide CPU_ALLOC_SIZE marco, so in ltp testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h, we define one.
#######################################################################################
#ifndef CPU_ALLOC_SIZE
#define CPU_ALLOC_SIZE(size) sizeof(cpu_set_t)
#endif
#######################################################################################

Then CPU_ALLOC_SIZE would always return 128 in RHEL5.10GA, that is when we test EFAULT for sched_setaffinity(2), the passed cpusetsize is 128. But look at __sched_setaffinity_new() above, it first call
raw sched_getaffinity(2) to get the size of the kernel cpumask_t, In RHEL5.10GA,
this value depends on CONFIG_NR_CPUS, if CONFIG_NR_CPUS is 255, the raw sched_getaffinity(2) will return 32. In this case, __kernel_cpumask_size would be 32, cpusetsize is 128. Give that we're testing EFAULT, cpuset is a invalid pointer, if cnt > 32, it will generate segmentation fault in glibc code, so this case exits abnormally

As why this test case can run normally in RHEL6.5GA or RHEL7.0GA, it's because sched_getaffinity(2) in old kernel(RHEL5.10GA) return sizeof(cpumask_t), which totally depends on CONFIG_NR_CPUS. In newer kernel, sched_getaffinity(2) returns the smaller one between min_t(size_t, len, cpumask_size()), here len is the value passed to sched_getaffinity as cpusetsize, cpumask_size() is the max allowed length. so we can ensure __kernel_cpumask_size will never smaller cpusetsize, so the segmentation fault won't occur.

So I also think CPU_ALLOC and CPU_ALLOC_SIZE is wrong in testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h. We should refer to the implementation in glibc. or we define CPU_ALLOC_SIZE using raw sched_getaffinity as a workaround in older kernel . See below code:
#########################################################################################

        int ret;
        cpu_set_t cst;

        memset(&cst, 0, sizeof(cst));

        ret = syscall(__NR_sched_getaffinity, getpid(),
                      sizeof(cpu_set_t), &cst);
        if (ret < 0) {
                fprintf(stderr, "sched_getaffinity failed: %s\n",
                        strerror(errno));
                return 1;
        } else {
                printf("length of bit mask the kernel uses to represent the CPU"
                       ": %d\n", ret);
        }

#########################################################################################

Regards,
Xiaoguang Wang

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Stanislav Kholmanskikh

2014-07-25 06:40 UTC #1 lore raw

Quoted 4 lines · Xiaoguang Wang

Hi,
When we run sched_setaffinity01 in RHEL5.10GA, it occurs a segmentation fault. Below is the possible reason.

Hi!

I though I'd checked this test case on OL5 before submission. Let me check it again.

Quoted 104 lines

Hi Jan, would you please help to confirm this problem. I'm afraid RHEL5.10GA is an old distribution, which many people may not use it now, thanks!
Glibc provides a encapsulation for the raw kernel sched_setaffinity(2) system call, the corresponding code is below(The version of glibc I used is glibc-2.5-20061008T1257-RHEL5.11Beta):
I delete some code just for simple.
#######################################################################################
/* Size definition for CPU sets. */
# define __CPU_SETSIZE  1024
# define __NCPUBITS     (8 * sizeof (__cpu_mask))
/* Type for array elements in 'cpu_set'. */
typedef unsigned long int __cpu_mask;
/* Basic access functions. */
# define __CPUELT(cpu)  ((cpu) / __NCPUBITS)
/* Data structure to describe CPU mask. */
typedef struct
{
   __cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
} cpu_set_t;
int __sched_setaffinity_new (pid_t pid, size_t cpusetsize, const cpu_set_t *cpuset)
{
   if (__builtin_expect (__kernel_cpumask_size == 0, 0))
     {
       int res;

       while (res = INTERNAL_SYSCALL (sched_getaffinity, err, 3, getpid (),
                                      psize, p),
              INTERNAL_SYSCALL_ERROR_P (res, err)
              && INTERNAL_SYSCALL_ERRNO (res, err) == EINVAL)
      	....

       __kernel_cpumask_size = res;
     }

   /* We now know the size of the kernel cpumask_t.  Make sure the user
      does not request to set a bit beyond that.  */
   for (size_t cnt = __kernel_cpumask_size; cnt < cpusetsize; ++cnt)
     if (((char *) cpuset)[cnt] != '\0')
       {
         /* Found a nonzero byte.  This means the user request cannot be
            fulfilled.  */
         __set_errno (EINVAL);
         return -1;
       }

   return INLINE_SYSCALL (sched_setaffinity, 3, pid, cpusetsize, cpuset);
}
#######################################################################################
Glibc in RHEL5.10GA does not provide CPU_ALLOC_SIZE marco, so in ltp testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h, we define one.
#######################################################################################
#ifndef CPU_ALLOC_SIZE
#define CPU_ALLOC_SIZE(size) sizeof(cpu_set_t)
#endif
#######################################################################################
Then CPU_ALLOC_SIZE would always return 128 in RHEL5.10GA, that is when we test EFAULT for sched_setaffinity(2), the passed cpusetsize is 128. But look at __sched_setaffinity_new() above, it first call
raw sched_getaffinity(2) to get the size of the kernel cpumask_t, In RHEL5.10GA,
this value depends on CONFIG_NR_CPUS, if CONFIG_NR_CPUS is 255, the raw sched_getaffinity(2) will return 32. In this case, __kernel_cpumask_size would be 32, cpusetsize is 128. Give that we're testing EFAULT, cpuset is a invalid pointer, if cnt > 32, it will generate segmentation fault in glibc code, so this case exits abnormally
As why this test case can run normally in RHEL6.5GA or RHEL7.0GA, it's because sched_getaffinity(2) in old kernel(RHEL5.10GA) return sizeof(cpumask_t), which totally depends on CONFIG_NR_CPUS. In newer kernel, sched_getaffinity(2) returns the smaller one between min_t(size_t, len, cpumask_size()), here len is the value passed to sched_getaffinity as cpusetsize, cpumask_size() is the max allowed length. so we can ensure __kernel_cpumask_size will never smaller cpusetsize, so the segmentation fault won't occur.
So I also think CPU_ALLOC and CPU_ALLOC_SIZE is wrong in testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h. We should refer to the implementation in glibc. or we define CPU_ALLOC_SIZE using raw sched_getaffinity as a workaround in older kernel . See below code:
#########################################################################################
         int ret;
         cpu_set_t cst;

         memset(&cst, 0, sizeof(cst));

         ret = syscall(__NR_sched_getaffinity, getpid(),
                       sizeof(cpu_set_t), &cst);
         if (ret < 0) {
                 fprintf(stderr, "sched_getaffinity failed: %s\n",
                         strerror(errno));
                 return 1;
         } else {
                 printf("length of bit mask the kernel uses to represent the CPU"
                        ": %d\n", ret);
         }
#########################################################################################
Regards,
Xiaoguang Wang

Quoted 11 lines

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Xiaoguang Wang

author

2014-07-25 07:33 UTC #2 lore raw

Hi,

Quoted 11 lines · Stanislav Kholmanskikh

Quoted 4 lines · Xiaoguang Wang
Hi,
When we run sched_setaffinity01 in RHEL5.10GA, it occurs a segmentation fault. Below is the possible reason.
Hi!
I though I'd checked this test case on OL5 before submission. Let me check it again.

OL5 is based RHEL5?
If it is, please also check your CONFIG_NR_CPUS in kernel config file. If it's 1024 or greater, this case will not fail. In my environment, this value is 255, thanks!

Regards,
Xiaoguang Wang

Quoted 116 lines

Quoted 105 lines
Hi Jan, would you please help to confirm this problem. I'm afraid RHEL5.10GA is an old distribution, which many people may not use it now, thanks!
Glibc provides a encapsulation for the raw kernel sched_setaffinity(2) system call, the corresponding code is below(The version of glibc I used is glibc-2.5-20061008T1257-RHEL5.11Beta):
I delete some code just for simple.
#######################################################################################
/* Size definition for CPU sets. */
# define __CPU_SETSIZE  1024
# define __NCPUBITS     (8 * sizeof (__cpu_mask))
/* Type for array elements in 'cpu_set'. */
typedef unsigned long int __cpu_mask;
/* Basic access functions. */
# define __CPUELT(cpu)  ((cpu) / __NCPUBITS)
/* Data structure to describe CPU mask. */
typedef struct
{
   __cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
} cpu_set_t;
int __sched_setaffinity_new (pid_t pid, size_t cpusetsize, const cpu_set_t *cpuset)
{
   if (__builtin_expect (__kernel_cpumask_size == 0, 0))
     {
       int res;

       while (res = INTERNAL_SYSCALL (sched_getaffinity, err, 3, getpid (),
                                      psize, p),
              INTERNAL_SYSCALL_ERROR_P (res, err)
              && INTERNAL_SYSCALL_ERRNO (res, err) == EINVAL)
          ....

       __kernel_cpumask_size = res;
     }

   /* We now know the size of the kernel cpumask_t.  Make sure the user
      does not request to set a bit beyond that.  */
   for (size_t cnt = __kernel_cpumask_size; cnt < cpusetsize; ++cnt)
     if (((char *) cpuset)[cnt] != '\0')
       {
         /* Found a nonzero byte.  This means the user request cannot be
            fulfilled.  */
         __set_errno (EINVAL);
         return -1;
       }

   return INLINE_SYSCALL (sched_setaffinity, 3, pid, cpusetsize, cpuset);
}
#######################################################################################
Glibc in RHEL5.10GA does not provide CPU_ALLOC_SIZE marco, so in ltp testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h, we define one.
#######################################################################################
#ifndef CPU_ALLOC_SIZE
#define CPU_ALLOC_SIZE(size) sizeof(cpu_set_t)
#endif
#######################################################################################
Then CPU_ALLOC_SIZE would always return 128 in RHEL5.10GA, that is when we test EFAULT for sched_setaffinity(2), the passed cpusetsize is 128. But look at __sched_setaffinity_new() above, it first call
raw sched_getaffinity(2) to get the size of the kernel cpumask_t, In RHEL5.10GA,
this value depends on CONFIG_NR_CPUS, if CONFIG_NR_CPUS is 255, the raw sched_getaffinity(2) will return 32. In this case, __kernel_cpumask_size would be 32, cpusetsize is 128. Give that we're testing EFAULT, cpuset is a invalid pointer, if cnt > 32, it will generate segmentation fault in glibc code, so this case exits abnormally
As why this test case can run normally in RHEL6.5GA or RHEL7.0GA, it's because sched_getaffinity(2) in old kernel(RHEL5.10GA) return sizeof(cpumask_t), which totally depends on CONFIG_NR_CPUS. In newer kernel, sched_getaffinity(2) returns the smaller one between min_t(size_t, len, cpumask_size()), here len is the value passed to sched_getaffinity as cpusetsize, cpumask_size() is the max allowed length. so we can ensure __kernel_cpumask_size will never smaller cpusetsize, so the segmentation fault won't occur.
So I also think CPU_ALLOC and CPU_ALLOC_SIZE is wrong in testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h. We should refer to the implementation in glibc. or we define CPU_ALLOC_SIZE using raw sched_getaffinity as a workaround in older kernel . See below code:
#########################################################################################
         int ret;
         cpu_set_t cst;

         memset(&cst, 0, sizeof(cst));

         ret = syscall(__NR_sched_getaffinity, getpid(),
                       sizeof(cpu_set_t), &cst);
         if (ret < 0) {
                 fprintf(stderr, "sched_getaffinity failed: %s\n",
                         strerror(errno));
                 return 1;
         } else {
                 printf("length of bit mask the kernel uses to represent the CPU"
                        ": %d\n", ret);
         }
#########################################################################################
Regards,
Xiaoguang Wang
Quoted 11 lines
------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Quoted 2 lines

.

Stanislav Kholmanskikh

2014-07-25 07:51 UTC #3 lore raw

Quoted 8 lines · Xiaoguang Wang

Hi,
Quoted 5 lines · Stanislav Kholmanskikh
Quoted 2 lines · Xiaoguang Wang
Hi,

Quoted 8 lines

Quoted 6 lines
Quoted 2 lines
When we run sched_setaffinity01 in RHEL5.10GA, it occurs a segmentation fault. Below is the possible reason.
Hi!
I though I'd checked this test case on OL5 before submission. Let me check it again.
OL5 is based RHEL5?

Yes.

Quoted 2 lines

If it is, please also check your CONFIG_NR_CPUS in kernel config file. If it's 1024 or greater, this case will not fail. In my environment, this value is 255, thanks!

I initially tested it with 2.6.39-400.212.1.el5uek on OL5, and it passed. [root@ol5-x64 sched_setaffinity]# grep CONFIG_NR_CPUS /boot/config-2.6.39-400.212.1.el5uek.debug CONFIG_NR_CPUS=4096

Now I checked it with 2.6.18-371.3.1.0.1.el5 and it failed as you said:
[root@ol5-x64 sched_setaffinity]# grep CONFIG_NR_CPUS /boot/config-2.6.18-371.3.1.0.1.el5
CONFIG_NR_CPUS=255

Thanks for your findings.

I would also vote for the ltp_syscall(__NR_sched_setaffinity, ...) variant proposed by Jan.

Quoted 122 lines

Regards,
Xiaoguang Wang
Quoted 118 lines
Quoted 105 lines
Hi Jan, would you please help to confirm this problem. I'm afraid RHEL5.10GA is an old distribution, which many people may not use it now, thanks!
Glibc provides a encapsulation for the raw kernel sched_setaffinity(2) system call, the corresponding code is below(The version of glibc I used is glibc-2.5-20061008T1257-RHEL5.11Beta):
I delete some code just for simple.
#######################################################################################
/* Size definition for CPU sets. */
# define __CPU_SETSIZE  1024
# define __NCPUBITS     (8 * sizeof (__cpu_mask))
/* Type for array elements in 'cpu_set'. */
typedef unsigned long int __cpu_mask;
/* Basic access functions. */
# define __CPUELT(cpu)  ((cpu) / __NCPUBITS)
/* Data structure to describe CPU mask. */
typedef struct
{
    __cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
} cpu_set_t;
int __sched_setaffinity_new (pid_t pid, size_t cpusetsize, const cpu_set_t *cpuset)
{
    if (__builtin_expect (__kernel_cpumask_size == 0, 0))
      {
        int res;

        while (res = INTERNAL_SYSCALL (sched_getaffinity, err, 3, getpid (),
                                       psize, p),
               INTERNAL_SYSCALL_ERROR_P (res, err)
               && INTERNAL_SYSCALL_ERRNO (res, err) == EINVAL)
           ....

        __kernel_cpumask_size = res;
      }

    /* We now know the size of the kernel cpumask_t.  Make sure the user
       does not request to set a bit beyond that.  */
    for (size_t cnt = __kernel_cpumask_size; cnt < cpusetsize; ++cnt)
      if (((char *) cpuset)[cnt] != '\0')
        {
          /* Found a nonzero byte.  This means the user request cannot be
             fulfilled.  */
          __set_errno (EINVAL);
          return -1;
        }

    return INLINE_SYSCALL (sched_setaffinity, 3, pid, cpusetsize, cpuset);
}
#######################################################################################
Glibc in RHEL5.10GA does not provide CPU_ALLOC_SIZE marco, so in ltp testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h, we define one.
#######################################################################################
#ifndef CPU_ALLOC_SIZE
#define CPU_ALLOC_SIZE(size) sizeof(cpu_set_t)
#endif
#######################################################################################
Then CPU_ALLOC_SIZE would always return 128 in RHEL5.10GA, that is when we test EFAULT for sched_setaffinity(2), the passed cpusetsize is 128. But look at __sched_setaffinity_new() above, it first call
raw sched_getaffinity(2) to get the size of the kernel cpumask_t, In RHEL5.10GA,
this value depends on CONFIG_NR_CPUS, if CONFIG_NR_CPUS is 255, the raw sched_getaffinity(2) will return 32. In this case, __kernel_cpumask_size would be 32, cpusetsize is 128. Give that we're testing EFAULT, cpuset is a invalid pointer, if cnt > 32, it will generate segmentation fault in glibc code, so this case exits abnormally
As why this test case can run normally in RHEL6.5GA or RHEL7.0GA, it's because sched_getaffinity(2) in old kernel(RHEL5.10GA) return sizeof(cpumask_t), which totally depends on CONFIG_NR_CPUS. In newer kernel, sched_getaffinity(2) returns the smaller one between min_t(size_t, len, cpumask_size()), here len is the value passed to sched_getaffinity as cpusetsize, cpumask_size() is the max allowed length. so we can ensure __kernel_cpumask_size will never smaller cpusetsize, so the segmentation fault won't occur.
So I also think CPU_ALLOC and CPU_ALLOC_SIZE is wrong in testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h. We should refer to the implementation in glibc. or we define CPU_ALLOC_SIZE using raw sched_getaffinity as a workaround in older kernel . See below code:
#########################################################################################
          int ret;
          cpu_set_t cst;

          memset(&cst, 0, sizeof(cst));

          ret = syscall(__NR_sched_getaffinity, getpid(),
                        sizeof(cpu_set_t), &cst);
          if (ret < 0) {
                  fprintf(stderr, "sched_getaffinity failed: %s\n",
                          strerror(errno));
                  return 1;
          } else {
                  printf("length of bit mask the kernel uses to represent the CPU"
                         ": %d\n", ret);
          }
#########################################################################################
Regards,
Xiaoguang Wang
Quoted 11 lines
------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and search up to 200,000 lines of code with a free copy of Black Duck Code Sight - the same software that powers the world's largest code search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list
.

Jan Stancek

2014-07-25 07:05 UTC #4 lore raw

In reply to Xiaoguang Wang

----- Original Message -----

Quoted 8 lines

From: "Xiaoguang Wang" <[email protected]>
To: "LTP" <[email protected]>
Cc: "Jan Stancek" <[email protected]>
Sent: Friday, 25 July, 2014 7:38:05 AM
Subject: Segmentation fault when running sched_setaffinity01 in RHEL5.10GA
Hi,

Quoted 7 lines

When we run sched_setaffinity01 in RHEL5.10GA, it occurs a segmentation fault.
Below is the possible reason.
Hi Jan, would you please help to confirm this problem. I'm afraid RHEL5.10GA is an old
distribution, which many people may not use it now, thanks!

Hi,

RHEL5.10 GA was January 2013, it's an older codebase, but release is quite recent.

I ran this on RHEL5.3:

(gdb) r
Starting program: /root/ltp/testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity01 Detaching after fork from child process 7692.

Program received signal SIGSEGV, Segmentation fault.
__sched_setaffinity_new (pid=<value optimized out>, cpusetsize=<value optimized out>, cpuset=<value optimized out>)
at ../sysdeps/unix/sysv/linux/sched_setaffinity.c:62
62 if (((char *) cpuset)[cnt] != '\0')
(gdb) list
57 }
58
59 /* We now know the size of the kernel cpumask_t. Make sure the user
60 does not request to set a bit beyond that. */
61 for (size_t cnt = __kernel_cpumask_size; cnt < cpusetsize; ++cnt)
62 if (((char *) cpuset)[cnt] != '\0')
63 {
64 /* Found a nonzero byte. This means the user request cannot be
65 fulfilled. */
66 __set_errno (EINVAL);
(gdb) p __kernel_cpumask_size
$1 = 32
(gdb) p cpusetsize
$2 = <value optimized out>

Looking at sources of glibc-2.5 (RHEL5.10) and glibc-2.17 (RHEL7), code leading up to this place looks identical. So, I agree with your findings. I see 2 options:

1) find cpuset size in same way as glibc does it

2) call syscall directly (and don't use glibc wrapper):

testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity01.c +2 -1

index 0ac4478..0c0488f 100644
@@ -42,6 +42,7 @@
 #include "usctest.h"
 #include "safe_macros.h"
 #include "sched_setaffinity.h"
+#include "linux_syscall_numbers.h"
 
 char *TCID = "sched_setaffinity01";

@@ -151,7 +152,7 @@ int main(int argc, char *argv[])
        for (lc = 0; TEST_LOOPING(lc); lc++) {
                tst_count = 0;
                for (i = 0; i < TST_TOTAL; i++) {
-                       TEST(sched_setaffinity(*(test_cases[i].pid),
+                       TEST(ltp_syscall(__NR_sched_setaffinity, *(test_cases[i].pid),
                                                *(test_cases[i].mask_size),
                                                *(test_cases[i].mask)));

# ./sched_setaffinity01 
sched_setaffinity01    1  TPASS  :  expected failure with 'Bad address'
sched_setaffinity01    2  TPASS  :  expected failure with 'Invalid argument'
sched_setaffinity01    3  TPASS  :  expected failure with 'No such process'
sched_setaffinity01    4  TPASS  :  expected failure with 'Operation not permitted'

# uname -r
2.6.18-128.37.1.el5

Regards,
Jan

Quoted 114 lines

Glibc provides a encapsulation for the raw kernel sched_setaffinity(2) system call,
the corresponding code is below(The version of glibc I used is glibc-2.5-20061008T1257-RHEL5.11Beta):
I delete some code just for simple.
#######################################################################################
/* Size definition for CPU sets. */
# define __CPU_SETSIZE  1024
# define __NCPUBITS     (8 * sizeof (__cpu_mask))
/* Type for array elements in 'cpu_set'. */
typedef unsigned long int __cpu_mask;
/* Basic access functions. */
# define __CPUELT(cpu)  ((cpu) / __NCPUBITS)
/* Data structure to describe CPU mask. */
typedef struct
{
__cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
} cpu_set_t;
int __sched_setaffinity_new (pid_t pid, size_t cpusetsize, const cpu_set_t *cpuset)
{
if (__builtin_expect (__kernel_cpumask_size == 0, 0))
    {
      int res;

      while (res = INTERNAL_SYSCALL (sched_getaffinity, err, 3, getpid (),
                                     psize, p),
             INTERNAL_SYSCALL_ERROR_P (res, err)
             && INTERNAL_SYSCALL_ERRNO (res, err) == EINVAL)
     	....

      __kernel_cpumask_size = res;
    }
/* We now know the size of the kernel cpumask_t. Make sure the user
does not request to set a bit beyond that. */
for (size_t cnt = __kernel_cpumask_size; cnt < cpusetsize; ++cnt)
    if (((char *) cpuset)[cnt] != '\0')
      {
        /* Found a nonzero byte.  This means the user request cannot be
           fulfilled.  */
        __set_errno (EINVAL);
        return -1;
      }
return INLINE_SYSCALL (sched_setaffinity, 3, pid, cpusetsize, cpuset);
}
#######################################################################################
Glibc in RHEL5.10GA does not provide CPU_ALLOC_SIZE marco, so in ltp testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h, we define one.
#######################################################################################
#ifndef CPU_ALLOC_SIZE
#define CPU_ALLOC_SIZE(size) sizeof(cpu_set_t)
#endif
#######################################################################################
Then CPU_ALLOC_SIZE would always return 128 in RHEL5.10GA, that is when we test EFAULT for sched_setaffinity(2),
the passed cpusetsize is 128. But look at __sched_setaffinity_new() above, it first call
raw sched_getaffinity(2) to get the size of the kernel cpumask_t, In RHEL5.10GA,
this value depends on CONFIG_NR_CPUS, if CONFIG_NR_CPUS is 255, the raw sched_getaffinity(2) will return 32.
In this case, __kernel_cpumask_size would be 32, cpusetsize is 128. Give that we're testing
EFAULT, cpuset is a invalid pointer, if cnt > 32, it will generate segmentation fault in glibc code,
so this case exits abnormally
As why this test case can run normally in RHEL6.5GA or RHEL7.0GA, it's because
sched_getaffinity(2) in old kernel(RHEL5.10GA) return sizeof(cpumask_t), which totally depends on CONFIG_NR_CPUS.
In newer kernel, sched_getaffinity(2) returns the smaller one between min_t(size_t, len, cpumask_size()),
here len is the value passed to sched_getaffinity as cpusetsize, cpumask_size() is the max allowed length.
so we can ensure __kernel_cpumask_size will never smaller cpusetsize, so the segmentation fault won't occur.
So I also think CPU_ALLOC and CPU_ALLOC_SIZE is wrong in testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h. We should refer to the implementation in glibc. or we define CPU_ALLOC_SIZE using raw sched_getaffinity as a workaround in older kernel . See below code:
#########################################################################################
        int ret;
        cpu_set_t cst;

        memset(&cst, 0, sizeof(cst));

        ret = syscall(__NR_sched_getaffinity, getpid(),
                      sizeof(cpu_set_t), &cst);
        if (ret < 0) {
                fprintf(stderr, "sched_getaffinity failed: %s\n",
                        strerror(errno));
                return 1;
        } else {
                printf("length of bit mask the kernel uses to represent the
                CPU"
                       ": %d\n", ret);
        }

Quoted 5 lines

#########################################################################################
Regards,
Xiaoguang Wang

Xiaoguang Wang

author

2014-07-25 07:29 UTC #5 lore raw

Hi Jan,

Quoted 71 lines · Jan Stancek

----- Original Message -----
Quoted 8 lines
From: "Xiaoguang Wang" <[email protected]>
To: "LTP" <[email protected]>
Cc: "Jan Stancek" <[email protected]>
Sent: Friday, 25 July, 2014 7:38:05 AM
Subject: Segmentation fault when running sched_setaffinity01 in RHEL5.10GA
Hi,
Quoted 7 lines
When we run sched_setaffinity01 in RHEL5.10GA, it occurs a segmentation fault.
Below is the possible reason.
Hi Jan, would you please help to confirm this problem. I'm afraid RHEL5.10GA is an old
distribution, which many people may not use it now, thanks!
Hi,
RHEL5.10 GA was January 2013, it's an older codebase, but release is quite recent.
I ran this on RHEL5.3:
(gdb) r
Starting program: /root/ltp/testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity01 Detaching after fork from child process 7692.
Program received signal SIGSEGV, Segmentation fault.
__sched_setaffinity_new (pid=<value optimized out>, cpusetsize=<value optimized out>, cpuset=<value optimized out>)
at ../sysdeps/unix/sysv/linux/sched_setaffinity.c:62
62 if (((char *) cpuset)[cnt] != '\0')
(gdb) list
57 }
58
59 /* We now know the size of the kernel cpumask_t. Make sure the user
60 does not request to set a bit beyond that. */
61 for (size_t cnt = __kernel_cpumask_size; cnt < cpusetsize; ++cnt)
62 if (((char *) cpuset)[cnt] != '\0')
63 {
64 /* Found a nonzero byte. This means the user request cannot be
65 fulfilled. */
66 __set_errno (EINVAL);
(gdb) p __kernel_cpumask_size
$1 = 32
(gdb) p cpusetsize
$2 = <value optimized out>
Looking at sources of glibc-2.5 (RHEL5.10) and glibc-2.17 (RHEL7), code leading up to this place looks identical. So, I agree with your findings. I see 2 options:
1) find cpuset size in same way as glibc does it
2) call syscall directly (and don't use glibc wrapper):
testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity01.c +1 -0
index 0ac4478..0c0488f 100644
@@ -42,6 +42,7 @@
 #include "usctest.h"
 #include "safe_macros.h"
 #include "sched_setaffinity.h"
+#include "linux_syscall_numbers.h"
 
 char *TCID = "sched_setaffinity01";
 

testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity01.c int main(int argc, char *argv[]) · line 158

Quoted 8 lines

testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity01.c +1 -1

@@ -151,7 +152,7 @@ int main(int argc, char *argv[])
        for (lc = 0; TEST_LOOPING(lc); lc++) {
                tst_count = 0;
                for (i = 0; i < TST_TOTAL; i++) {
-                       TEST(sched_setaffinity(*(test_cases[i].pid),
+                       TEST(ltp_syscall(__NR_sched_setaffinity, *(test_cases[i].pid),
                                                *(test_cases[i].mask_size),
                                                *(test_cases[i].mask)));

Thanks very much for your quick response, cool.
Using syscall directly would be OK, but we should also add a note to later reader explaining why we do not use glibc wrapper.

Regards,
Xiaoguang Wang

Quoted 128 lines

# ./sched_setaffinity01 
sched_setaffinity01    1  TPASS  :  expected failure with 'Bad address'
sched_setaffinity01    2  TPASS  :  expected failure with 'Invalid argument'
sched_setaffinity01    3  TPASS  :  expected failure with 'No such process'
sched_setaffinity01    4  TPASS  :  expected failure with 'Operation not permitted'
# uname -r
2.6.18-128.37.1.el5
Regards,
Jan
Quoted 116 lines
Glibc provides a encapsulation for the raw kernel sched_setaffinity(2) system call,
the corresponding code is below(The version of glibc I used is glibc-2.5-20061008T1257-RHEL5.11Beta):
I delete some code just for simple.
#######################################################################################
/* Size definition for CPU sets. */
# define __CPU_SETSIZE  1024
# define __NCPUBITS     (8 * sizeof (__cpu_mask))
/* Type for array elements in 'cpu_set'. */
typedef unsigned long int __cpu_mask;
/* Basic access functions. */
# define __CPUELT(cpu)  ((cpu) / __NCPUBITS)
/* Data structure to describe CPU mask. */
typedef struct
{
__cpu_mask __bits[__CPU_SETSIZE / __NCPUBITS];
} cpu_set_t;
int __sched_setaffinity_new (pid_t pid, size_t cpusetsize, const cpu_set_t *cpuset)
{
if (__builtin_expect (__kernel_cpumask_size == 0, 0))
    {
      int res;

      while (res = INTERNAL_SYSCALL (sched_getaffinity, err, 3, getpid (),
                                     psize, p),
             INTERNAL_SYSCALL_ERROR_P (res, err)
             && INTERNAL_SYSCALL_ERRNO (res, err) == EINVAL)
     	....

      __kernel_cpumask_size = res;
    }
/* We now know the size of the kernel cpumask_t. Make sure the user
does not request to set a bit beyond that. */
for (size_t cnt = __kernel_cpumask_size; cnt < cpusetsize; ++cnt)
    if (((char *) cpuset)[cnt] != '\0')
      {
        /* Found a nonzero byte.  This means the user request cannot be
           fulfilled.  */
        __set_errno (EINVAL);
        return -1;
      }
return INLINE_SYSCALL (sched_setaffinity, 3, pid, cpusetsize, cpuset);
}
#######################################################################################
Glibc in RHEL5.10GA does not provide CPU_ALLOC_SIZE marco, so in ltp testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h, we define one.
#######################################################################################
#ifndef CPU_ALLOC_SIZE
#define CPU_ALLOC_SIZE(size) sizeof(cpu_set_t)
#endif
#######################################################################################
Then CPU_ALLOC_SIZE would always return 128 in RHEL5.10GA, that is when we test EFAULT for sched_setaffinity(2),
the passed cpusetsize is 128. But look at __sched_setaffinity_new() above, it first call
raw sched_getaffinity(2) to get the size of the kernel cpumask_t, In RHEL5.10GA,
this value depends on CONFIG_NR_CPUS, if CONFIG_NR_CPUS is 255, the raw sched_getaffinity(2) will return 32.
In this case, __kernel_cpumask_size would be 32, cpusetsize is 128. Give that we're testing
EFAULT, cpuset is a invalid pointer, if cnt > 32, it will generate segmentation fault in glibc code,
so this case exits abnormally
As why this test case can run normally in RHEL6.5GA or RHEL7.0GA, it's because
sched_getaffinity(2) in old kernel(RHEL5.10GA) return sizeof(cpumask_t), which totally depends on CONFIG_NR_CPUS.
In newer kernel, sched_getaffinity(2) returns the smaller one between min_t(size_t, len, cpumask_size()),
here len is the value passed to sched_getaffinity as cpusetsize, cpumask_size() is the max allowed length.
so we can ensure __kernel_cpumask_size will never smaller cpusetsize, so the segmentation fault won't occur.
So I also think CPU_ALLOC and CPU_ALLOC_SIZE is wrong in testcases/kernel/syscalls/sched_setaffinity/sched_setaffinity.h. We should refer to the implementation in glibc. or we define CPU_ALLOC_SIZE using raw sched_getaffinity as a workaround in older kernel . See below code:
#########################################################################################
        int ret;
        cpu_set_t cst;

        memset(&cst, 0, sizeof(cst));

        ret = syscall(__NR_sched_getaffinity, getpid(),
                      sizeof(cpu_set_t), &cst);
        if (ret < 0) {
                fprintf(stderr, "sched_getaffinity failed: %s\n",
                        strerror(errno));
                return 1;
        } else {
                printf("length of bit mask the kernel uses to represent the
                CPU"
                       ": %d\n", ret);
        }
#########################################################################################

Quoted 5 lines

Quoted 3 lines
Regards,
Xiaoguang Wang
.