Skip to contentSkip to conversation

nfs-utils: rpc.svcgssd segmentation fault in nss_gss_princ_to_ids()

PATCH Matthias Gerstner 3 participants 6 messages Feb 20, 2018 – Feb 22, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello!

I hope I have found the right place to report this.

I have recently upgraded from nfs-utils-1.3.4 to nfs-utils-2.3.1 on a Gentoo Linux (hardened) system. After this upgrade the rpc.svcgssd crashes on my kerberized NFS server each time a client tries to mount an NFS export.

The crash is a segmentation fault:

kernel: rpc.svcgssd[19772]: segfault at 8 ip 00007f6b6aafa98c sp 00007fff2ca8ce30 error 4 in nsswitch.so[7f6b6aaf9000+7000]

The call stack leading up to this looks as follows:

(gdb)
Program received signal SIGSEGV, Segmentation fault.
nss_gss_princ_to_ids (secname=<optimized out>, princ=0x55cabc7d9940 "nfs/[email protected]",

      uid=0x7ffca550e0d8, gid=0x7ffca550e0dc, UNUSED_ex=<optimized out>) at nss.c:415

415 nss.c: No such file or directory.

(gdb) bt
#0 nss_gss_princ_to_ids (secname=<optimized out>, princ=0x55cabc7d9940 "nfs/[email protected]",

      uid=0x7ffca550e0d8, gid=0x7ffca550e0dc, UNUSED_ex=<optimized out>) at nss.c:415
  #1  0x00007f6c97cc2886 in nfs4_gss_princ_to_ids (secname=secname@entry=0x55caba8e2030 <m2f+16> "krb5", 
      princ=princ@entry=0x55cabc7d9940 "nfs/[email protected]", uid=0x7ffca550e0d8, uid@entry=0x1, 
      gid=0x7ffca550e0dc, gid@entry=0x7f6c95cf3a76 <conf_get_section+86>) at libnfsidmap.c:682
  #2  0x000055caba6da138 in get_ids (cred=0x7ffca550e038, mech=<optimized out>, client_name=<optimized out>)
      at svcgssd_proc.c:251
  #3  handle_nullreq (f=f@entry=3) at svcgssd_proc.c:407
  #4  0x000055caba6d996a in gssd_run () at svcgssd_main_loop.c:91
  #5  0x000055caba6d83b1 in main (argc=1, argv=<optimized out>) at svcgssd.c:211

(gdb) p realms
$1 = (struct conf_list *) 0x0

I debugged this a bit and strange things came about. There seem to be duplicate instances of the local_realms variable in the support/nfsidmap/nfsidmap_common.c compilation unit:

gdb --args /usr/sbin/rpc.svcgssd -f
(gdb) start
Temporary breakpoint 1 at 0x2170: file svcgssd.c, line 94. Starting program: /usr/sbin/rpc.svcgssd -f
Temporary breakpoint 1, main (argc=2, argv=0x7fffffffddc8) at svcgssd.c:94
94 svcgssd.c: No such file or directory.

(gdb) b get_local_realms
Breakpoint 2 at 0x7ffff7bd1fc0: file nfsidmap_common.c, line 30. (gdb) c
Continuing.

Breakpoint 2, get_local_realms () at nfsidmap_common.c:30
30 nfsidmap_common.c: No such file or directory.
(gdb) disassemble
Dump of assembler code for function get_local_realms:
=> 0x00007ffff5e0b4a0 <+0>: sub $0x1038,%rsp

     0x00007ffff5e0b4a7 <+7>:	orq    $0x0,(%rsp)
     0x00007ffff5e0b4ac <+12>:	add    $0x1020,%rsp
     0x00007ffff5e0b4b3 <+19>:	mov    %fs:0x28,%rax
     0x00007ffff5e0b4bc <+28>:	mov    %rax,0x8(%rsp)
     0x00007ffff5e0b4c1 <+33>:	xor    %eax,%eax
     0x00007ffff5e0b4c3 <+35>:	mov    0x204f7e(%rip),%rax        # 0x7ffff6010448 <local_realms>
     0x00007ffff5e0b4ca <+42>:	mov    0x8(%rsp),%rdx
     0x00007ffff5e0b4cf <+47>:	xor    %fs:0x28,%rdx
     0x00007ffff5e0b4d8 <+56>:	jne    0x7ffff5e0b4df <get_local_realms+63>
     0x00007ffff5e0b4da <+58>:	add    $0x18,%rsp
     0x00007ffff5e0b4de <+62>:	retq   
     0x00007ffff5e0b4df <+63>:	callq  0x7ffff5e0a250 <__stack_chk_fail@plt>

End of assembler dump.

(gdb) p local_realms
$1 = (struct conf_list *) 0x555555764300
(gdb) p &local_realms
$2 = (struct conf_list **) 0x7ffff7dd7208 <local_realms>

(gdb) x /1g 0x7ffff6010448
0x7ffff6010448 <local_realms>: 0x0000000000000000

So there's one instance of the variable that was actually assigned the expected data. And there's another one that is at NULL and returned from get_local_realms().

This is related to the "#pragma GCC visibility push(hidden)" in the compilation unit. Moving it after the local_realms variable declaration fixes the issue. I've attached a patch that does just this.

I am not quite sure whether this is a compiler issue or an invalid use of the visibility hidden pragma. The compiler used is:

gcc (Gentoo Hardened 6.4.0-r1 p1.3) 6.4.0

Please advise.

Regards

Matthias

commit b1935a982aa68890ffe888381f7905f57c6b055d

Author: Matthias Gerstner <[email protected]>
Date:   Tue Feb 20 23:33:18 2018 +0100

get_local_realms: work around strange duplication of local_realms

The hidden visibility seems to cause the get_local_realms() function to return a NULL pointer, although the local_realms global variable is actually assigned. Assembler code shows there are two instances of this variable around, one set to NULL, the other to the expected value.

This happened on Gentoo Hardened with gcc 4.6.0-r1.

support/nfsidmap/nfsidmap_common.c +2 -2
index 891c855..e5c33a8 100644
@@ -19,13 +19,13 @@
 #include "nfsidmap_plugin.h"
 #include "conffile.h"
 
-#pragma GCC visibility push(hidden)
-
 int reformat_group = 0;
 int no_strip = 0;
 
 struct conf_list *local_realms;
 
+#pragma GCC visibility push(hidden)
+
 struct conf_list *get_local_realms(void)
 {
 	return local_realms;
#1 lore raw

Hello,

Quoted 3 lines · Matthias Gerstner

Hello!

I hope I have found the right place to report this.

It is... But in the future please in line your
patch (which I have a the bottom of this reply)
instead attaching it.. I just makes it easier
to review.

Quoted 4 lines

I have recently upgraded from nfs-utils-1.3.4 to nfs-utils-2.3.1 on a Gentoo Linux (hardened) system. After this upgrade the rpc.svcgssd crashes on my kerberized NFS server each time a client tries to mount an NFS export.

The main difference in these releases is we rolled libnfsidmap into nfs-utils...

Also note, at least with Fedora, we no longer use rpc.svcgssd instead we use gssproxy to access GSSAPI credentials

Quoted 72 lines

The crash is a segmentation fault:

kernel: rpc.svcgssd[19772]: segfault at 8 ip 00007f6b6aafa98c sp 00007fff2ca8ce30 error 4 in nsswitch.so[7f6b6aaf9000+7000]

The call stack leading up to this looks as follows:

(gdb)
Program received signal SIGSEGV, Segmentation fault.
nss_gss_princ_to_ids (secname=<optimized out>, princ=0x55cabc7d9940 "nfs/[email protected]",

      uid=0x7ffca550e0d8, gid=0x7ffca550e0dc, UNUSED_ex=<optimized out>) at nss.c:415

415 nss.c: No such file or directory.

(gdb) bt
#0 nss_gss_princ_to_ids (secname=<optimized out>, princ=0x55cabc7d9940 "nfs/[email protected]",

      uid=0x7ffca550e0d8, gid=0x7ffca550e0dc, UNUSED_ex=<optimized out>) at nss.c:415
  #1  0x00007f6c97cc2886 in nfs4_gss_princ_to_ids (secname=secname@entry=0x55caba8e2030 <m2f+16> "krb5", 
      princ=princ@entry=0x55cabc7d9940 "nfs/[email protected]", uid=0x7ffca550e0d8, uid@entry=0x1, 
      gid=0x7ffca550e0dc, gid@entry=0x7f6c95cf3a76 <conf_get_section+86>) at libnfsidmap.c:682
  #2  0x000055caba6da138 in get_ids (cred=0x7ffca550e038, mech=<optimized out>, client_name=<optimized out>)
      at svcgssd_proc.c:251
  #3  handle_nullreq (f=f@entry=3) at svcgssd_proc.c:407
  #4  0x000055caba6d996a in gssd_run () at svcgssd_main_loop.c:91
  #5  0x000055caba6d83b1 in main (argc=1, argv=<optimized out>) at svcgssd.c:211

(gdb) p realms
$1 = (struct conf_list *) 0x0

I debugged this a bit and strange things came about. There seem to be duplicate instances of the local_realms variable in the support/nfsidmap/nfsidmap_common.c compilation unit:

gdb --args /usr/sbin/rpc.svcgssd -f
(gdb) start
Temporary breakpoint 1 at 0x2170: file svcgssd.c, line 94. Starting program: /usr/sbin/rpc.svcgssd -f
Temporary breakpoint 1, main (argc=2, argv=0x7fffffffddc8) at svcgssd.c:94
94 svcgssd.c: No such file or directory.

(gdb) b get_local_realms
Breakpoint 2 at 0x7ffff7bd1fc0: file nfsidmap_common.c, line 30. (gdb) c
Continuing.

Breakpoint 2, get_local_realms () at nfsidmap_common.c:30
30 nfsidmap_common.c: No such file or directory.
(gdb) disassemble
Dump of assembler code for function get_local_realms:
=> 0x00007ffff5e0b4a0 <+0>: sub $0x1038,%rsp

     0x00007ffff5e0b4a7 <+7>:	orq    $0x0,(%rsp)
     0x00007ffff5e0b4ac <+12>:	add    $0x1020,%rsp
     0x00007ffff5e0b4b3 <+19>:	mov    %fs:0x28,%rax
     0x00007ffff5e0b4bc <+28>:	mov    %rax,0x8(%rsp)
     0x00007ffff5e0b4c1 <+33>:	xor    %eax,%eax
     0x00007ffff5e0b4c3 <+35>:	mov    0x204f7e(%rip),%rax        # 0x7ffff6010448 <local_realms>
     0x00007ffff5e0b4ca <+42>:	mov    0x8(%rsp),%rdx
     0x00007ffff5e0b4cf <+47>:	xor    %fs:0x28,%rdx
     0x00007ffff5e0b4d8 <+56>:	jne    0x7ffff5e0b4df <get_local_realms+63>
     0x00007ffff5e0b4da <+58>:	add    $0x18,%rsp
     0x00007ffff5e0b4de <+62>:	retq   
     0x00007ffff5e0b4df <+63>:	callq  0x7ffff5e0a250 <__stack_chk_fail@plt>

End of assembler dump.

(gdb) p local_realms
$1 = (struct conf_list *) 0x555555764300
(gdb) p &local_realms
$2 = (struct conf_list **) 0x7ffff7dd7208 <local_realms>

(gdb) x /1g 0x7ffff6010448
0x7ffff6010448 <local_realms>: 0x0000000000000000

Quoted 7 lines

So there's one instance of the variable that was actually assigned the expected data. And there's another one that is at NULL and returned from get_local_realms().

This is related to the "#pragma GCC visibility push(hidden)" in the compilation unit. Moving it after the local_realms variable declaration fixes the issue. I've attached a patch that does just this.

Nice work... BTW...

Quoted 30 lines

I am not quite sure whether this is a compiler issue or an invalid use of the visibility hidden pragma. The compiler used is:

gcc (Gentoo Hardened 6.4.0-r1 p1.3) 6.4.0

Please advise.

Regards

Matthias
commit b1935a982aa68890ffe888381f7905f57c6b055d

Author: Matthias Gerstner <[email protected]>
Date:   Tue Feb 20 23:33:18 2018 +0100

get_local_realms: work around strange duplication of local_realms

The hidden visibility seems to cause the get_local_realms() function to return a NULL pointer, although the local_realms global variable is actually assigned. Assembler code shows there are two instances of this variable around, one set to NULL, the other to the expected value.

This happened on Gentoo Hardened with gcc 4.6.0-r1.

support/nfsidmap/nfsidmap_common.c +0 -0
index 891c855..e5c33a8 100644
@@ -19,13 +19,13 @@
 #include "nfsidmap_plugin.h"

support/nfsidmap/nfsidmap_common.c line 27

Quoted 10 lines
support/nfsidmap/nfsidmap_common.c +1 -2
 #include "conffile.h"
 
-#pragma GCC visibility push(hidden)
-
 int reformat_group = 0;
 int no_strip = 0;
 
 struct conf_list *local_realms;
 
+#pragma GCC visibility push(hidden)

By moving this pragama statement both the no_strip and reformat_group are also being exposed as global variables in the libnfsidmap, correct? I'm not sure we want to do that, if that is the case.

I guess exposing local_realms is necessary
but it is adding to the API... which is probably
not good...

steved.

support/nfsidmap/nfsidmap_common.c line 30

Quoted 3 lines
support/nfsidmap/nfsidmap_common.c +1 -0
+
 struct conf_list *get_local_realms(void)
 {

return local_realms;

#2 lore raw

Hello,

thanks for your reply.

Quoted 4 lines

It is... But in the future please in line your
patch (which I have a the bottom of this reply)
instead attaching it.. I just makes it easier
to review.

I will do so.

Quoted 2 lines

Also note, at least with Fedora, we no longer use rpc.svcgssd instead we use gssproxy to access GSSAPI credentials

Okay. I have been running this NFS setup for years without touching much. It looks like I should invest some time to check if everything is still up to date.

Quoted 4 lines

By moving this pragama statement both the no_strip and reformat_group are also being exposed as global variables in the libnfsidmap, correct? I'm not sure we want to do that, if that is the case.

I was not suggesting to accept this as a patch. I just wanted to make clear what "helped" in my case.

I will try some more experiments when I have time. Maybe I can find a suitable patch. Getter/Setter functions instead of the extern declared variables might help.

Regards

Matthias

#3 lore raw
In reply to Matthias Gerstner
Quoted 7 lines · Matthias Gerstner

Hello!

I hope I have found the right place to report this.

I have recently upgraded from nfs-utils-1.3.4 to nfs-utils-2.3.1 on a Gentoo Linux (hardened) system. After this upgrade the rpc.svcgssd crashes on my kerberized NFS server each time a client tries to mount an NFS export.

<snip>

Quoted 4 lines

I debugged this a bit and strange things came about. There seem to be duplicate instances of the local_realms variable in the support/nfsidmap/nfsidmap_common.c compilation unit:

<snip>

Quoted 10 lines

So there's one instance of the variable that was actually assigned the expected data. And there's another one that is at NULL and returned from get_local_realms().

This is related to the "#pragma GCC visibility push(hidden)" in the compilation unit. Moving it after the local_realms variable declaration fixes the issue. I've attached a patch that does just this.

I am not quite sure whether this is a compiler issue or an invalid use of the visibility hidden pragma. The compiler used is:

This is part of my previous work to fold libnfsidmap into the nfs-utils tree, in this case the #pragma is used to make the local_realms global accessible only by other parts of the library that need it, external users such as the nss plugin get access to it via the get_local_realms() function.

I will run some tests to see if this is isolated to certain compiler versions, or if the pragma isnt working as I expected.

However simply moving the #pragma hidden will change the ABI which is a problem for some, so if necessary I will instead reorganise the code to achieve the same functionality without using the pragma

#4 lore raw

Hello,

Quoted 6 lines

I will run some tests to see if this is isolated to certain compiler versions, or if the pragma isnt working as I expected.

However simply moving the #pragma hidden will change the ABI which is a problem for some, so if necessary I will instead reorganise the code to achieve the same functionality without using the pragma

yes I understand. I was not suggesting this patch to be an actual solution.

Thanks for looking into this.

Regards

Matthias

#5 lore raw
In reply to Justin Mitchell

Hello,

Quoted 2 lines

I will run some tests to see if this is isolated to certain compiler versions, or if the pragma isnt working as I expected.

I just want to report that I just tested with a more recent gcc 7.3.0 and the result was the same.

Regards

Matthias