RH 7 nfs-utils update causes

I recently updated a Red Hat 7 host. After rebooting, the nfs mounts on it (nfsver3,sec=krb5) failed to mount, due to the gss daemon segfaulting when the mount attempt happened:

[    7.816487] FS-Cache: Loaded
[    7.887575] FS-Cache: Netfs 'nfs' registered for caching
[    7.931164] rpc.gssd[498]: segfault at 5544452e ip 00007fbc9d704ee6 sp 00007ffc37291678 error 4 in libc-2.17.so[7fbc9d5ca000+1b
4000]
[    7.964578] abrt-hook-ccpp[994]: segfault at 0 ip 00007fba2e09431b sp 00007ffefa92fb50 error 4 in libreport.so.0.0.1[7fba2e0860
00+25000]
[    7.965398] Process 994(abrt-hook-ccpp) has RLIMIT_CORE set to 1
[    7.965483] Aborting core

At mount time, The console displays these messages as well:

RPC: AUTH_GSS upcall timed out.
Please check user daemon is running.

gssd will start without issue if started manually on its own, but dies if I subsequently try to manually mount any of the nfs mounts.

I've isolated the issue down to the latest update in the nfs-utils package:

Working: nfs-utils-1.3.0-0.21.el7.x86_64
Broken: nfs-utils-1.3.0-0.21.el7_2.x86_64

(note the '_v2' difference)

gssd/the mounting works without issue after backing down to the older nfs-utils version.

It looks similar to the following bug report (fixed in 1.3.1), but I'm not 100% convinced it's the same:

https://bugzilla.redhat.com/show_bug.cgi?id=1108615

If this is in fact the issue, I would be happy keeping the updated nfs-utils version, but working around the issue via changing my krb5.conf, but I'm not sure where the *preferred*_realm comes from - I do have *default*_realm set in my krb5.conf.

If that's not it, I'd be happy to provide any additional information that may assist in troubleshooting & welcome any suggestions, but I'd greatly prefer to retain the OS-supplied nfs-utils.

-Mark

Hey Mark, could it be this one?

https://bugzilla.redhat.com/show_bug.cgi?id=1313090

Ben

Doesn't look like it:

$ sudo restorecon -v /etc/krb5.keytab
<no output>
$ ls -laZ /etc/krb5.keytab
-rw-------. root root unconfined_u:object_r:krb5_keytab_t:s0 /etc/krb5.keytab

I'd only had selinux in permissive mode, so I tried disabling it as well, but that too unfortunately made no difference.

That bug was only tickled in my case if the normal host principals were not found in the keytab. In that specific case, selinux prohibited access to the keytab, causing the host keys to be unable to be looked up, and the bug was hit.

Best bet would be to attache a debugger to rpc.gssd and get a stack trace of it when it crashes. That will give us something to work from.

You can do that with gdb -p <pid of gssd>. Something like gdb -p $(pidof rpc.gssd) will work -- then type "continue" to let the process continue to run. When it fails you can enter "bt" to emit a back trace. That information would be very helpful.

Ben

Thank you for the gdb usage suggestion - I needed it...

Here's the output:

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 492
Reading symbols from /usr/sbin/rpc.gssd...Reading symbols from /usr/lib/debug/usr/sbin/rpc.gssd.debug...done.
done.
Reading symbols from /lib64/libtirpc.so.1...Reading symbols from /lib64/libtirpc.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libtirpc.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libgssapi_krb5.so.2.2.debug...done.
done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5.so.3.3.debug...done.
done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libk5crypto.so.3.1.debug...done.
done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libcom_err.so.2.1.debug...done.
done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libnsl.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libnsl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libpthread.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libpthread-2.17.so.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5support.so.0.1.debug...done.
done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libdl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libkeyutils.so.1.5.debug...done.
done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libresolv-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libselinux.so.1.debug...done.
done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/liblzma.so.5...Reading symbols from /lib64/liblzma.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblzma.so.5
Reading symbols from /usr/lib64/gssproxy/proxymech.so...Reading symbols from /usr/lib/debug/usr/lib64/gssproxy/proxymech.so.debug...done. done.
Loaded symbols for /usr/lib64/gssproxy/proxymech.so
Reading symbols from /lib64/libgssrpc.so.4...Reading symbols from /usr/lib/debug/usr/lib64/libgssrpc.so.4.2.debug...done.
done.
Loaded symbols for /lib64/libgssrpc.so.4
0x00007f4cd7ba4be9 in __GI_ppoll (fds=fds@entry=0x7f4cd9dc8b50, nfds=nfds@entry=1024, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x7ffdd6f44770)

    at ../sysdeps/unix/sysv/linux/ppoll.c:51

51 result = INLINE_SYSCALL (ppoll, 5, fds, nfds, timeout, sigmask, _NSIG / 8);
Missing separate debuginfos, use: debuginfo-install libtirpc-0.2.4-0.6.el7.x86_64 pcre-8.32-15.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 (gdb) continue
Continuing.

Program received signal SIG37, Real-time event 37.
0x00007f4cd7ba4be9 in __GI_ppoll (fds=fds@entry=0x7f4cd9dc8b50, nfds=nfds@entry=1024, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x7ffdd6f44770)

    at ../sysdeps/unix/sysv/linux/ppoll.c:51
51          result = INLINE_SYSCALL (ppoll, 5, fds, nfds, timeout, sigmask, _NSIG / 8);
(gdb) bt
#0  0x00007f4cd7ba4be9 in __GI_ppoll (fds=fds@entry=0x7f4cd9dc8b50, nfds=nfds@entry=1024, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x7ffdd6f44770)
    at ../sysdeps/unix/sysv/linux/ppoll.c:51
#1  0x00007f4cd8c30587 in ppoll (__ss=0x7ffdd6f44770, __timeout=0x0, __nfds=1024, __fds=0x7f4cd9dc8b50) at /usr/include/bits/poll2.h:77
#2  gssd_poll (nfds=1024, fds=0x7f4cd9dc8b50) at gssd_main_loop.c:194
#3  gssd_run () at gssd_main_loop.c:258
#4  0x00007f4cd8c2f11a in main (argc=<optimized out>, argv=0x7ffdd6f449d8) at gssd.c:216

That's weird -- you're getting SIG37 - I don't know what that could be. Can you just do another 'continue' there? If not, can you do 'ptype $_siginfo' there?

Ben

Here's a new gdb session:

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 464
Reading symbols from /usr/sbin/rpc.gssd...Reading symbols from /usr/lib/debug/usr/sbin/rpc.gssd.debug...done.
done.
Reading symbols from /lib64/libtirpc.so.1...Reading symbols from /lib64/libtirpc.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libtirpc.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libgssapi_krb5.so.2.2.debug...done.
done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5.so.3.3.debug...done.
done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libk5crypto.so.3.1.debug...done.
done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libcom_err.so.2.1.debug...done.
done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libnsl.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libnsl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libpthread.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libpthread-2.17.so.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5support.so.0.1.debug...done.
done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libdl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libkeyutils.so.1.5.debug...done.
done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libresolv-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libselinux.so.1.debug...done.
done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/liblzma.so.5...Reading symbols from /lib64/liblzma.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblzma.so.5
Reading symbols from /usr/lib64/gssproxy/proxymech.so...Reading symbols from /usr/lib/debug/usr/lib64/gssproxy/proxymech.so.debug...done. done.
Loaded symbols for /usr/lib64/gssproxy/proxymech.so
Reading symbols from /lib64/libgssrpc.so.4...Reading symbols from /usr/lib/debug/usr/lib64/libgssrpc.so.4.2.debug...done.
done.
Loaded symbols for /lib64/libgssrpc.so.4
0x00007f4c7041fbe9 in __GI_ppoll (fds=fds@entry=0x7f4c72c45b50, nfds=nfds@entry=1024, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x7ffd00c8b7d0)

    at ../sysdeps/unix/sysv/linux/ppoll.c:51

51 result = INLINE_SYSCALL (ppoll, 5, fds, nfds, timeout, sigmask, _NSIG / 8);
Missing separate debuginfos, use: debuginfo-install libtirpc-0.2.4-0.6.el7.x86_64 pcre-8.32-15.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 (gdb) continue
Continuing.

Program received signal SIG37, Real-time event 37.
0x00007f4c7041fbe9 in __GI_ppoll (fds=fds@entry=0x7f4c72c45b50, nfds=nfds@entry=1024, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x7ffd00c8b7d0)

    at ../sysdeps/unix/sysv/linux/ppoll.c:51
51          result = INLINE_SYSCALL (ppoll, 5, fds, nfds, timeout, sigmask, _NSIG / 8);
(gdb) bt
#0  0x00007f4c7041fbe9 in __GI_ppoll (fds=fds@entry=0x7f4c72c45b50, nfds=nfds@entry=1024, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x7ffd00c8b7d0)
    at ../sysdeps/unix/sysv/linux/ppoll.c:51
#1  0x00007f4c714ab587 in ppoll (__ss=0x7ffd00c8b7d0, __timeout=0x0, __nfds=1024, __fds=0x7f4c72c45b50) at /usr/include/bits/poll2.h:77
#2  gssd_poll (nfds=1024, fds=0x7f4c72c45b50) at gssd_main_loop.c:194
#3  gssd_run () at gssd_main_loop.c:258
#4  0x00007f4c714aa11a in main (argc=<optimized out>, argv=0x7ffd00c8ba38) at gssd.c:216

(gdb) continue
Continuing.

Program received signal SIG37, Real-time event 37.
0x00007f4c7041fbe9 in __GI_ppoll (fds=fds@entry=0x7f4c72c45b50, nfds=nfds@entry=1024, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x7ffd00c8b7d0)

    at ../sysdeps/unix/sysv/linux/ppoll.c:51
51          result = INLINE_SYSCALL (ppoll, 5, fds, nfds, timeout, sigmask, _NSIG / 8);
(gdb) ptype $_siginfo
type = struct {
    int si_signo;
    int si_errno;
    int si_code;
    union {
        int _pad __attribute__ ((vector_size(28)));
        struct {...} _kill;
        struct {...} _timer;
        struct {...} _rt;
        struct {...} _sigchld;
        struct {...} _sigfault;
        struct {...} _sigpoll;
    } _sifields;
}

In order to keep the gssd process alive (to be able to attach gbd to it), I've commented out my nfs mounts in vfstab & am then mounting the export manually:

$ sudo mount /my/mountpoint
Error getting authority: Error initializing authority: Error calling StartServiceByName for org.freedesktop.PolicyKit1: Timeout was reached (g-io-error-quark, 24)
Failed to start rpc-statd.service: Connection timed out
mount.nfs: access denied by server while mounting nfsserver:/the_export

$ systemctl --failed
  UNIT           LOAD   ACTIVE SUB    DESCRIPTION
 polkit.service loaded failed failed Authorization Manager

I see now that when mounting manually this way, the gss daemon actually doesn't die. The rpc-statd process the mount whines about failing to start *is* running.

-Mark

Before the first 'continue' use 'handle SIG37 nostop'..

Ben

I get a segfault that way:

GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 473
Reading symbols from /usr/sbin/rpc.gssd...Reading symbols from /usr/lib/debug/usr/sbin/rpc.gssd.debug...done.
done.
Reading symbols from /lib64/libtirpc.so.1...Reading symbols from /lib64/libtirpc.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libtirpc.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libgssapi_krb5.so.2.2.debug...done.
done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5.so.3.3.debug...done.
done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libk5crypto.so.3...Reading symbols from /usr/lib/debug/usr/lib64/libk5crypto.so.3.1.debug...done.
done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libcom_err.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libcom_err.so.2.1.debug...done.
done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libc.so.6...Reading symbols from /usr/lib/debug/usr/lib64/libc-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libnsl.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libnsl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libpthread.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libpthread-2.17.so.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libkrb5support.so.0...Reading symbols from /usr/lib/debug/usr/lib64/libkrb5support.so.0.1.debug...done.
done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libdl.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libdl-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libkeyutils.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libkeyutils.so.1.5.debug...done.
done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...Reading symbols from /usr/lib/debug/usr/lib64/libresolv-2.17.so.debug...done.
done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libselinux.so.1...Reading symbols from /usr/lib/debug/usr/lib64/libselinux.so.1.debug...done.
done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /lib64/libpcre.so.1...Reading symbols from /lib64/libpcre.so.1...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/liblzma.so.5...Reading symbols from /lib64/liblzma.so.5...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Loaded symbols for /lib64/liblzma.so.5
Reading symbols from /usr/lib64/gssproxy/proxymech.so...Reading symbols from /usr/lib/debug/usr/lib64/gssproxy/proxymech.so.debug...done. done.
Loaded symbols for /usr/lib64/gssproxy/proxymech.so
Reading symbols from /lib64/libgssrpc.so.4...Reading symbols from /usr/lib/debug/usr/lib64/libgssrpc.so.4.2.debug...done.
done.
Loaded symbols for /lib64/libgssrpc.so.4
0x00007ff64f5a6be9 in __GI_ppoll (fds=fds@entry=0x7ff6518b8b50, nfds=nfds@entry=1024, timeout=<optimized out>, timeout@entry=0x0, sigmask=sigmask@entry=0x7ffce9f91570)

    at ../sysdeps/unix/sysv/linux/ppoll.c:51
51          result = INLINE_SYSCALL (ppoll, 5, fds, nfds, timeout, sigmask, _NSIG / 8);
Missing separate debuginfos, use: debuginfo-install libtirpc-0.2.4-0.6.el7.x86_64 pcre-8.32-15.el7.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64
(gdb) handle SIG37 nostop
Signal        Stop      Print   Pass to program Description
SIG37         No        Yes     Yes             Real-time event 37

(gdb) continue
Continuing.

Program received signal SIG37, Real-time event 37.

Program received signal SIG37, Real-time event 37.

Program received signal SIG37, Real-time event 37.

Program received signal SIG37, Real-time event 37.

Program received signal SIG37, Real-time event 37.

Program received signal SIGSEGV, Segmentation fault. __strcmp_ssse3 () at ../sysdeps/x86_64/strcmp.S:212
212 movlpd (%rdi), %xmm1
(gdb) continue
Continuing.

Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists.
(gdb)

That's where we want the backtrace; that's when you should do 'bt'. Feel free to leave out everything else.

I think these SIG37s are somehow the DNOTIFY_SIGNAL, though they should be SIG35 (SIGRTMIN + 3), not 37.

Ben

Wow - Sorry about that.

(gdb) handle SIG37 nostop

Signal        Stop      Print   Pass to program Description
SIG37         No        Yes     Yes             Real-time event 37

(gdb) continue
Continuing.

Program received signal SIG37, Real-time event 37.

Program received signal SIG37, Real-time event 37.

Program received signal SIG37, Real-time event 37.

Program received signal SIG37, Real-time event 37.

Program received signal SIG37, Real-time event 37.

Program received signal SIGSEGV, Segmentation fault.
__strcmp_ssse3 () at ../sysdeps/x86_64/strcmp.S:212
212 movlpd (%rdi), %xmm1
(gdb) bt
#0 __strcmp_ssse3 () at ../sysdeps/x86_64/strcmp.S:212
#1 0x00007fec0dfb60f4 in find_keytab_entry (context=0x7fec0f82d6d0, kt=0x7fec0f829a10, tgtname=tgtname@entry=0x7fec0f829910 "ubfs1.itorg.ad.buffalo.edu",

    kte=kte@entry=0x7fffc8ca1ec0, 
svcnames=svcnames@entry=0x7fffc8ca1e90) at krb5_util.c:894
#2  0x00007fec0dfb6ded in gssd_refresh_krb5_machine_credential (hostname=0x7fec0f829910 "ubfs1.itorg.ad.buffalo.edu", ple=ple@entry=0x0, service=service@entry=0x0)
    at krb5_util.c:1311
#3  0x00007fec0dfb4030 in krb5_use_machine_creds (clp=clp@entry=0x7fec0f82bcb0, uid=uid@entry=0, tgtname=tgtname@entry=0x0, service=service@entry=0x0,
    rpc_clnt=rpc_clnt@entry=0x7fffc8ca2000) at gssd_proc.c:1086
#4  0x00007fec0dfb4416 in process_krb5_upcall (clp=clp@entry=0x7fec0f82bcb0, uid=0, fd=13, tgtname=tgtname@entry=0x0, service=service@entry=0x0) at gssd_proc.c:1222
#5  0x00007fec0dfb4d03 in handle_gssd_upcall (clp=clp@entry=0x7fec0f82bcb0) at gssd_proc.c:1413
#6  0x00007fec0dfb264e in scan_poll_results (ret=1) at gssd_main_loop.c:85
#7  gssd_poll (nfds=<optimized out>, fds=<optimized out>) at gssd_main_loop.c:201
#8  gssd_run () at gssd_main_loop.c:258
#9  0x00007fec0dfb111a in main (argc=<optimized out>, argv=0x7fffc8ca2418) at gssd.c:216

(gdb) continue
Continuing.

Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists.
(gdb)

FWIW, the ubfs1.itorg.ad.buffalo.edu hostname mentioned in the backtrace is the nfs server.

OK, that's pretty certainly this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1313090

You can add your email to the CC list there, and leave a comment stating you're hitting it as well. If you have Red Hat support, you can open a support case.

The patch in that BZ should fix it up for you, or you can use the previous nfs-utils version until that BZ changes to CLOSED, at which time a fixed nfs-utils pacakge will be available.

Nice job working through gdb over email, and sorry for the trouble.

Ben

Done.

Although I know I specifically mentioned RHEL 7 originally, we're actually now an Oracle Linux shop (sorry...) - does opening a case with Oracle help at all? (I'm actually not 100% sure we have support with them either though.)

Thank you for your patience & walking me through it. I haven't touched gdb since college & they don't let me drink at work like I surely did last time I used it...

I'll probably stay at the previous nfs-utils version until a patch comes out & trickles down to Oracle Linux.

Thanks again!
-Mark