Skip to contentSkip to conversation

Two rpc.gssd improvements

PATCH Superseded Scott Mayhew 1 participant 3 messages May 27, 2021

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Two rpc.gssd improvements

Changes since v1:

- Replaced the upcall_thread_info.cancelled field with a flags field, to facilitate having the watchdog thread print an error message only once for each timed-out upcall thread.
- Removed the "created thread id" log message.
- Added missing break when parsing the "-C" option.
- Added some comments.

These patches provide the following improvements for rpc.gssd:
1) deal with failed thread creation
2) add a timeout for upcall threads

Both of these issues can leave kernel mount processes hanging indefinitely. A timeout was originally proposed in the kernel
(https://lore.kernel.org/linux-nfs/[email protected]/)
but this approach was rejected by Trond:

I'm saying that we can do this entirely in userland without any kernel changes. As long as that hasn't been attempted and proven to be flawed, then there is no reason to accept any kernel patches.

So this is my attempt at doing the timeout in userland.

The first patch was tested using a program that intercepts clone() and changes the return code to -EAGAIN.

For the second patch, I have two different tests I've been running:

1) In an IPA domain in our lab, I have a server running 100 kerberized nfsd containers. The client has mountpoints to all 100 of those servers defined in its /etc/fstab. I run 'systemctl start remote-fs.target' to kick off all those mounts in parallel, while running the following systemtap script to periodically mess with the mount processes:

---8<---
global i

probe begin { i=0 }

probe process("/lib64/libgssapi_krb5.so.2").function("gss_acquire_cred")

{
        if (++i % 100 == 0) {
                printf("delay (i=%d)\n", i)
                mdelay(30000)
        }
}

---8<---

I actually run the test in a loop... the driver script looks like this:

---8<---
#!/bin/bash
let i=1
while :; do
echo "Round $i"
echo "Mounting"
systemctl start remote-fs.target
echo -n "Waiting on mount.nfs processes to complete " while pgrep mount.nfs >/dev/null; do
echo -n "."
sleep 1
done
echo -e "\nNumber of nfs4 mounts: $(grep -c nfs4 /proc/mounts)" echo -e "Unmounting"
umount -a -t nfs4
if ! pgrep gssd >/dev/null; then
echo "gssd is not running - check for crash" break
fi
echo "Sleeping 5 seconds"
sleep 5
let i=$i+1
done
---8<---

2) In an AD environment in our lab, I added 1000 test users. On a client machine I have all those users run a script that writes to files on a NetApp SVM and while that script is running I trigger a LIF migration on the filer. That forces all those users to establish new creds with the SVM.

That test looks basically like this

# for i in `seq 1 1000`; do su - testuser$i -c "echo 'PASSWORD'|kinit"; done
# for i in `seq 1 1000`; do su - testuser$i -c "date >/mnt/t/tmp/testuser$i-testfile" & done
# for i in `seq 1 1000`; do su - testuser$i -c test.sh & done

where test.sh is a simple script that writes the date to a file in a loop:

---8<---
#!/bin/bash
filename=/mnt/t/tmp/$(whoami)-testfile
for i in $(seq 1 300)
do
date >$filename
sleep 1
done
---8<---

While the test users are running the script I run one of the following commands on the NetApp filer:

network interface migrate -vserver VSERVER -lif LIF -destination-node NODE network interface revert -vserver VSERVER -lif LIF

-Scott

Scott Mayhew (2):
gssd: deal with failed thread creation
gssd: add timeout for upcall threads

 nfs.conf               |   2 +
 utils/gssd/gssd.c      | 256 +++++++++++++++++++++++-----------
 utils/gssd/gssd.h      |  29 +++-
 utils/gssd/gssd.man    |  31 ++++-
 utils/gssd/gssd_proc.c | 306 ++++++++++++++++++++++++++++++++++-------
 5 files changed, 491 insertions(+), 133 deletions(-)
Signature · 1 line
2.30.2
#1 lore raw

PATCH v2 1/2 gssd: deal with failed thread creation

If we fail to create a thread to handle an upcall, we still need to do a downcall to tell the kernel about the failure, otherwise the process that is trying to establish gss credentials will hang.

This patch shifts the thread creation down a level in the call chain so now the main thread does a little more work up front (reading & parsing the data from the pipefs file) so it has the info it needs to be able to do the error downcall.

---
 utils/gssd/gssd.c      |  83 +-----------------
 utils/gssd/gssd.h      |  11 ++-
 utils/gssd/gssd_proc.c | 188 +++++++++++++++++++++++++++++++++--------
 3 files changed, 164 insertions(+), 118 deletions(-)
utils/gssd/gssd.c +3 -80
index 1541d371..eb440470 100644
@@ -364,7 +364,7 @@ out:
 /* Actually frees clp and fields that might be used from other
  * threads if was last reference.
  */
-static void
+void
 gssd_free_client(struct clnt_info *clp)
 {
 	int refcnt;
@@ -416,55 +416,6 @@ gssd_destroy_client(struct clnt_info *clp)
 
 static void gssd_scan(void);
 
-static int
-start_upcall_thread(void (*func)(struct clnt_upcall_info *), void *info)
-{
-	pthread_attr_t attr;
-	pthread_t th;
-	int ret;
-
-	ret = pthread_attr_init(&attr);
-	if (ret != 0) {
-		printerr(0, "ERROR: failed to init pthread attr: ret %d: %s\n",
-			 ret, strerror(errno));
-		return ret;
-	}
-	ret = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
-	if (ret != 0) {
-		printerr(0, "ERROR: failed to create pthread attr: ret %d: "
-			 "%s\n", ret, strerror(errno));
-		return ret;
-	}
-
-	ret = pthread_create(&th, &attr, (void *)func, (void *)info);
-	if (ret != 0)
-		printerr(0, "ERROR: pthread_create failed: ret %d: %s\n",
-			 ret, strerror(errno));
-	return ret;
-}
-
-static struct clnt_upcall_info *alloc_upcall_info(struct clnt_info *clp)
-{
-	struct clnt_upcall_info *info;
-
-	info = malloc(sizeof(struct clnt_upcall_info));
-	if (info == NULL)
-		return NULL;
-
-	pthread_mutex_lock(&clp_lock);
-	clp->refcount++;
-	pthread_mutex_unlock(&clp_lock);
-	info->clp = clp;
-
-	return info;
-}
-
-void free_upcall_info(struct clnt_upcall_info *info)
-{
-	gssd_free_client(info->clp);
-	free(info);
-}
-
 /* For each upcall read the upcall info into the buffer, then create a
  * thread in a detached state so that resources are released back into
  * the system without the need for a join.
@@ -473,44 +424,16 @@ static void
 gssd_clnt_gssd_cb(int UNUSED(fd), short UNUSED(which), void *data)
 {
 	struct clnt_info *clp = data;
-	struct clnt_upcall_info *info;
-
-	info = alloc_upcall_info(clp);
-	if (info == NULL)
-		return;
 
-	info->lbuflen = read(clp->gssd_fd, info->lbuf, sizeof(info->lbuf));
-	if (info->lbuflen <= 0 || info->lbuf[info->lbuflen-1] != '\n') {
-		printerr(0, "WARNING: %s: failed reading request\n", __func__);
-		free_upcall_info(info);
-		return;
-	}
-	info->lbuf[info->lbuflen-1] = 0;
-
-	if (start_upcall_thread(handle_gssd_upcall, info))
-		free_upcall_info(info);
+	handle_gssd_upcall(clp);
 }
 
 static void
 gssd_clnt_krb5_cb(int UNUSED(fd), short UNUSED(which), void *data)
 {
 	struct clnt_info *clp = data;
-	struct clnt_upcall_info *info;
-
-	info = alloc_upcall_info(clp);
-	if (info == NULL)
-		return;
-
-	if (read(clp->krb5_fd, &info->uid,
-			sizeof(info->uid)) < (ssize_t)sizeof(info->uid)) {
-		printerr(0, "WARNING: %s: failed reading uid from krb5 "
-			 "upcall pipe: %s\n", __func__, strerror(errno));
-		free_upcall_info(info);
-		return;
-	}
 
-	if (start_upcall_thread(handle_krb5_upcall, info))
-		free_upcall_info(info);
+	handle_krb5_upcall(clp);
 }
 
 static struct clnt_info *
utils/gssd/gssd.h +7 -4
index 1e8c58d4..6d53647e 100644
@@ -84,14 +84,17 @@ struct clnt_info {
 
 struct clnt_upcall_info {
 	struct clnt_info 	*clp;
-	char			lbuf[RPC_CHAN_BUF_SIZE];
-	int			lbuflen;
 	uid_t			uid;
+	int			fd;
+	char			*srchost;
+	char			*target;
+	char			*service;
 };
 
-void handle_krb5_upcall(struct clnt_upcall_info *clp);
-void handle_gssd_upcall(struct clnt_upcall_info *clp);
+void handle_krb5_upcall(struct clnt_info *clp);
+void handle_gssd_upcall(struct clnt_info *clp);
 void free_upcall_info(struct clnt_upcall_info *info);
+void gssd_free_client(struct clnt_info *clp);
 
 
 #endif /* _RPC_GSSD_H_ */
utils/gssd/gssd_proc.c +154 -34

252 lines not on this page · raw message

Signature · 1 line
2.30.2
#2 lore raw

PATCH v2 2/2 gssd: add timeout for upcall threads

In reply to Scott Mayhew

Add a global list of active upcalls and a watchdog thread that walks the list, looking for threads running longer than timeout seconds. By default, the watchdog thread will cancel these threads and report an error of -ETIMEDOUT to the kernel. Passing the -C option or setting cancel-timed-out-upcalls=0 in nfs.conf disables this behavior and causes the watchdog thread to simply log an error message instead. The upcall timeout can be specified by passing the -U option or by setting the upcall-timeout parameter in nfs.conf.

---
 nfs.conf               |   2 +
 utils/gssd/gssd.c      | 179 ++++++++++++++++++++++++++++++++++++++++-
 utils/gssd/gssd.h      |  18 +++++
 utils/gssd/gssd.man    |  31 ++++++-
 utils/gssd/gssd_proc.c | 138 +++++++++++++++++++++++++------
 5 files changed, 340 insertions(+), 28 deletions(-)
nfs.conf +2 -0
index 31994f61..7a987788 100644
@@ -25,6 +25,8 @@
 # cred-cache-directory=
 # preferred-realm=
 # set-home=1
+# upcall-timeout=30
+# cancel-timed-out-upcalls=1
 #
 [lockd]
 # port=0
utils/gssd/gssd.c +177 -2

229 lines not on this page · raw message

utils/gssd/gssd.h +18 -0
index 6d53647e..c52c5b48 100644
@@ -50,6 +50,12 @@
 #define GSSD_DEFAULT_KEYTAB_FILE		"/etc/krb5.keytab"
 #define GSSD_SERVICE_NAME			"nfs"
 #define RPC_CHAN_BUF_SIZE			32768
+
+/* timeouts are in seconds */
+#define MIN_UPCALL_TIMEOUT			5
+#define DEF_UPCALL_TIMEOUT			30
+#define MAX_UPCALL_TIMEOUT			600
+
 /*
  * The gss mechanisms that we can handle
  */
@@ -91,10 +97,22 @@ struct clnt_upcall_info {
 	char			*service;
 };
 
+struct upcall_thread_info {
+	TAILQ_ENTRY(upcall_thread_info) list;
+	pthread_t		tid;
+	struct timespec		timeout;
+	uid_t			uid;
+	int			fd;
+	unsigned short		flags;
+#define UPCALL_THREAD_CANCELED	0x0001
+#define UPCALL_THREAD_WARNED	0x0002
+};
+
 void handle_krb5_upcall(struct clnt_info *clp);
 void handle_gssd_upcall(struct clnt_info *clp);
 void free_upcall_info(struct clnt_upcall_info *info);
 void gssd_free_client(struct clnt_info *clp);
+int do_error_downcall(int k5_fd, uid_t uid, int err);
 
 
 #endif /* _RPC_GSSD_H_ */
utils/gssd/gssd.man +29 -2
index 9ae6def9..20fea729 100644
@@ -8,7 +8,7 @@
 rpc.gssd \- RPCSEC_GSS daemon
 .SH SYNOPSIS
 .B rpc.gssd
-.RB [ \-DfMnlvrH ]
+.RB [ \-DfMnlvrHC ]
 .RB [ \-k
 .IR keytab ]
 .RB [ \-p
@@ -17,6 +17,10 @@ rpc.gssd \- RPCSEC_GSS daemon
 .IR ccachedir ]
 .RB [ \-t
 .IR timeout ]
+.RB [ \-T
+.IR timeout ]
+.RB [ \-U
+.IR timeout ]
 .RB [ \-R
 .IR realm ]
 .SH INTRODUCTION
@@ -275,7 +279,7 @@ seconds, which allows changing Kerberos tickets and identities frequently.
 The default is no explicit timeout, which means the kernel context will live
 the lifetime of the Kerberos service ticket used in its creation.
 .TP
-.B -T timeout
+.BI "-T " timeout
 Timeout, in seconds, to create an RPC connection with a server while
 establishing an authenticated gss context for a user.
 The default timeout is set to 5 seconds.
@@ -283,6 +287,18 @@ If you get messages like "WARNING: can't create tcp rpc_clnt to server
 %servername% for user with uid %uid%: RPC: Remote system error -
 Connection timed out", you should consider an increase of this timeout.
 .TP
+.BI "-U " timeout
+Timeout, in seconds, for upcall threads.  Threads executing longer than
+.I timeout
+seconds will be cancelled and an error of -ETIMEDOUT will be reported
+to the kernel.  The default
+.I timeout
+is 30 seconds.  The minimum is 5 seconds.  The maximum is 600 seconds.
+.TP
+.B -C
+Instead of cancelling upcall threads that have timed out, cause an error
+message to be logged to the syslog (no error will be reported to the kernel).
+.TP
 .B -H
 Avoids setting $HOME to "/". This allows rpc.gssd to read per user k5identity
 files versus trying to read /.k5identity for each user.
@@ -350,6 +366,17 @@ Equivalent to
 Equivalent to
 .BR -R .
 .TP
+.B upcall-timeout
+Equivalent to
+.BR -U .
+.TP
+.B cancel-timed-out-upcalls
+Setting to
+.B false
+is equivalent to providing the
+.B -C
+flag.
+.TP
 .B set-home
 Setting to
 .B false
utils/gssd/gssd_proc.c +114 -24

250 lines not on this page · raw message

Signature · 1 line
2.30.2