Skip to contentSkip to conversation

add principal to the data being tracked by nfsdcld

RFC PATCH Superseded Scott Mayhew 1 participant 5 messages Aug 8, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover add principal to the data being tracked by nfsdcld

At the spring bakeathon, Chuck suggested that we should store the kerberos principal in addition to the client id string in nfsdcld. The idea is to prevent an illegitimate client from reclaiming another client's opens by supplying that client's id string. This is an initial attempt at doing that.

The first patch adds support for a "GetVersion" upcall which allows nfsd to determine the maximum message version that nfsdcld supports. Right now it's based on the value of CLD_UPCALL_VERSION from cld.h, but I was thinking we may wish to add a command-line option (and an nfs.conf) option to make it possible to use a lower version than CLD_UPCALL_VERSION. My thinking here is that an older nfsdcld daemon won't be compatible with the new database schema... rather than worrying about messing with downgrading the database, just use the command-line option to make it behave like an older daemon.

The second patch adds handling for the v2 Cld_Create and Cld_GraceStart upcalls, which can include the kerberos principal which we'll store along with the client id string in the database. Note that if we're talking to an old kernel that does the v1 upcall, everything still works (we just ignore the new columns in the database).

The third patch adds a tool for manipulating nfsdcld's database schema. It's mostly intended to be used to downgrade the database in the (hopefully rare) event that an admin would want to downgrade nfsdcld. It also provides the ability for fixing broken recovery table names (which nfsdcld also fixes automatically) as well as the ability to print the contents of the database.

The final patch updates the nfsdcld man page.

Questions:
1. Why do we have a copy of cld.h in support/include? It seems unnecessary... maybe we should get rid of it so that we're always using the cld.h from the kernel headers?
2. Should there be a command line option to allow nfsdcld to advertise a lower upcall version to nfsd in the GetVersion upcall reply?

Changes since v1:
- added a tool for manipulating nfsdcld's sqlite database schema
- updated the nfsdcld man page

Scott Mayhew (4):
nfsdcld: add a "GetVersion" upcall
nfsdcld: add support for upcall version 2
Add a tool for manipulating the nfsdcld sqlite database schema.
nfsdcld: update nfsdcld.man

 configure.ac                    |   1 +
 support/include/cld.h           |  37 ++++-
 tools/Makefile.am               |   4 +
 tools/clddb-tool/Makefile.am    |  13 ++
 tools/clddb-tool/clddb-tool.man |  83 ++++++++++
 tools/clddb-tool/clddb-tool.py  | 261 ++++++++++++++++++++++++++++++++
 utils/nfsdcld/cld-internal.h    |  13 +-
 utils/nfsdcld/nfsdcld.c         | 140 ++++++++++++++---
 utils/nfsdcld/nfsdcld.man       |  32 +++-
 utils/nfsdcld/sqlite.c          | 238 ++++++++++++++++++++++++-----
 utils/nfsdcld/sqlite.h          |   2 +
 11 files changed, 755 insertions(+), 69 deletions(-)
 create mode 100644 tools/clddb-tool/Makefile.am
 create mode 100644 tools/clddb-tool/clddb-tool.man
 create mode 100644 tools/clddb-tool/clddb-tool.py
Signature · 1 line
2.17.2
#1 lore raw

RFC PATCH v2 1/4 nfsdcld: add a "GetVersion" upcall

Add a "GetVersion" upcall to allow the kernel to determine the maximum upcall version that nfsdcld supports.

---
 support/include/cld.h        | 11 ++++++++-
 utils/nfsdcld/cld-internal.h |  4 ++-
 utils/nfsdcld/nfsdcld.c      | 47 ++++++++++++++++++++++++++++++------
 utils/nfsdcld/sqlite.c       |  2 +-
 4 files changed, 53 insertions(+), 11 deletions(-)
support/include/cld.h +10 -1
index c1f5b70..00a40da 100644
@@ -33,7 +33,8 @@ enum cld_command {
 	Cld_Remove,		/* remove record of this cm_id */
 	Cld_Check,		/* is this cm_id allowed? */
 	Cld_GraceDone,		/* grace period is complete */
-	Cld_GraceStart,
+	Cld_GraceStart,		/* grace start (upload client records) */
+	Cld_GetVersion,		/* query max supported upcall version */
 };
 
 /* representation of long-form NFSv4 client ID */
@@ -51,7 +52,15 @@ struct cld_msg {
 	union {
 		int64_t		cm_gracetime;	/* grace period start time */
 		struct cld_name	cm_name;
+		uint8_t		cm_version;	/* for getting max version */
 	} __attribute__((packed)) cm_u;
 } __attribute__((packed));
 
+struct cld_msg_hdr {
+	uint8_t		cm_vers;		/* upcall version */
+	uint8_t		cm_cmd;			/* upcall command */
+	int16_t		cm_status;		/* return code */
+	uint32_t	cm_xid;			/* transaction id */
+} __attribute__((packed));
+
 #endif /* !_NFSD_CLD_H */
utils/nfsdcld/cld-internal.h +3 -1
index 76e97db..f33cb04 100644
@@ -21,7 +21,9 @@
 struct cld_client {
 	int			cl_fd;
 	struct event		cl_event;
-	struct cld_msg	cl_msg;
+	union {
+		struct cld_msg	cl_msg;
+	} cl_u;
 };
 
 uint64_t current_epoch;
utils/nfsdcld/nfsdcld.c +39 -8

106 lines not on this page · raw message

utils/nfsdcld/sqlite.c +1 -1
index fa81df8..6525fc1 100644
@@ -1152,7 +1152,7 @@ sqlite_iterate_recovery(int (*cb)(struct cld_client *clnt), struct cld_client *c
 {
 	int ret;
 	sqlite3_stmt *stmt = NULL;
-	struct cld_msg *cmsg = &clnt->cl_msg;
+	struct cld_msg *cmsg = &clnt->cl_u.cl_msg;
 
 	if (recovery_epoch == 0) {
 		xlog(D_GENERAL, "%s: not in grace!", __func__);
Signature · 1 line
2.17.2
#2 lore raw

RFC PATCH v2 2/4 nfsdcld: add support for upcall version 2

In reply to Scott Mayhew

Version 2 upcalls will allow the nfsd to include the kerberos principal string (actually the first 1024 bytes of it) in the Cld_Create upcall. If present, the principal will be stored along with the client id string in the database, and will be included in the Cld_GraceStart downcall whenever nfsd restarts.

---
 support/include/cld.h        |  26 +++-
 utils/nfsdcld/cld-internal.h |  11 +-
 utils/nfsdcld/nfsdcld.c      |  97 +++++++++++---
 utils/nfsdcld/sqlite.c       | 236 +++++++++++++++++++++++++++++------
 utils/nfsdcld/sqlite.h       |   2 +
 5 files changed, 316 insertions(+), 56 deletions(-)
support/include/cld.h +25 -1
index 00a40da..c58efda 100644
@@ -23,7 +23,7 @@
 #define _NFSD_CLD_H
 
 /* latest upcall version available */
-#define CLD_UPCALL_VERSION 1
+#define CLD_UPCALL_VERSION 2
 
 /* defined by RFC3530 */
 #define NFS4_OPAQUE_LIMIT 1024
@@ -43,6 +43,17 @@ struct cld_name {
 	unsigned char	cn_id[NFS4_OPAQUE_LIMIT];	/* client-provided */
 } __attribute__((packed));
 
+/* principal of the form servicetype@hostname */
+struct cld_principal {
+	uint16_t	cp_len;				/* length of cp_data */
+	unsigned char	cp_data[NFS4_OPAQUE_LIMIT];	/* princ from cred */
+} __attribute__((packed));
+
+struct cld_clntinfo {
+	struct cld_name		cc_name;
+	struct cld_principal	cc_principal;
+} __attribute__((packed));
+
 /* message struct for communication with userspace */
 struct cld_msg {
 	uint8_t		cm_vers;		/* upcall version */
@@ -56,6 +67,19 @@ struct cld_msg {
 	} __attribute__((packed)) cm_u;
 } __attribute__((packed));
 
+/* version 2 message includes the principal */
+struct cld_msg_v2 {
+	uint8_t		cm_vers;		/* upcall version */
+	uint8_t		cm_cmd;			/* upcall command */
+	int16_t		cm_status;		/* return code */
+	uint32_t	cm_xid;			/* transaction id */
+	union {
+		struct cld_name	cm_name;
+		uint8_t		cm_version;	/* for getting max version */
+		struct cld_clntinfo cm_clntinfo; /* name & princ */
+	} __attribute__((packed)) cm_u;
+} __attribute__((packed));
+
 struct cld_msg_hdr {
 	uint8_t		cm_vers;		/* upcall version */
 	uint8_t		cm_cmd;			/* upcall command */
utils/nfsdcld/cld-internal.h +10 -1
index f33cb04..05f01be 100644
@@ -18,11 +18,20 @@
 #ifndef _CLD_INTERNAL_H_
 #define _CLD_INTERNAL_H_
 
+#if CLD_UPCALL_VERSION >= 2
+#define UPCALL_VERSION		2
+#else
+#define UPCALL_VERSION		1
+#endif
+
 struct cld_client {
 	int			cl_fd;
 	struct event		cl_event;
 	union {
-		struct cld_msg	cl_msg;
+		struct cld_msg		cl_msg;
+#if UPCALL_VERSION >= 2
+		struct cld_msg_v2	cl_msg_v2;
+#endif
 	} cl_u;
 };
 
utils/nfsdcld/nfsdcld.c +79 -18

222 lines not on this page · raw message

utils/nfsdcld/sqlite.c +200 -36

368 lines not on this page · raw message

utils/nfsdcld/sqlite.h +2 -0
index 7741382..aa52189 100644
@@ -24,6 +24,8 @@ struct cld_client;
 
 int sqlite_prepare_dbh(const char *topdir);
 int sqlite_insert_client(const unsigned char *clname, const size_t namelen);
+int sqlite_insert_client_and_princ(const unsigned char *clname, const size_t namelen,
+		const unsigned char *clprinc, const size_t princlen);
 int sqlite_remove_client(const unsigned char *clname, const size_t namelen);
 int sqlite_check_client(const unsigned char *clname, const size_t namelen);
 int sqlite_grace_start(void);
Signature · 1 line
2.17.2
#3 lore raw

RFC PATCH v2 3/4 Add a tool for manipulating the nfsdcld sqlite database schema.

In reply to Scott Mayhew

The "clddb-tool" is mainly for downgrading the nfsdcld sqlite database schema in the event that an admin wants to downgrade nfsdcld. It also provides options for fixing corrupt table names (note newer versions of nfsdcld take care of this automatically) and for printing the contents of the database.

---
 configure.ac                    |   1 +
 tools/Makefile.am               |   4 +
 tools/clddb-tool/Makefile.am    |  13 ++
 tools/clddb-tool/clddb-tool.man |  83 ++++++++++
 tools/clddb-tool/clddb-tool.py  | 261 ++++++++++++++++++++++++++++++++
 5 files changed, 362 insertions(+)
 create mode 100644 tools/clddb-tool/Makefile.am
 create mode 100644 tools/clddb-tool/clddb-tool.man
 create mode 100644 tools/clddb-tool/clddb-tool.py
configure.ac +1 -0
index 50002b4..954abfd 100644
@@ -652,6 +652,7 @@ AC_CONFIG_FILES([
 	tools/mountstats/Makefile
 	tools/nfs-iostat/Makefile
 	tools/nfsconf/Makefile
+        tools/clddb-tool/Makefile
 	utils/Makefile
 	utils/blkmapd/Makefile
 	utils/nfsdcld/Makefile
tools/Makefile.am +4 -0
index 4266da4..53e6117 100644
@@ -8,6 +8,10 @@ endif
 
 OPTDIRS += nfsconf
 
+if CONFIG_NFSDCLD
+OPTDIRS += clddb-tool
+endif
+
 SUBDIRS = locktest rpcdebug nlmtest mountstats nfs-iostat $(OPTDIRS)
 
 MAINTAINERCLEANFILES = Makefile.in
tools/clddb-tool/Makefile.am new file +13 -0
new file mode 100644
index 0000000..15a8fd4
@@ -0,0 +1,13 @@
+## Process this file with automake to produce Makefile.in
+PYTHON_FILES =  clddb-tool.py
+
+man8_MANS	= clddb-tool.man
+
+EXTRA_DIST	= $(man8_MANS) $(PYTHON_FILES)
+
+all-local: $(PYTHON_FILES)
+
+install-data-hook:
+	$(INSTALL) -m 755 clddb-tool.py $(DESTDIR)$(sbindir)/clddb-tool
+
+MAINTAINERCLEANFILES=Makefile.in
tools/clddb-tool/clddb-tool.man new file +83 -0

83 lines not on this page · raw message

tools/clddb-tool/clddb-tool.py new file +261 -0

261 lines not on this page · raw message

Signature · 1 line
2.17.2
#4 lore raw

RFC PATCH v2 4/4 nfsdcld: update nfsdcld.man

In reply to Scott Mayhew

Added some historical information to the notes section, along with some information regarding upgrading and downgrading nfsdcld.

---
 utils/nfsdcld/nfsdcld.man | 32 +++++++++++++++++++++++++++-----
 1 file changed, 27 insertions(+), 5 deletions(-)
utils/nfsdcld/nfsdcld.man +27 -5
index c271d14..4c2b1e8 100644
@@ -185,15 +185,37 @@ on stable storage by manipulating information on the filesystem
 directly, in the directory to which \fI/proc/fs/nfsd/nfsv4recoverydir\fR
 points.
 .PP
-This daemon requires a kernel that supports the nfsdcld upcall. If the
-kernel does not support the new upcall, or is using the legacy client
-name tracking code then it will not create the pipe that nfsdcld uses to
-talk to the kernel.
+This changed with the original introduction of \fBnfsdcld\fR upcall in kernel version 3.4,
+which was later deprecated in favor of the \fBnfsdcltrack\fR(8) usermodehelper
+program, support for which was added in kernel version 3.8.  However, since the
+usermodehelper upcall does not work in containers, support for a new version of
+the \fBnfsdcld\fR upcall was added in kernel version 5.2.
+.PP
+This daemon requires a kernel that supports the \fBnfsdcld\fR upcall. On older kernels, if
+the legacy client name tracking code was in use, then the kernel would not create the
+pipe that \fBnfsdcld\fR uses to talk to the kernel.  On newer kernels, nfsd attempts to
+initialize client tracking in the following order:  First, the \fBnfsdcld\fR upcall.  Second,
+the \fBnfsdcltrack\fR usermodehelper upcall.  Finally, the legacy client tracking.
 .PP
 This daemon should be run as root, as the pipe that it uses to communicate
 with the kernel is only accessable by root. The daemon however does drop all
 superuser capabilities after starting. Because of this, the \fIstoragedir\fR
 should be owned by root, and be readable and writable by owner.
+.PP
+The daemon now supports different upcall versions to allow the kernel to pass additional
+data to be stored in the on-disk database.  The kernel will query the supported upcall
+version from \fBnfsdcld\fR during client tracking initialization.  A restart of \fBnfsd\fR is
+not necessary after upgrading \fBnfsdcld\fR, however \fBnfsd\fR will not use a later upcall
+version until restart.  A restart of \fBnfsd is necessary\fR after downgrading \fBnfsdcld\fR,
+to ensure that \fBnfsd\fR does not use an upcall version that \fBnfsdcld\fR does not support.
+Additionally, a downgrade of \fBnfsdcld\fR requires the schema of the on-disk database to
+be downgraded as well.  That can be accomplished using the \fBclddb-tool\fR(8) utility.
+.SH FILES
+.TP
+.B /var/lib/nfs/nfsdcld/main.sqlite
+.SH SEE ALSO
+.BR nfsdcltrack "(8), " clddb-tool (8)
 .SH "AUTHORS"
 .IX Header "AUTHORS"
-The nfsdcld daemon was developed by Jeff Layton <[email protected]>.
+The nfsdcld daemon was developed by Jeff Layton <[email protected]>
+with modifications from Scott Mayhew <[email protected]>.
Signature · 1 line
2.17.2