Skip to contentSkip to conversation

new statd [take 2]

PATCH In review Chuck Lever 7 participants 39 messages Aug 5, 2009 – Sep 15, 2009 9 messages not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover new statd [take 2]

Apparently there is a 100KB per-email size limit imposed by vger.kernel.org. I've broken up the large statd patch, and postponed the nfs(8) man page patches to another day.

Yesterday it was decided that:

[PATCH 3/5] statd: change default state dir to /var/lib/nfs/statd

was overkill. That one has been dropped from this series.

The first two of the present patches introduce the .c files under utils/new-statd/ . The next patch adds actools infrastructure for building the new statd and sm-notify executables. The last patch adds two new man pages for statd and sm-notify.

The old statd remains in utils/statd. If sqlite3 and TI-RPC is present on the build system, and TI-RPC is enabled during configuration, the new statd is built and installed instead of the old one.

See the description in the first patch for more details on the features of the new implementation.

---

Chuck Lever (4):
nfs-utils: introduce new statd implementation (4th part)
nfs-utils: introduce new statd implementation (3rd part)
nfs-utils: introduce new statd implementation (2nd part)
nfs-utils: introduce new statd implementation (1st part)

 .gitignore                    |    4 
 Makefile.am                   |    4 
 aclocal/libcap.m4             |   15 +
 aclocal/libsqlite3.m4         |   33 ++
 aclocal/libtirpc.m4           |    2 
 configure.ac                  |   12 +
 utils/Makefile.am             |    7 
 utils/new-statd/Makefile.am   |  102 +++++
 utils/new-statd/file.c        |  779 ++++++++++++++++++++++++++++++++++++++
 utils/new-statd/hostname.c    |  520 +++++++++++++++++++++++++
 utils/new-statd/main.c        |  319 ++++++++++++++++
 utils/new-statd/nlmcall.c     |  525 ++++++++++++++++++++++++++
 utils/new-statd/sm-notify.c   |  155 ++++++++
 utils/new-statd/sm-notify.man |  286 ++++++++++++++
 utils/new-statd/sm_inter.x    |  107 +++++
 utils/new-statd/smncall.c     |  774 ++++++++++++++++++++++++++++++++++++++
 utils/new-statd/start-statd   |   12 +
 utils/new-statd/statd.h       |  135 +++++++
 utils/new-statd/statd.man     |  388 +++++++++++++++++++
 utils/new-statd/svc.c         |  841 +++++++++++++++++++++++++++++++++++++++++
 utils/statd/Makefile.am       |    9 
 21 files changed, 5024 insertions(+), 5 deletions(-)
 create mode 100644 aclocal/libcap.m4
 create mode 100644 aclocal/libsqlite3.m4
 create mode 100644 utils/new-statd/Makefile.am
 create mode 100644 utils/new-statd/file.c
 create mode 100644 utils/new-statd/hostname.c
 create mode 100644 utils/new-statd/main.c
 create mode 100644 utils/new-statd/nlmcall.c
 create mode 100644 utils/new-statd/sm-notify.c
 create mode 100644 utils/new-statd/sm-notify.man
 create mode 100644 utils/new-statd/sm_inter.x
 create mode 100644 utils/new-statd/smncall.c
 create mode 100644 utils/new-statd/start-statd
 create mode 100644 utils/new-statd/statd.h
 create mode 100644 utils/new-statd/statd.man
 create mode 100644 utils/new-statd/svc.c
Signature · 1 line
Signature

A message that is not in the archive (sent off-list or not yet received): 20090805143550.12866.8377.stgit-RytpoXr2tKZ9HhUboXbp9zCvJB+x5qRC@public.gmane.org

#2 lore raw

PATCH 1/4 nfs-utils: introduce new statd implementation (1st part)

Provide a new implementation of statd that supports IPv6. The new statd implementation resides under

utils/new-statd/

The contents of this directory are built if --enable-tirpc is set on the ./configure command line, and sqlite3 is available on the build system. Otherwise, the legacy version of statd, which still resides under utils/statd/, is built.

The goals of this re-write are:

o Support IPv6 networking

Support interoperation with TI-RPC-based NSM implementations. Transport Independent RPC, or TI-RPC, provides IPv6 network support for Linux's NSM implementation.

To support TI-RPC, open code to construct RPC requests in socket buffers and then schedule them has been replaced with standard library calls.

o Support notification via TCP

As a secondary benefit of using TI-RPC library calls, reboot notifications and NLM callbacks can now be sent via connection- oriented transport protocols.

Note that lockd does not (yet) tell statd what transport protocol to use when sending reboot notifications. statd/sm-notify will continue to use UDP for the time being.

o Use an embedded database for storing on-disk callback data

This whole exercise is for the purpose of crash robustness. There are well-known deficiencies with simple create/rename/unlink disk storage schemes during system crashes. Replace the current flat-file monitor list mechanism which uses sync(2) with sqlite3, which uses fsync(3).

o Share code between sm-notify and statd

Statd and sm-notify access the same set of on-disk data. These separate programs now share the same code and implementation, with access to on-disk data serialized by sqlite3. The two remain separate executables to allow other system facilities to send reboot notifications without poking statd.

o Reduce impact of DNS outages

The heuristics used by SM_NOTIFY to figure out which remote peer has rebooted are heavily dependent on DNS. If the DNS service is slow or hangs, that will make the NSM listener unresponsive. Incoming SM_NOTIFY requests are now handled in a sidecar process to reduce the impact of DNS outages on the NSM service listener.

o Proper my_name support

The current version of statd uses gethostname(3) to generate the mon_name argument of SM_NOTIFY. This value can change across a reboot. The new version of statd records lockd's my_name, passed by every SM_MON request, and uses that when sending SM_NOTIFY.

This can be useful for multi-homed and DHCP configured hosts.

o Send SM_NOTIFY more aggressively

It has been recommended that statd/sm-notify send SM_NOTIFY more aggressively (for example, to the entire list returned by getaddrinfo(3)). Since SM_NOTIFY's reply is NULL, there's no way to tell whether the remote peer recognized the mon_name we sent. More study is required, but this implementation attempts to send an SM_NOTIFY request to each address returned by getaddrinfo(3).

This re-implementation paves the way for a number of future improvements. However, it does not immediately address:

o lockd/statd start-up serialization issues

Sending reboot notifications, starting statd and lockd, and opening the lockd grace period are still determined independently in user space and the kernel.

o Binding mon_names to caller IP addresses

By default, lockd continues to send IP addresses as the mon_name argument of the SM_MON procedure. This provides a better guarantee of being able to contact remote peers during a reboot, but means statd must continue to use heuristics to match incoming SM_NOTIFY requests with peers on the monitor list.

o Distinct logic for NFS client- and server-side

Client-side and server-side monitoring requirements are different. Statd continues to use the same logic for both NFS client and server, as the NSMv1 protocol does not provide any indication that a mon_name is for a client or server peer.

---

 utils/new-statd/file.c     |  779 ++++++++++++++++++++++++++++++++++++++++++++
 utils/new-statd/hostname.c |  520 +++++++++++++++++++++++++++++
 utils/new-statd/nlmcall.c  |  525 ++++++++++++++++++++++++++++++
 3 files changed, 1824 insertions(+), 0 deletions(-)
 create mode 100644 utils/new-statd/file.c
 create mode 100644 utils/new-statd/hostname.c
 create mode 100644 utils/new-statd/nlmcall.c
utils/new-statd/file.c new file +779 -0
new file mode 100644
index 0000000..db54bf7
@@ -0,0 +1,779 @@
+/*
+ * Copyright 2009 Oracle.  All rights reserved.
+ *
+ * This file is part of nfs-utils.
+ *
+ * nfs-utils is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * nfs-utils is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with nfs-utils.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+/*
+ * NSM for Linux.
+ *
+ * Callback information is stored in an sqlite database that provides
+ * transactional semantics, to preserve state in the event of a system
+ * crash, and can easily be updated to store additional data elements
+ * without need for versioning
+ */
+
+#ifdef HAVE_CONFIG_H
+#include <config.h>
+#endif
+
+#include <sys/types.h>
+#include <sys/capability.h>
+#include <sys/prctl.h>
+#include <sys/stat.h>
+#include <sys/ipc.h>
+#include <sys/sem.h>
+
+#include <stdint.h>
+#include <unistd.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <dirent.h>
+#include <grp.h>
+#include <arpa/inet.h>
+
+#include "statd.h"
+
+#ifdef NFS_STATEDIR
+#define STATD_DEFAULT_STATEDIR		NFS_STATEDIR "/statd"
+#else	/* !defined(NFS_STATEDIR) */
+#define STATD_DEFAULT_STATEDIR		"/var/lib/nfs/statd"
+#endif	/* !defined(NFS_STATEDIR) */
+
+#define STATD_DATABASE_FILE		"statdb"
+
+static char statd_db_dirname[PATH_MAX];
+static char statd_db_filename[PATH_MAX];
+
+/**
+ * statd_open_db - open a database connection to statd's sqlite3 database
+ * @flags: flags passed to sqlite3_open_v2
+ *
+ * Returns an open database connection; otherwise NULL is returned if
+ * an error occurred.
+ */
+sqlite3 *
+statd_open_db(int flags)
+{
+	sqlite3 *db;
+	int rc;
+
+	rc = sqlite3_open_v2(statd_db_filename, &db, flags, NULL);
+	if (rc != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to open database in %s: %s",
+				statd_db_filename, sqlite3_errmsg(db));
+		xlog(L_ERROR, "Check that the state directory pathname "
+				"is correct and has proper permissions");
+		(void)sqlite3_close(db);
+		return NULL;
+	}
+
+	/*
+	 * Retry SQLITE_BUSY for 100 msec before returning an error.
+	 */
+	(void)sqlite3_busy_timeout(db, 100);
+
+	return db;
+}
+
+/**
+ * statd_close_db - close an open sqlite3 database connection
+ * @db: open sqlite3 database connection descriptor
+ *
+ */
+void
+statd_close_db(sqlite3 *db)
+{
+	if (sqlite3_close(db) != SQLITE_OK)
+		xlog(L_ERROR, "Failed to close database: %s",
+				sqlite3_errmsg(db));
+}
+
+/**
+ * statd_prepare_stmt - prepare a C string SQL statement
+ * @db: open database connection
+ * @stmt: OUT: pointer to prepared SQL statement object
+ * @sql: C string containing SQL statement to prepare
+ *
+ * Returns TRUE if the statement was prepared successfully; otherwise
+ * FALSE is returned and an error is logged.
+ */
+bool_t
+statd_prepare_stmt(sqlite3 *db, sqlite3_stmt **stmt, const char *sql)
+{
+	int rc;
+
+	rc = sqlite3_prepare_v2(db, sql, -1, stmt, NULL);
+	if (rc != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to compile SQL: %s",
+				sqlite3_errmsg(db));
+		xlog(L_ERROR, "SQL: %s", sql);
+		return FALSE;
+	}
+	return TRUE;
+}
+
+/**
+ * statd_finalize_stmt - release a prepared SQL statement
+ * @stmt: sqlite3_stmt to release
+ *
+ */
+void
+statd_finalize_stmt(sqlite3_stmt *stmt)
+{
+	sqlite3 *db = sqlite3_db_handle(stmt);
+	int rc;
+
+	rc = sqlite3_finalize(stmt);
+	switch(rc) {
+	case SQLITE_OK:
+	case SQLITE_ABORT:
+	case SQLITE_CONSTRAINT:
+		break;
+	default:
+		xlog(L_ERROR, "Failed to finalize SQL statement: %s",
+				sqlite3_errmsg(db));
+	}
+}
+
+/**
+ * statd_begin_transaction - start a sqlite3 transaction
+ * @db: open database connection
+ *
+ * Returns TRUE if the transaction was started; otherwise FALSE is
+ * returned if sqlite3 reported an error.
+ */
+bool_t
+statd_begin_transaction(sqlite3 *db)
+{
+	char *err_msg;
+	int rc;
+
+	err_msg = NULL;
+	rc = sqlite3_exec(db, "BEGIN IMMEDIATE TRANSACTION;", NULL, 0, &err_msg);
+	if (rc != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to start transaction: %s", err_msg);
+		sqlite3_free(err_msg);
+		return FALSE;
+	}
+
+	xlog(D_CALL, "Transaction started");
+	return TRUE;
+}
+
+/**
+ * statd_end_transaction - commit a sqlite3 transaction
+ * @db: open database connection
+ *
+ */
+void
+statd_end_transaction(sqlite3 *db)
+{
+	char *err_msg;
+	int rc;
+
+	err_msg = NULL;
+	rc = sqlite3_exec(db, "COMMIT TRANSACTION;", NULL, 0, &err_msg);
+	if (rc != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to commit transaction: %s", err_msg);
+		sqlite3_free(err_msg);
+		return;
+	}
+
+	xlog(D_CALL, "Transaction committed");
+}
+
+/**
+ * statd_rollback_transaction - rollback an active sqlite3 transaction
+ * @db: open database connection
+ *
+ */
+void
+statd_rollback_transaction(sqlite3 *db)
+{
+	char *err_msg;
+	int rc;
+
+	err_msg = NULL;
+	rc = sqlite3_exec(db, "ROLLBACK TRANSACTION;", NULL, 0, &err_msg);
+	if (rc != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to roll back transaction: %s", err_msg);
+		sqlite3_free(err_msg);
+		return;
+	}
+
+	xlog(D_CALL, "Transaction rolled back");
+}
+
+/**
+ * statd_check_pathname - set up pathname
+ * @parentdir: C string containing pathname to on-disk state
+ *
+ * Returns TRUE if pathname was valid; otherwise FALSE is returned.
+ */
+bool_t
+statd_check_pathname(const char *parentdir)
+{
+	struct stat st;
+	int len;
+
+	if (parentdir != NULL) {
+		if (strlen(parentdir) >= PATH_MAX) {
+			xlog(L_ERROR, "Invalid directory name");
+			return FALSE;
+		}
+	} else
+		parentdir = STATD_DEFAULT_STATEDIR;
+
+	if (lstat(parentdir, &st) == -1) {
+		xlog(L_ERROR, "Failed to stat %s: %m", parentdir);
+		return FALSE;
+	}
+
+	if (chdir(parentdir) == -1) {
+		xlog(L_ERROR, "Failed to change working directory to %s: %m",
+				parentdir);
+		return FALSE;
+	}
+
+	strncpy(statd_db_dirname, parentdir, sizeof(statd_db_dirname));
+
+	len = snprintf(statd_db_filename, sizeof(statd_db_filename),
+			"%s/%s", parentdir, STATD_DATABASE_FILE);
+	if (__error_check(len, sizeof(statd_db_filename))) {
+		xlog(L_ERROR, "Illegal directory name");
+		return FALSE;
+	}
+
+	return TRUE;
+}
+
+/*
+ * Returns TRUE if info table creation was successful, or if
+ * info table already exists; otherwise FALSE is returned if
+ * any error occurs.
+ */
+static bool_t
+statd_create_info_table(sqlite3 *db)
+{
+	sqlite3_stmt *stmt;
+	bool_t result;
+	int rc;
+
+	result = FALSE;
+
+	rc = sqlite3_prepare_v2(db, "CREATE TABLE " STATD_INFO_TABLENAME
+					" (name TEXT PRIMARY KEY,"
+					" value INTEGER);",
+						-1, &stmt, NULL);
+	switch (rc) {
+	case SQLITE_OK:
+		rc = sqlite3_step(stmt);
+		if (rc != SQLITE_DONE) {
+			xlog(L_ERROR, "Failed to create table '"
+					STATD_INFO_TABLENAME "': %s",
+					sqlite3_errmsg(db));
+			break;
+		}
+
+		result = TRUE;
+		xlog(D_GENERAL, "Table '" STATD_INFO_TABLENAME
+				"' created successfully");
+		break;
+	case SQLITE_ERROR:
+		result = TRUE;
+		xlog(D_GENERAL, "Table '" STATD_INFO_TABLENAME "' already exists");
+		goto out;
+	default:
+		xlog(L_ERROR, "Failed to compile SQL: %s",
+				sqlite3_errmsg(db));
+		xlog(L_ERROR, "SQL: %s");
+		goto out;
+	}
+
+	statd_finalize_stmt(stmt);
+
+out:
+	return result;
+}
+
+/*
+ * Returns TRUE if the monitor table exists, or was created; otherwise
+ * FALSE is returned if an error occurs.
+ *
+ * This could be done in svc.c.  Since sm-notify links file.o, that
+ * would mean it would have to link svc.c as well.
+ */
+static bool_t
+statd_create_monitor_table(sqlite3 *db)
+{
+	sqlite3_stmt *stmt;
+	bool_t result;
+	int rc;
+
+	result = FALSE;
+
+	rc = sqlite3_prepare_v2(db, "CREATE TABLE " STATD_MONITOR_TABLENAME
+					" (priv BLOB,"
+					" mon_name TEXT NOT NULL,"
+					" my_name TEXT NOT NULL,"
+					" program INTEGER,"
+					" version INTEGER,"
+					" procedure INTEGER,"
+					" protocol TEXT NOT NULL,"
+					" state INTEGER,"
+					" UNIQUE(mon_name,my_name));",
+						-1, &stmt, NULL);
+	switch (rc) {
+	case SQLITE_OK:
+		rc = sqlite3_step(stmt);
+		if (rc != SQLITE_DONE) {
+			xlog(L_ERROR, "Failed to create table '"
+					STATD_MONITOR_TABLENAME "': %s",
+					sqlite3_errmsg(db));
+			break;
+		}
+
+		result = 1;
+		xlog(D_GENERAL, "Table '" STATD_MONITOR_TABLENAME
+				"' created successfully");
+		break;
+	case SQLITE_ERROR:
+		result = TRUE;
+		xlog(D_GENERAL, "Table '" STATD_MONITOR_TABLENAME
+				"' already exists");
+		goto out;
+	default:
+		xlog(L_ERROR, "Failed to compile SQL: %s",
+				sqlite3_errmsg(db));
+		xlog(L_ERROR, "SQL: %s");
+		goto out;
+	}
+
+	statd_finalize_stmt(stmt);
+
+out:
+	return result;
+}
+
+/*
+ * Ensure database file exists.  This should create the file
+ * with the proper ownership.  Also ensure tables are created.
+ *
+ * XXX: Is it enough to set journal_mode and synchronous just once?
+ */
+static bool_t
+statd_init_database(void)
+{
+	bool_t result;
+	char *err_msg;
+	sqlite3 *db;
+	int rc;
+
+	xlog(D_CALL, "Initializing database");
+
+	result = FALSE;
+	db = statd_open_db(SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE);
+	if (db == NULL)
+		goto out;
+
+	/*
+	 * Don't delete the journal file after each transaction.
+	 * This provides better performance and crash robustness.
+	 */
+	err_msg = NULL;
+	rc = sqlite3_exec(db, "PRAGMA journal_mode=TRUNCATE;", NULL, 0, &err_msg);
+	if (rc != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to enable persistent journaling: %s",
+				err_msg);
+		sqlite3_free(err_msg);
+		goto out_close;
+	}
+
+	/*
+	 * Protect against db corruption during crashes.
+	 */
+	err_msg = NULL;
+	rc = sqlite3_exec(db, "PRAGMA synchronous=FULL;", NULL, 0, &err_msg);
+	if (rc != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to enable full synchronous mode: %s",
+				err_msg);
+		sqlite3_free(err_msg);
+		goto out_close;
+	}
+
+	if (!statd_create_info_table(db))
+		goto out_close;
+
+	if (!statd_create_monitor_table(db))
+		goto out_close;
+
+	result = TRUE;
+
+out_close:
+	statd_close_db(db);
+
+out:
+	return result;
+}
+
+/*
+ * Clear all capabilities but CAP_NET_BIND_SERVICE.  This permits
+ * statd to acquire privileged source ports, but all other
+ * capabilities are disallowed.
+ */
+static bool_t
+statd_clear_capabilities(void)
+{
+	bool_t result;
+	cap_t caps;
+
+	result = FALSE;
+
+	caps = cap_from_text("cap_net_bind_service=ep");
+	if (caps == NULL) {
+		xlog(L_ERROR, "Failed to allocate working storage: %m");
+		return result;
+	}
+
+	if (cap_set_proc(caps) == -1) {
+		xlog(L_ERROR, "Failed to set capability flags: %m");
+		goto out_free;
+	}
+
+	result = TRUE;
+
+out_free:
+	(void)cap_free(caps);
+	return result;
+}
+
+/**
+ * statd_drop_privileges - drop root privileges
+ * @keep_bind: caller sets this to keep CAP_NET_BIND_SERVICE
+ *
+ * Returns TRUE if successful, or FALSE if some error occurred.
+ *
+ * Set our effective UID and GID to that of our on-disk database.
+ */
+bool_t
+statd_drop_privileges(const int keep_bind)
+{
+	struct stat st;
+
+	/*
+	 * XXX: If we can't stat dirname, or if dirname is owned by
+	 *      root, we should use "statduser" instead, which is set up
+	 *      by configure.ac
+	 *
+	 *      Nothing in nfs-utils seems to use statduser, though.
+	 */
+	if (lstat(statd_db_dirname, &st) == -1) {
+		xlog(L_ERROR, "Failed to stat %s: %m", statd_db_dirname);
+		return FALSE;
+	}
+
+	if (st.st_uid == 0) {
+		xlog(L_WARNING,
+			"Running as root. chown %s to choose different user",
+				statd_db_dirname);
+		return TRUE;
+	}
+
+        /*
+	 * Don't clear capabilities when dropping root.
+	 */
+        if (keep_bind && prctl(PR_SET_KEEPCAPS, 1, 0, 0, 0) == -1) {
+                xlog(L_ERROR, "prctl(PR_SET_KEEPCAPS) failed: %m");
+		return FALSE;
+	}
+
+	if (setgroups(0, NULL) == -1) {
+		xlog(L_ERROR, "Failed to drop supplementary groups: %m");
+		return FALSE;
+	}
+
+	/*
+	 * ORDER
+	 *
+	 * setgid(2) first, as setuid(2) may remove privileges needed
+	 * to set the group id.
+	 */
+	if (setgid(st.st_gid) == -1 || setuid(st.st_uid) == -1) {
+		xlog(L_ERROR, "Failed to drop privileges: %m");
+		return FALSE;
+	}
+	xlog(D_GENERAL, "Effective UID, GID: %u, %u", st.st_uid, st.st_gid);
+
+	if (keep_bind && !statd_clear_capabilities())
+		return FALSE;
+
+	/*
+	 * ORDER
+	 *
+	 * If the on-disk database doesn't exist yet, it is created
+	 * here, after lowering our privileges.  This ensures that
+	 * statd can continue to access the database after privileges
+	 * are dropped.
+	 */
+	return statd_init_database();
+}
+
+/**
+ * statd_system_rebooted - check if we've run since the last reboot
+ *
+ * Returns TRUE if the system has rebooted since we last ran, or if
+ * we can't tell if we rebooted.  Otherwise FALSE is returned.
+ */
+bool_t
+statd_system_rebooted(void)
+{
+	key_t key;
+
+	key = ftok(statd_db_filename, 'r');	/* 'r' as in aRbitrary */
+
+	if (semget(key, 1, IPC_CREAT | IPC_EXCL | S_IRUSR | S_IWUSR) == -1) {
+		if (errno == EEXIST) {
+			xlog(D_GENERAL, "Already ran since last reboot");
+			return FALSE;
+		} else
+		xlog(D_GENERAL, "Failed to find shared semaphor: %m");
+	}
+
+	xlog(D_GENERAL, "First run since last reboot");
+	return TRUE;
+}
+
+/*
+ * Returns TRUE if successful, or FALSE if an error occurred.
+ */
+static bool_t
+statd_insert_info(sqlite3 *db, const char *name, const int64_t value)
+{
+	sqlite3_stmt *stmt;
+	bool_t result;
+	int rc;
+
+	result = FALSE;
+
+	if (!statd_prepare_stmt(db, &stmt, "INSERT OR REPLACE INTO "
+			STATD_INFO_TABLENAME " (name,value) VALUES(?,?);"))
+		goto out;
+
+	rc = sqlite3_bind_text(stmt, 1, name, -1, SQLITE_STATIC);
+	if (rc != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to bind name %s: %s",
+				name, sqlite3_errmsg(db));
+		goto out_finalize;
+	}
+
+	rc = sqlite3_bind_int64(stmt, 2, (sqlite3_int64)value);
+	if (rc != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to bind value: %s",
+				sqlite3_errmsg(db));
+		goto out_finalize;
+	}
+
+	rc = sqlite3_step(stmt);
+	if (rc != SQLITE_DONE) {
+		xlog(L_ERROR, "Failed to write NSM state number: %s",
+				sqlite3_errmsg(db));
+		goto out_finalize;
+	}
+
+	result = TRUE;
+
+out_finalize:
+	statd_finalize_stmt(stmt);
+
+out:
+	return result;
+}
+
+/*
+ * Returns TRUE if successful, or FALSE if an error occurred.
+ */
+static bool_t
+statd_get_info(sqlite3 *db, const char *name, int64_t *value)
+{
+	sqlite3_stmt *stmt;
+	bool_t result;
+	int rc;
+
+	result = FALSE;
+
+	if (!statd_prepare_stmt(db, &stmt,
+			"SELECT * FROM " STATD_INFO_TABLENAME " WHERE name=?;"))
+		goto out;
+
+	rc = sqlite3_bind_text(stmt, 1, name, -1, SQLITE_STATIC);
+	if (rc != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to bind name %s: %s",
+				name, sqlite3_errmsg(db));
+		goto out_finalize;
+	}
+
+	rc = sqlite3_step(stmt);
+	switch (rc) {
+	case SQLITE_ROW:
+		*value = (int64_t)sqlite3_column_int64(stmt, 1);
+		result = TRUE;
+		break;
+	case SQLITE_DONE:
+		xlog(D_GENERAL, "Table '" STATD_INFO_TABLENAME
+				"' did not contain a row for '%s'", name);
+		break;
+	default:
+		xlog(L_ERROR, "Failed to read value of '%s': %s",
+				name, sqlite3_errmsg(db));
+	}
+
+out_finalize:
+	statd_finalize_stmt(stmt);
+
+out:
+	return result;
+}
+
+/**
+ * statd_get_nsm_state - retrieve on-disk NSM state number
+ *
+ * Returns an odd NSM state number, or a starting NSM state number
+ * of 1 if some error occurred.
+ */
+int
+statd_get_nsm_state(void)
+{
+	int64_t state;
+	sqlite3 *db;
+
+	state = 1;
+	db = statd_open_db(SQLITE_OPEN_READONLY);
+	if (db == NULL)
+		goto out;
+
+	if (!statd_get_info(db, "state", &state))
+		goto out_close;
+
+	if (!(state & 1)) {
+		xlog(L_WARNING, "Database contained an even state number");
+		state++;
+	}
+
+out_close:
+	statd_close_db(db);
+
+out:
+	return (int)state;
+}
+
+/**
+ * statd_update_nsm_state - unconditionally update NSM state number
+ * @db: open database connection
+ *
+ * Returns TRUE if the on-disk NSM state number was updated successfully;
+ * otherwise FALSE.
+ */
+bool_t
+statd_update_nsm_state(sqlite3 *db)
+{
+	int64_t state;
+
+	if (!statd_get_info(db, "state", &state))
+		state = 1;
+	else
+		state += 2;
+
+	if (!statd_insert_info(db, "state", state))
+		return FALSE;
+
+	xlog(D_GENERAL, "Bumped NSM state, new value: %d", state);
+	return TRUE;
+}
+
+/**
+ * statd_dump_monitor_list - Display the monitor list
+ *
+ * For debugging.
+ */
+void
+statd_dump_monitor_list(void)
+{
+	int result, err, i, nrow, ncolumn;
+	char **resultp;
+	char *err_msg;
+	sqlite3 *db;
+
+	result = 0;
+
+	db = statd_open_db(SQLITE_OPEN_READONLY);
+	if (db == NULL)
+		return;
+
+	err_msg = NULL;
+	err = sqlite3_get_table(db, "SELECT"
+		" mon_name,my_name,state  FROM " STATD_MONITOR_TABLENAME ";",
+			&resultp, &nrow, &ncolumn, &err_msg);
+	if (err != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to get table '"
+				STATD_MONITOR_TABLENAME "': %s", err_msg);
+		sqlite3_free(err_msg);
+		goto out_close;
+	}
+	if (ncolumn == 0) {
+		xlog(D_GENERAL, "The monitor list currently is empty");
+		goto out_free_table;
+	}
+	if (ncolumn != 3) {
+		xlog(L_ERROR, "Incorrect column count %d in SELECT result",
+				ncolumn);
+		goto out_free_table;
+	}
+
+	xlog(D_GENERAL, "Monitor list (mon_name, my_name, state)");
+	for (i = ncolumn; i < (nrow + 1) * ncolumn ; i += ncolumn)
+		xlog(D_GENERAL, "%s, %s, %d",
+			resultp[i], resultp[i + 1], resultp[i + 2]);
+
+out_free_table:
+	sqlite3_free_table(resultp);
+
+out_close:
+	statd_close_db(db);
+}
+
+/**
+ * statd_print_capabilities - dump process capability set to log
+ *
+ * For debugging.
+ */
+void
+statd_print_capabilities(void)
+{
+	cap_t caps;
+
+	caps = cap_get_proc();
+	if (caps == NULL) {
+		xlog(L_ERROR, "Failed to get process capaabilities: %m");
+		return;
+	}
+
+	xlog(L_NOTICE, "Capabilities: %s", cap_to_text(caps, NULL));
+
+	(void)cap_free(caps);
+}
utils/new-statd/hostname.c new file +520 -0

520 lines not on this page · raw message

utils/new-statd/nlmcall.c new file +525 -0

525 lines not on this page · raw message

A message that is not in the archive (sent off-list or not yet received): 20090805144540.12866.22084.stgit-RytpoXr2tKZ9HhUboXbp9zCvJB+x5qRC@public.gmane.org

#4 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 32 lines · Chuck Lever

Provide a new implementation of statd that supports IPv6. The new statd implementation resides under

utils/new-statd/

The contents of this directory are built if --enable-tirpc is set on the ./configure command line, and sqlite3 is available on the build system. Otherwise, the legacy version of statd, which still resides under utils/statd/, is built.

The goals of this re-write are:

o Support IPv6 networking

Support interoperation with TI-RPC-based NSM implementations. Transport Independent RPC, or TI-RPC, provides IPv6 network support for Linux's NSM implementation.

To support TI-RPC, open code to construct RPC requests in socket buffers and then schedule them has been replaced with standard library calls.

o Support notification via TCP

As a secondary benefit of using TI-RPC library calls, reboot notifications and NLM callbacks can now be sent via connection- oriented transport protocols.

Note that lockd does not (yet) tell statd what transport protocol to use when sending reboot notifications. statd/sm-notify will continue to use UDP for the time being.

Quoted 7 lines

o Use an embedded database for storing on-disk callback data

This whole exercise is for the purpose of crash robustness. There are well-known deficiencies with simple create/rename/unlink disk storage schemes during system crashes. Replace the current flat-file monitor list mechanism which uses sync(2) with sqlite3, which uses fsync(3).

If someone wants to move around that data, is it still simple to do that? (Where is it kept on the filesystem?)

(I'm thinking of someone that shares it for high-availabity, as in:

http://www.howtoforge.com/high_availability_nfs_drbd_heartbeat_p3

Or maybe somebody that just needs to move their /var partition to a different disk one day.)

Quoted 52 lines

o Share code between sm-notify and statd

Statd and sm-notify access the same set of on-disk data. These separate programs now share the same code and implementation, with access to on-disk data serialized by sqlite3. The two remain separate executables to allow other system facilities to send reboot notifications without poking statd.

o Reduce impact of DNS outages

The heuristics used by SM_NOTIFY to figure out which remote peer has rebooted are heavily dependent on DNS. If the DNS service is slow or hangs, that will make the NSM listener unresponsive. Incoming SM_NOTIFY requests are now handled in a sidecar process to reduce the impact of DNS outages on the NSM service listener.

o Proper my_name support

The current version of statd uses gethostname(3) to generate the mon_name argument of SM_NOTIFY. This value can change across a reboot. The new version of statd records lockd's my_name, passed by every SM_MON request, and uses that when sending SM_NOTIFY.

This can be useful for multi-homed and DHCP configured hosts.

o Send SM_NOTIFY more aggressively

It has been recommended that statd/sm-notify send SM_NOTIFY more aggressively (for example, to the entire list returned by getaddrinfo(3)). Since SM_NOTIFY's reply is NULL, there's no way to tell whether the remote peer recognized the mon_name we sent. More study is required, but this implementation attempts to send an SM_NOTIFY request to each address returned by getaddrinfo(3).

This re-implementation paves the way for a number of future improvements. However, it does not immediately address:

o lockd/statd start-up serialization issues

Sending reboot notifications, starting statd and lockd, and opening the lockd grace period are still determined independently in user space and the kernel.

o Binding mon_names to caller IP addresses

By default, lockd continues to send IP addresses as the mon_name argument of the SM_MON procedure. This provides a better guarantee of being able to contact remote peers during a reboot, but means statd must continue to use heuristics to match incoming SM_NOTIFY requests with peers on the monitor list.

Quoted 6 lines

o Distinct logic for NFS client- and server-side

Client-side and server-side monitoring requirements are different. Statd continues to use the same logic for both NFS client and server, as the NSMv1 protocol does not provide any indication that a mon_name is for a client or server peer.

Note we probably don't need to be limited by the protocol here, only by kernel backwards-compatibility requirements, as long as this is just kernel<->statd communication and not something that goes across the wire to other statd implementations.

--b.

#5 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 44 lines · J. Bruce Fields
Quoted 32 lines · Chuck Lever

Provide a new implementation of statd that supports IPv6. The new statd implementation resides under

utils/new-statd/

The contents of this directory are built if --enable-tirpc is set on the ./configure command line, and sqlite3 is available on the build system. Otherwise, the legacy version of statd, which still resides under utils/statd/, is built.

The goals of this re-write are:

o Support IPv6 networking

Support interoperation with TI-RPC-based NSM implementations. Transport Independent RPC, or TI-RPC, provides IPv6 network support for Linux's NSM implementation.

To support TI-RPC, open code to construct RPC requests in socket buffers and then schedule them has been replaced with standard library calls.

o Support notification via TCP

As a secondary benefit of using TI-RPC library calls, reboot notifications and NLM callbacks can now be sent via connection- oriented transport protocols.

Note that lockd does not (yet) tell statd what transport protocol to use when sending reboot notifications. statd/sm-notify will continue to use UDP for the time being.

Quoted 7 lines

o Use an embedded database for storing on-disk callback data

This whole exercise is for the purpose of crash robustness. There are well-known deficiencies with simple create/rename/unlink disk storage schemes during system crashes. Replace the current flat-file monitor list mechanism which uses sync(2) with sqlite3, which uses fsync(3).

If someone wants to move around that data, is it still simple to do that? (Where is it kept on the filesystem?)

Quoted 6 lines

(I'm thinking of someone that shares it for high-availabity, as in:

http://www.howtoforge.com/high_availability_nfs_drbd_heartbeat_p3

Or maybe somebody that just needs to move their /var partition to a different disk one day.)

Statd's monitor lists and state number are stored in a single regular file, /var/lib/nfs/statd/statdb by default. This file can be easily backed up, or used on other systems, if desired. I would recommend ensuring the NSM state number is reset in the latter case, which can be done with the sqlite3 command.

I've had some dialog with Lon Hohberger about clustering requirements. I think we are looking at crafting a separate utility that uses sqlite3 C function calls to extract data that's interesting to the clustering implementation. Again, this could even be scripted with bash and the sqlite3 command, but perhaps a C program is more maintainable.

Quoted 59 lines
Quoted 52 lines

o Share code between sm-notify and statd

Statd and sm-notify access the same set of on-disk data. These separate programs now share the same code and implementation, with access to on-disk data serialized by sqlite3. The two remain separate executables to allow other system facilities to send reboot notifications without poking statd.

o Reduce impact of DNS outages

The heuristics used by SM_NOTIFY to figure out which remote peer has rebooted are heavily dependent on DNS. If the DNS service is slow or hangs, that will make the NSM listener unresponsive. Incoming SM_NOTIFY requests are now handled in a sidecar process to reduce the impact of DNS outages on the NSM service listener.

o Proper my_name support

The current version of statd uses gethostname(3) to generate the mon_name argument of SM_NOTIFY. This value can change across a reboot. The new version of statd records lockd's my_name, passed by every SM_MON request, and uses that when sending SM_NOTIFY.

This can be useful for multi-homed and DHCP configured hosts.

o Send SM_NOTIFY more aggressively

It has been recommended that statd/sm-notify send SM_NOTIFY more aggressively (for example, to the entire list returned by getaddrinfo(3)). Since SM_NOTIFY's reply is NULL, there's no way to tell whether the remote peer recognized the mon_name we sent. More study is required, but this implementation attempts to send an SM_NOTIFY request to each address returned by getaddrinfo(3).

This re-implementation paves the way for a number of future improvements. However, it does not immediately address:

o lockd/statd start-up serialization issues

Sending reboot notifications, starting statd and lockd, and opening the lockd grace period are still determined independently in user space and the kernel.

o Binding mon_names to caller IP addresses

By default, lockd continues to send IP addresses as the mon_name argument of the SM_MON procedure. This provides a better guarantee of being able to contact remote peers during a reboot, but means statd must continue to use heuristics to match incoming SM_NOTIFY requests with peers on the monitor list.

Quoted 6 lines

o Distinct logic for NFS client- and server-side

Client-side and server-side monitoring requirements are different. Statd continues to use the same logic for both NFS client and server, as the NSMv1 protocol does not provide any indication that a mon_name is for a client or server peer.

Quoted 4 lines

Note we probably don't need to be limited by the protocol here, only by kernel backwards-compatibility requirements, as long as this is just kernel<->statd communication and not something that goes across the wire to other statd implementations.

Agreed.

It would be possible to export the kernel's NSM host cache via sysfs, for instance. An SM_MON upcall could cause statd to look in /sys for more information like whether the remote peer is a client or server, and what transport protocol and what network address the caller used to contact the local host. This kind of scheme would work well for both old kernels running the new statd, and new kernels running the old statd.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#6 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 58 lines · Chuck Lever
Quoted 44 lines · J. Bruce Fields
Quoted 32 lines · Chuck Lever

Provide a new implementation of statd that supports IPv6. The new statd implementation resides under

utils/new-statd/

The contents of this directory are built if --enable-tirpc is set on the ./configure command line, and sqlite3 is available on the build system. Otherwise, the legacy version of statd, which still resides under utils/statd/, is built.

The goals of this re-write are:

o Support IPv6 networking

Support interoperation with TI-RPC-based NSM implementations. Transport Independent RPC, or TI-RPC, provides IPv6 network support for Linux's NSM implementation.

To support TI-RPC, open code to construct RPC requests in socket buffers and then schedule them has been replaced with standard library calls.

o Support notification via TCP

As a secondary benefit of using TI-RPC library calls, reboot notifications and NLM callbacks can now be sent via connection- oriented transport protocols.

Note that lockd does not (yet) tell statd what transport protocol to use when sending reboot notifications. statd/sm-notify will continue to use UDP for the time being.

Quoted 7 lines

o Use an embedded database for storing on-disk callback data

This whole exercise is for the purpose of crash robustness. There are well-known deficiencies with simple create/rename/unlink disk storage schemes during system crashes. Replace the current flat-file monitor list mechanism which uses sync(2) with sqlite3, which uses fsync(3).

If someone wants to move around that data, is it still simple to do that? (Where is it kept on the filesystem?)

Quoted 6 lines

(I'm thinking of someone that shares it for high-availabity, as in:

http://www.howtoforge.com/high_availability_nfs_drbd_heartbeat_p3

Or maybe somebody that just needs to move their /var partition to a different disk one day.)

Statd's monitor lists and state number are stored in a single regular file, /var/lib/nfs/statd/statdb by default. This file can be easily backed up, or used on other systems, if desired. I would recommend ensuring the NSM state number is reset in the latter case, which can be done with the sqlite3 command.

Quoted 5 lines

I've had some dialog with Lon Hohberger about clustering requirements. I think we are looking at crafting a separate utility that uses sqlite3 C function calls to extract data that's interesting to the clustering implementation. Again, this could even be scripted with bash and the sqlite3 command, but perhaps a C program is more maintainable.

OK, good.

And for the simplest cases, it should still be enough to just copy /var/lib/nfs/, right?

--b.

#7 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 66 lines · J. Bruce Fields
Quoted 59 lines · Chuck Lever
Quoted 45 lines · J. Bruce Fields
Quoted 33 lines · Chuck Lever

Provide a new implementation of statd that supports IPv6. The new statd implementation resides under

utils/new-statd/

The contents of this directory are built if --enable-tirpc is set on the ./configure command line, and sqlite3 is available on the build system. Otherwise, the legacy version of statd, which still resides under utils/statd/, is built.

The goals of this re-write are:

o Support IPv6 networking

Support interoperation with TI-RPC-based NSM implementations. Transport Independent RPC, or TI-RPC, provides IPv6 network
support
for Linux's NSM implementation.

To support TI-RPC, open code to construct RPC requests in socket buffers and then schedule them has been replaced with standard library calls.

o Support notification via TCP

As a secondary benefit of using TI-RPC library calls, reboot notifications and NLM callbacks can now be sent via connection- oriented transport protocols.

Note that lockd does not (yet) tell statd what transport protocol to use when sending reboot notifications. statd/sm-notify will continue to use UDP for the time being.

Quoted 7 lines

o Use an embedded database for storing on-disk callback data

This whole exercise is for the purpose of crash robustness. There are well-known deficiencies with simple create/rename/unlink disk storage schemes during system crashes. Replace the current flat-file monitor list mechanism which uses sync(2) with sqlite3, which uses fsync(3).

If someone wants to move around that data, is it still simple to do that? (Where is it kept on the filesystem?)

Quoted 6 lines

(I'm thinking of someone that shares it for high-availabity, as in:

http://www.howtoforge.com/high_availability_nfs_drbd_heartbeat_p3

Or maybe somebody that just needs to move their /var partition to a different disk one day.)

Statd's monitor lists and state number are stored in a single regular file, /var/lib/nfs/statd/statdb by default. This file can be easily backed up, or used on other systems, if desired. I would recommend ensuring the NSM state number is reset in the latter case, which can be done with the sqlite3 command.

Quoted 5 lines

I've had some dialog with Lon Hohberger about clustering requirements. I think we are looking at crafting a separate utility that uses sqlite3 C function calls to extract data that's interesting to the clustering implementation. Again, this could even be scripted with bash and the sqlite3 command, but perhaps a C program is more maintainable.

Quoted 4 lines

OK, good.

And for the simplest cases, it should still be enough to just copy /var/lib/nfs/, right?

I don't see why that wouldn't work, as long statd/sm-notify aren't updating the database at that moment. For safety I think there is an sqlite3 backup mechanism for database files that respects the library's locking semantics.

sqlite3 doesn't do anything special under the covers. It uses only POSIX file access and locking calls, as far as I know. So I think hosting /var on most well-behaved clustering file systems won't have any problem with this arrangement.

One (admittedly minor) reason I did this is so we have some sample code to try for other NFS-related daemons that need to store information in /var robustly, potentially in clustered environments.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#8 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 4 lines · Chuck Lever

sqlite3 doesn't do anything special under the covers. It uses only POSIX file access and locking calls, as far as I know. So I think hosting /var on most well-behaved clustering file systems won't have any problem with this arrangement.

So we're basically introducing a dependency on a completely new library that will have to be added to boot partitions/nfsroot/etc, and we have no real reason for doing it other than because we want to move from using sync() to fsync()?

Sounds like a NACK to me...

Trond

A message that is not in the archive (sent off-list or not yet received): 1249507356.5428.11.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#10 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 6 lines · Trond Myklebust
Quoted 4 lines · Chuck Lever

sqlite3 doesn't do anything special under the covers. It uses only POSIX file access and locking calls, as far as I know. So I think hosting /var on most well-behaved clustering file systems won't have any problem with this arrangement.

Quoted 6 lines

So we're basically introducing a dependency on a completely new library that will have to be added to boot partitions/nfsroot/etc, and we have no real reason for doing it other than because we want to move from using sync() to fsync()?

Sounds like a NACK to me...

Which library are you talking about, libsqlite3 or libtirpc? Because NEITHER of those is in /lib.

In any event, it's not just sync(2) that is a problem. sync(2) by itself is a boot performance problem, but it's the combination of rename and sync that is known to be especially unreliable during system crashes. Statd, being a crash monitor, shouldn't depend on rename/sync to maintain persistent data in the face of system instability. I'd call that a real reason to use something more robust.

Can we try to be a little more constructive, please? Asking the list (which includes distributors, who actually have to worry about such things) whether this would be a problem is significantly less abrasive then just saying "NACK" outright.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#11 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 13 lines · Chuck Lever
Quoted 12 lines · Trond Myklebust
Quoted 4 lines · Chuck Lever

sqlite3 doesn't do anything special under the covers. It uses only POSIX file access and locking calls, as far as I know. So I think hosting /var on most well-behaved clustering file systems won't have any problem with this arrangement.

So we're basically introducing a dependency on a completely new library
that will have to be added to boot partitions/nfsroot/etc, and we have no real reason for doing it other than because we want to move from using sync() to fsync()?

Quoted 4 lines
Quoted 1 line

Sounds like a NACK to me...

Which library are you talking about, libsqlite3 or libtirpc? Because NEITHER of those is in /lib.

libsqlite is the problem. Unlike libtirpc, it's utility has yet to be established.

Quoted 6 lines

In any event, it's not just sync(2) that is a problem. sync(2) by itself is a boot performance problem, but it's the combination of rename and sync that is known to be especially unreliable during system crashes. Statd, being a crash monitor, shouldn't depend on rename/sync to maintain persistent data in the face of system instability. I'd call that a real reason to use something more robust.

What are you talking about? Is this about the truncate + rename issue leaving empty files upon a crash?
That issue is solved trivially by doing an fsync() before you rename the file. That entire discussion was about whether or not existing applications should be _required_ to do this kind of POSIX pedantry, when previously they could get away without it.

IOW: that issue alone does not justify replacing the current simple file based scheme.

Quoted 4 lines

Can we try to be a little more constructive, please? Asking the list (which includes distributors, who actually have to worry about such things) whether this would be a problem is significantly less abrasive then just saying "NACK" outright.

It would be constructive if you could actually _justify_ these backward-incompatible changes instead of hand waving, and accusing others of being obstructionist.

Trond

#12 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 14 lines · On Wed, 05 Aug 2009 19:30:04 -0400 Trond Myklebust…
Quoted 13 lines · Chuck Lever
Quoted 12 lines · Trond Myklebust
Quoted 4 lines · Chuck Lever

sqlite3 doesn't do anything special under the covers. It uses only POSIX file access and locking calls, as far as I know. So I think hosting /var on most well-behaved clustering file systems won't have any problem with this arrangement.

So we're basically introducing a dependency on a completely new library
that will have to be added to boot partitions/nfsroot/etc, and we have no real reason for doing it other than because we want to move from using sync() to fsync()?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Sounds like a NACK to me...

Which library are you talking about, libsqlite3 or libtirpc? Because NEITHER of those is in /lib.

libsqlite is the problem. Unlike libtirpc, it's utility has yet to be established.

Sorry to revive this so late, but I think we need to come to some sort of resolution here. The only missing piece for client side IPv6 support is statd...

I'm not sure I understand the objection to using libsqlite3 here. We certainly could roll our own routines to handle data storage, but why would we want to do so? sqlite3 is quite good at what it does. Why wouldn't we want to use it?

Quoted 14 lines
Quoted 6 lines

In any event, it's not just sync(2) that is a problem. sync(2) by itself is a boot performance problem, but it's the combination of rename and sync that is known to be especially unreliable during system crashes. Statd, being a crash monitor, shouldn't depend on rename/sync to maintain persistent data in the face of system instability. I'd call that a real reason to use something more robust.

What are you talking about? Is this about the truncate + rename issue leaving empty files upon a crash?
That issue is solved trivially by doing an fsync() before you rename the file. That entire discussion was about whether or not existing applications should be _required_ to do this kind of POSIX pedantry, when previously they could get away without it.

Quoted 3 lines

IOW: that issue alone does not justify replacing the current simple file based scheme.

There are other reasons, not to use the simple file-based scheme too...

Internationalized domain names will be easier to deal with via sqlite3, for instance.

Certainly we could code this up ourselves, but what's the benefit to doing that when we have a perfectly good data storage engine available?

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20090909142945.755da393-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#14 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 26 lines · Jeff Layton
Quoted 14 lines · On Wed, 05 Aug 2009 19:30:04 -0400 Trond Myklebust…
Quoted 13 lines · Chuck Lever
Quoted 12 lines · Trond Myklebust
Quoted 4 lines · Chuck Lever

sqlite3 doesn't do anything special under the covers. It uses only POSIX file access and locking calls, as far as I know. So I think hosting /var on most well-behaved clustering file systems won't have any problem with this arrangement.

So we're basically introducing a dependency on a completely new library
that will have to be added to boot partitions/nfsroot/etc, and we have no real reason for doing it other than because we want to move from using sync() to fsync()?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Sounds like a NACK to me...

Which library are you talking about, libsqlite3 or libtirpc? Because NEITHER of those is in /lib.

libsqlite is the problem. Unlike libtirpc, it's utility has yet to be established.

Quoted 8 lines

Sorry to revive this so late, but I think we need to come to some sort of resolution here. The only missing piece for client side IPv6 support is statd...

I'm not sure I understand the objection to using libsqlite3 here. We certainly could roll our own routines to handle data storage, but why would we want to do so? sqlite3 is quite good at what it does. Why wouldn't we want to use it?

Backwards compatibility is one major reason. statd already exists, and is in use out there. I shouldn't be forced to reboot all my clients when I upgrade the nfs-utils package on my server.

Simplicity is another reason. WTF do we need a full SQL database, when all we want to do is store 2 pieces of data (a hostname and a cookie)? It isn't as if this has been a major problem for us previously.

Quoted 14 lines
Quoted 14 lines
Quoted 6 lines

In any event, it's not just sync(2) that is a problem. sync(2) by itself is a boot performance problem, but it's the combination of rename and sync that is known to be especially unreliable during system crashes. Statd, being a crash monitor, shouldn't depend on rename/sync to maintain persistent data in the face of system instability. I'd call that a real reason to use something more robust.

What are you talking about? Is this about the truncate + rename issue leaving empty files upon a crash?
That issue is solved trivially by doing an fsync() before you rename the file. That entire discussion was about whether or not existing applications should be _required_ to do this kind of POSIX pedantry, when previously they could get away without it.

Quoted 8 lines
Quoted 3 lines

IOW: that issue alone does not justify replacing the current simple file based scheme.

There are other reasons, not to use the simple file-based scheme too...

Internationalized domain names will be easier to deal with via sqlite3, for instance.

Please explain...

Quoted 2 lines

Certainly we could code this up ourselves, but what's the benefit to doing that when we have a perfectly good data storage engine available?

Why change something that works???? Rewriting from scratch is _NOT_ the Linux way, and has usually bitten us hard when we've done it.

The 2.6.19 rewrite of the kernel mount code springs to mind...

Trond

#15 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 31 lines · On Wed, 09 Sep 2009 14:39:59 -0400 Trond Myklebust…
Quoted 30 lines · Jeff Layton
Quoted 14 lines · On Wed, 05 Aug 2009 19:30:04 -0400 Trond Myklebust…
Quoted 13 lines · Chuck Lever
Quoted 12 lines · Trond Myklebust
Quoted 4 lines · Chuck Lever

sqlite3 doesn't do anything special under the covers. It uses only POSIX file access and locking calls, as far as I know. So I think hosting /var on most well-behaved clustering file systems won't have any problem with this arrangement.

So we're basically introducing a dependency on a completely new library
that will have to be added to boot partitions/nfsroot/etc, and we have no real reason for doing it other than because we want to move from using sync() to fsync()?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Sounds like a NACK to me...

Which library are you talking about, libsqlite3 or libtirpc? Because NEITHER of those is in /lib.

libsqlite is the problem. Unlike libtirpc, it's utility has yet to be established.

Sorry to revive this so late, but I think we need to come to some sort of resolution here. The only missing piece for client side IPv6 support is statd...

Quoted 9 lines
Quoted 4 lines

I'm not sure I understand the objection to using libsqlite3 here. We certainly could roll our own routines to handle data storage, but why would we want to do so? sqlite3 is quite good at what it does. Why wouldn't we want to use it?

Backwards compatibility is one major reason. statd already exists, and is in use out there. I shouldn't be forced to reboot all my clients when I upgrade the nfs-utils package on my server.

We could roll a conversion utility for this if it would help. nfs-utils upgrades usually mean restarting statd anyway:

shut down old-statd
convert flat file db to sqlite
start up new-statd

...so I don't think we necessarily need to reboot the clients for this. It should (in theory) be possible to do the reverse even.

Quoted 4 lines

Simplicity is another reason. WTF do we need a full SQL database, when all we want to do is store 2 pieces of data (a hostname and a cookie)? It isn't as if this has been a major problem for us previously.

Been a little while since I took my initial look at new-statd, but I see that Chuck is has this (just a for-instance):

rc = sqlite3_prepare_v2(db, "CREATE TABLE " STATD_MONITOR_TABLENAME
" (priv BLOB,"
" mon_name TEXT NOT NULL," " my_name TEXT NOT NULL," " program INTEGER,"
" version INTEGER,"
" procedure INTEGER,"
" protocol TEXT NOT NULL," " state INTEGER,"
" UNIQUE(mon_name,my_name));",
-1, &stmt, NULL);

He's tracking some other info there too. Is this all necessary? Maybe not now, but having a storage engine that can cope with tracking extra info will make it easier to handle things like multihomed clients and servers correctly (something that the existing statd is not very good at at the moment).

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines

In any event, it's not just sync(2) that is a problem. sync(2) by itself is a boot performance problem, but it's the combination of rename and sync that is known to be especially unreliable during system crashes. Statd, being a crash monitor, shouldn't depend on rename/sync to maintain persistent data in the face of system instability. I'd call that a real reason to use something more robust.

What are you talking about? Is this about the truncate + rename issue leaving empty files upon a crash?
That issue is solved trivially by doing an fsync() before you rename the file. That entire discussion was about whether or not existing applications should be _required_ to do this kind of POSIX pedantry, when previously they could get away without it.

IOW: that issue alone does not justify replacing the current simple file based scheme.

Quoted 7 lines
Quoted 4 lines

There are other reasons, not to use the simple file-based scheme too...

Internationalized domain names will be easier to deal with via sqlite3, for instance.

Please explain...

Well, we currently store statd info in flat files named with the hostname. With an internationalized domain name, we may have a multibyte character in that name. We could try to store that as an ASCII or UTF8 name, but we'd have to roll conversion routines for it. Why bother when we have a storage engine that does that work for us?

Quoted 8 lines
Quoted 2 lines

Certainly we could code this up ourselves, but what's the benefit to doing that when we have a perfectly good data storage engine available?

Why change something that works???? Rewriting from scratch is _NOT_ the Linux way, and has usually bitten us hard when we've done it.

The 2.6.19 rewrite of the kernel mount code springs to mind...

A good point. Given who I work for, I *really* hate regressions since they tend to mean a lot of my time tends to get eaten up.

At some point however it becomes very difficult to patch up old code. old-statd in particular hasn't seen the attention that it probably should have.

I trust that Chuck will be willing to fix problems that come along. The new code seems to be on par with the old in complexity so I don't think we'd be taking on a great maintenance burden with it even if Chuck isn't available.

Could we add IPv6 support as a patchset to the existing statd instead? Sure. That patch would be smaller than Chuck's rewrite, but I still think there are advantages to considering a major overhaul here.

Signature · 1 line
Jeff Layton <[email protected]>

A message that is not in the archive (sent off-list or not yet received): 20090909151354.504ccdf6-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#17 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 130 lines · Jeff Layton
Quoted 31 lines · On Wed, 09 Sep 2009 14:39:59 -0400 Trond Myklebust…
Quoted 30 lines · Jeff Layton
Quoted 14 lines · On Wed, 05 Aug 2009 19:30:04 -0400 Trond Myklebust…
Quoted 13 lines · Chuck Lever
Quoted 12 lines · Trond Myklebust
Quoted 4 lines · Chuck Lever

sqlite3 doesn't do anything special under the covers. It uses only POSIX file access and locking calls, as far as I know. So I think hosting /var on most well-behaved clustering file systems won't have any problem with this arrangement.

So we're basically introducing a dependency on a completely new library
that will have to be added to boot partitions/nfsroot/etc, and we have no real reason for doing it other than because we want to move from using sync() to fsync()?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Sounds like a NACK to me...

Which library are you talking about, libsqlite3 or libtirpc? Because NEITHER of those is in /lib.

libsqlite is the problem. Unlike libtirpc, it's utility has yet to be established.

Sorry to revive this so late, but I think we need to come to some sort of resolution here. The only missing piece for client side IPv6 support is statd...

Quoted 9 lines
Quoted 4 lines

I'm not sure I understand the objection to using libsqlite3 here. We certainly could roll our own routines to handle data storage, but why would we want to do so? sqlite3 is quite good at what it does. Why wouldn't we want to use it?

Backwards compatibility is one major reason. statd already exists, and is in use out there. I shouldn't be forced to reboot all my clients when I upgrade the nfs-utils package on my server.

We could roll a conversion utility for this if it would help. nfs-utils upgrades usually mean restarting statd anyway:

shut down old-statd
convert flat file db to sqlite
start up new-statd

...so I don't think we necessarily need to reboot the clients for this. It should (in theory) be possible to do the reverse even.

Quoted 4 lines

Simplicity is another reason. WTF do we need a full SQL database, when all we want to do is store 2 pieces of data (a hostname and a cookie)? It isn't as if this has been a major problem for us previously.

Been a little while since I took my initial look at new-statd, but I see that Chuck is has this (just a for-instance):

rc = sqlite3_prepare_v2(db, "CREATE TABLE "
STATD_MONITOR_TABLENAME
" (priv BLOB,"
" mon_name TEXT NOT NULL," " my_name TEXT NOT NULL," " program INTEGER," " version INTEGER," " procedure INTEGER," " protocol TEXT NOT NULL," " state INTEGER,"
" UNIQUE(mon_name,my_name));",
-1, &stmt, NULL);

He's tracking some other info there too. Is this all necessary? Maybe not now, but having a storage engine that can cope with tracking extra info will make it easier to handle things like multihomed clients and servers correctly (something that the existing statd is not very good at at the moment).

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines

In any event, it's not just sync(2) that is a problem. sync(2) by itself is a boot performance problem, but it's the combination of rename and sync that is known to be especially unreliable during system crashes. Statd, being a crash monitor, shouldn't depend on rename/sync to maintain persistent data in the face of system instability. I'd call that a real reason to use something more robust.

What are you talking about? Is this about the truncate + rename issue leaving empty files upon a crash?
That issue is solved trivially by doing an fsync() before you rename the file. That entire discussion was about whether or not existing applications should be _required_ to do this kind of POSIX pedantry, when previously they could get away without it.

IOW: that issue alone does not justify replacing the current simple file based scheme.

Quoted 7 lines
Quoted 4 lines

There are other reasons, not to use the simple file-based scheme too...

Internationalized domain names will be easier to deal with via sqlite3, for instance.

Please explain...

Well, we currently store statd info in flat files named with the hostname. With an internationalized domain name, we may have a multibyte character in that name. We could try to store that as an ASCII or UTF8 name, but we'd have to roll conversion routines for it. Why bother when we have a storage engine that does that work for us?

Quoted 8 lines
Quoted 2 lines

Certainly we could code this up ourselves, but what's the benefit to doing that when we have a perfectly good data storage engine available?

Why change something that works???? Rewriting from scratch is _NOT_ the Linux way, and has usually bitten us hard when we've done it.

The 2.6.19 rewrite of the kernel mount code springs to mind...

A good point. Given who I work for, I *really* hate regressions since they tend to mean a lot of my time tends to get eaten up.

At some point however it becomes very difficult to patch up old code. old-statd in particular hasn't seen the attention that it probably should have.

Quoted 7 lines

I trust that Chuck will be willing to fix problems that come along. The new code seems to be on par with the old in complexity so I don't think we'd be taking on a great maintenance burden with it even if Chuck isn't available.

Could we add IPv6 support as a patchset to the existing statd instead? Sure.

Not as easy as you might think.

The old statd open codes a bunch of the RPC client side, which means all of that would have to be rewritten anyway.

Quoted 2 lines

That patch would be smaller than Chuck's rewrite, but I still think there are advantages to considering a major overhaul here.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#18 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

In reply to Trond Myklebust
Quoted 31 lines · Trond Myklebust
Quoted 30 lines · Jeff Layton
Quoted 14 lines · On Wed, 05 Aug 2009 19:30:04 -0400 Trond Myklebust…
Quoted 13 lines · Chuck Lever
Quoted 12 lines · Trond Myklebust
Quoted 4 lines · Chuck Lever

sqlite3 doesn't do anything special under the covers. It uses only POSIX file access and locking calls, as far as I know. So I think hosting /var on most well-behaved clustering file systems won't have any problem with this arrangement.

So we're basically introducing a dependency on a completely new library
that will have to be added to boot partitions/nfsroot/etc, and we have no real reason for doing it other than because we want to move from using sync() to fsync()?

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Sounds like a NACK to me...

Which library are you talking about, libsqlite3 or libtirpc? Because NEITHER of those is in /lib.

libsqlite is the problem. Unlike libtirpc, it's utility has yet to be established.

Sorry to revive this so late, but I think we need to come to some sort of resolution here. The only missing piece for client side IPv6 support is statd...

Quoted 8 lines
Quoted 4 lines

I'm not sure I understand the objection to using libsqlite3 here. We certainly could roll our own routines to handle data storage, but why would we want to do so? sqlite3 is quite good at what it does. Why wouldn't we want to use it?

Backwards compatibility is one major reason. statd already exists, and is in use out there. I shouldn't be forced to reboot all my clients when I upgrade the nfs-utils package on my server.

The old statd still exists in nfs-utils. The new statd is an entirely separate component. Distributions can continue to use the old statd as long as they want. This is a red herring.

Quoted 3 lines

Simplicity is another reason. WTF do we need a full SQL database, when all we want to do is store 2 pieces of data (a hostname and a cookie)? It isn't as if this has been a major problem for us previously.

Because we are not storing just a hostname and a cookie. We are storing several different data items for each host, and we need to search over the records, and provide uniqueness constraints, and handle data conversion (for binary data like the cookie, for string data like the hostname, and for integers, like the prog/vers/proc tuple). We need to store them durably on persistent storage to have some protection against crashes. These are all things that an embedded database can do well, and that we therefore don't have to code ourselves.

Simplicity is in the eye of the beholder. I can easily counterargue that it's simpler to rely on a library than to duplicate the functionality in statd. If we think it is appropriate to use glibc for managing a memory heap rather than open coding it, and libtirpc for handling RPC calls rather than open coding it, then why shouldn't we use another library for managing our host records on disk, rather than open coding our data conversion and record searching logic?

Again, this is a red herring.

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 6 lines

In any event, it's not just sync(2) that is a problem. sync(2) by itself is a boot performance problem, but it's the combination of rename and sync that is known to be especially unreliable during system crashes. Statd, being a crash monitor, shouldn't depend on rename/sync to maintain persistent data in the face of system instability. I'd call that a real reason to use something more robust.

What are you talking about? Is this about the truncate + rename issue leaving empty files upon a crash?
That issue is solved trivially by doing an fsync() before you rename the file. That entire discussion was about whether or not existing applications should be _required_ to do this kind of POSIX pedantry, when previously they could get away without it.

IOW: that issue alone does not justify replacing the current simple file based scheme.

Quoted 6 lines
Quoted 4 lines

There are other reasons, not to use the simple file-based scheme too...

Internationalized domain names will be easier to deal with via sqlite3, for instance.

Please explain...

IPv6 is used in Asia, where they almost certainly need to use non-ASCII characters in their hostnames. Internationalized domain names are stored in double-wide character sets. To provide reliable support for IDNs in statd, we will have to guarantee somehow that we can store an IDN as a file name (if we want to stay with the current scheme), no matter what file system is used for /var.

What's more, multi-homed host support will need to store multiple records for the same hostname. The mon_name is the same, but my_name is different, for each of these records. So we could do that by adding more than one line in each hostname file, but it's also a simple matter to set this up in SQL.

When we want to have statd remember things like multiple addresses for the same hostname, or whether the remote is a client or server, we will need to make more adjustments to the files.

As we get more and more new requirements, why lock ourselves into the current on-disk format? Using statd means we can store new fields and new records without any backwards-compatibility issues. It's all handled by the database code. So, we can think about the high level problem of getting statd to behave correctly rather than worry about the details of exactly how we are going to get the next data item stored in our current files in a backward compatible way.

Quoted 5 lines
Quoted 2 lines

Certainly we could code this up ourselves, but what's the benefit to doing that when we have a perfectly good data storage engine available?

Why change something that works???? Rewriting from scratch is _NOT_ the Linux way, and has usually bitten us hard when we've done it.

Because we are adding a bunch of new feature requirements. Internationalized domain names, multi-homed host support, IPv6 and TI-RPC, fast boot times, keeping better track of remote host addresses, keeping track of which remotes are clients and which are servers, and support for sending notifications via TCP all require significant modifications to this code base.

At some point you have to look at the code you have, and decide it's simply not going to be adequate, going forward.

Quoted 1 line

The 2.6.19 rewrite of the kernel mount code springs to mind...

One can just as easily argue that we've been bitten hard precisely because we've let things rot, or because we have inadequate testing for these components.

Another red herring, and especially annoying because you've known I was rewriting statd for months. Only now, when I'm done, do you say "rewriting is not the Linux way."

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#19 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 4 lines · Chuck Lever

On Sep 9, 2009, at 2:39 PM, Trond Myklebust wrote:
The old statd still exists in nfs-utils. The new statd is an entirely separate component. Distributions can continue to use the old statd as long as they want. This is a red herring.

Bullshit. If they are adding IPv6 support, then they will have to upgrade at some point.

Quoted 4 lines
Quoted 3 lines

Simplicity is another reason. WTF do we need a full SQL database, when all we want to do is store 2 pieces of data (a hostname and a cookie)? It isn't as if this has been a major problem for us previously.

Quoted 9 lines

Because we are not storing just a hostname and a cookie. We are storing several different data items for each host, and we need to search over the records, and provide uniqueness constraints, and handle data conversion (for binary data like the cookie, for string data like the hostname, and for integers, like the prog/vers/proc tuple). We need to store them durably on persistent storage to have some protection against crashes. These are all things that an embedded database can do well, and that we therefore don't have to code ourselves.

Speaking of red herrings. Why are we adding all this crap?

This is a legacy filesystem! We shouldn't not be rewriting NLM/NSM from scratch, just add minimal support for IPv6.

Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
Quoted 17 lines
Quoted 7 lines

In any event, it's not just sync(2) that is a problem. sync(2) by itself is a boot performance problem, but it's the combination of rename and sync that is known to be especially unreliable during system crashes. Statd, being a crash monitor, shouldn't depend on rename/sync to maintain persistent data in the face of system instability. I'd call that a real reason to use something more robust.

What are you talking about? Is this about the truncate + rename issue
leaving empty files upon a crash?
That issue is solved trivially by doing an fsync() before you rename the
file. That entire discussion was about whether or not existing applications should be _required_ to do this kind of POSIX pedantry, when previously they could get away without it.

Quoted 4 lines

IOW: that issue alone does not justify replacing the current simple file
based scheme.

There are other reasons, not to use the simple file-based scheme too...

Internationalized domain names will be easier to deal with via sqlite3,
for instance.

Quoted 8 lines
Quoted 1 line

Please explain...

IPv6 is used in Asia, where they almost certainly need to use non- ASCII characters in their hostnames. Internationalized domain names are stored in double-wide character sets. To provide reliable support for IDNs in statd, we will have to guarantee somehow that we can store an IDN as a file name (if we want to stay with the current scheme), no matter what file system is used for /var.

So, what's stopping us? These are POSIX filesystems. They can store any filename as long as it doesn't contain '/' or '\0'.

Quoted 10 lines

What's more, multi-homed host support will need to store multiple records for the same hostname. The mon_name is the same, but my_name is different, for each of these records. So we could do that by adding more than one line in each hostname file, but it's also a simple matter to set this up in SQL.

When we want to have statd remember things like multiple addresses for the same hostname, or whether the remote is a client or server, we will need to make more adjustments to the files.

Quoted 7 lines

As we get more and more new requirements, why lock ourselves into the current on-disk format? Using statd means we can store new fields and new records without any backwards-compatibility issues. It's all handled by the database code. So, we can think about the high level problem of getting statd to behave correctly rather than worry about the details of exactly how we are going to get the next data item stored in our current files in a backward compatible way.

Again. This is a legacy filesystem. Why are we adding requirements?

Quoted 20 lines
Quoted 7 lines
Quoted 3 lines

Certainly we could code this up ourselves, but what's the benefit to doing that when we have a perfectly good data storage engine available?

Why change something that works???? Rewriting from scratch is _NOT_ the
Linux way, and has usually bitten us hard when we've done it.

Because we are adding a bunch of new feature requirements. Internationalized domain names, multi-homed host support, IPv6 and TI- RPC, fast boot times, keeping better track of remote host addresses, keeping track of which remotes are clients and which are servers, and support for sending notifications via TCP all require significant modifications to this code base.

At some point you have to look at the code you have, and decide it's simply not going to be adequate, going forward.

Quoted 1 line

The 2.6.19 rewrite of the kernel mount code springs to mind...

Quoted 7 lines

One can just as easily argue that we've been bitten hard precisely because we've let things rot, or because we have inadequate testing for these components.

Another red herring, and especially annoying because you've known I was rewriting statd for months. Only now, when I'm done, do you say "rewriting is not the Linux way."

I have _NEVER_ agreed to a rewrite of the storage formats. You sprang this crap on me a month ago, and I made my feelings quite clear then.

#20 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 8 lines · Trond Myklebust
Quoted 4 lines · Chuck Lever

On Sep 9, 2009, at 2:39 PM, Trond Myklebust wrote:
The old statd still exists in nfs-utils. The new statd is an entirely separate component. Distributions can continue to use the old statd as long as they want. This is a red herring.

Bullshit. If they are adding IPv6 support, then they will have to upgrade at some point.

I don't see a problem with a distribution upgrade using old statd and a fresh install using new statd. You have to install a lot of new components to get NFS/IPv6 support. It's not like the only thing that needs to change is statd. People will install a new distribution to get IPv6 support. With so many simple ways to install from scratch, the days of someone upgrading just a few pieces of an old system to get a new feature, especially one as extensive as NFS/IPv6, are long gone.

I don't hear a lot of distributors objecting to this idea.

And you have never clearly answered why it wouldn't be enough to add a little code to convert the current on-disk format to sqlite3 when upgrading to the new statd, if upgradability is truly an important requirement. Possibly this is because it eliminates the only real technical objection you have to using sqlite3 here.

Quoted 14 lines
Quoted 4 lines
Quoted 3 lines

Simplicity is another reason. WTF do we need a full SQL database, when all we want to do is store 2 pieces of data (a hostname and a cookie)? It isn't as if this has been a major problem for us previously.

Quoted 9 lines

Because we are not storing just a hostname and a cookie. We are storing several different data items for each host, and we need to search over the records, and provide uniqueness constraints, and handle data conversion (for binary data like the cookie, for string data like the hostname, and for integers, like the prog/vers/proc tuple). We need to store them durably on persistent storage to have some protection against crashes. These are all things that an embedded database can do well, and that we therefore don't have to code ourselves.

Quoted 4 lines

Speaking of red herrings. Why are we adding all this crap?

This is a legacy filesystem! We shouldn't not be rewriting NLM/NSM from scratch, just add minimal support for IPv6.

You and Bruce brought up a number of work items related to statd, including having distinct statd behavior for remotes who are clients and remotes who are servers. Tom Talpey suggested we needed to send multiple SM_NOTIFY requests to each host, and use TCP to do it when possible, and you even specifically encouraged me to read his connectathon presentation on this. If Asian countries are driving the IPv6 requirement, why wouldn't they want IDN support as well? Interoperable NFS/IPv6 support requires TI-RPC. Plus, NFS/IPv6 practically requires multi-homed NLM/NSM support -- see Alex's RFC draft for details on that.

Which would you like me to drop?

Let me also point out that old statd is already broken in a number of ways, and I certainly haven't heard a lot of complaints about it. Our client NLM has sent "0" as our NSM state number for years, for example. Thus I hardly think there is a lot of risk in making changes here. It can only get better.

Quoted 38 lines
Quoted 29 lines
Quoted 29 lines
Quoted 28 lines
Quoted 17 lines
Quoted 7 lines

In any event, it's not just sync(2) that is a problem. sync(2) by itself is a boot performance problem, but it's the combination of rename and sync that is known to be especially unreliable during system crashes. Statd, being a crash monitor, shouldn't depend on rename/sync to maintain persistent data in the face of system instability. I'd call that a real reason to use something more robust.

What are you talking about? Is this about the truncate + rename issue
leaving empty files upon a crash?
That issue is solved trivially by doing an fsync() before you rename the
file. That entire discussion was about whether or not existing applications should be _required_ to do this kind of POSIX pedantry, when previously they could get away without it.

Quoted 4 lines

IOW: that issue alone does not justify replacing the current simple file
based scheme.

There are other reasons, not to use the simple file-based scheme too...

Internationalized domain names will be easier to deal with via sqlite3,
for instance.

Quoted 8 lines
Quoted 1 line

Please explain...

IPv6 is used in Asia, where they almost certainly need to use non- ASCII characters in their hostnames. Internationalized domain names are stored in double-wide character sets. To provide reliable support for IDNs in statd, we will have to guarantee somehow that we can store an IDN as a file name (if we want to stay with the current scheme), no matter what file system is used for /var.

Quoted 2 lines

So, what's stopping us? These are POSIX filesystems. They can store any filename as long as it doesn't contain '/' or '\0'.

IDNs are UTF16. /var therefore has to support UTF16 filenames; either byte in a double-byte character can be '/' or '\0'. That means the underlying fs implementation has to support UTF16 (FAT32 anyone?), and the system's locale has to be configured correctly. If we decide not to depend on the file system to support UTF16 filenames, then statd has to be intelligent enough to figure out how to deal with converting UTF16 hostnames before storing them as filenames. Then, we have to teach matchhostname() and friends how to deal with double-byte character strings...

Or we just tell sqlite3 that this is a double-byte character string, and let it handle the collation and on-disk storage details for us.

The point is, this is yet another detail we have to either worry about and open code in statd, or we can simply rely on what's already provided in sqlite3. No one, repeat NO ONE, is arguing that you can't implement these features without sqlite3. My argument is that we quickly bury a whole bunch of details if we use sqlite3, and can then focus on larger issues. That's the prime goal of software layering with libraries.

We can open code any or all of statd. In fact the current statd open codes RPC request creation in socket buffers rather than using glibc's RPC API, and I think we agree that is not an optimal solution. The question is: should we duplicate code and bugs by open coding statd's RPC and data storage? Or should we pretend to be modern software engineers, and use widely used and known good code that other people have written already to handle these details?

Quoted 18 lines
Quoted 10 lines

What's more, multi-homed host support will need to store multiple records for the same hostname. The mon_name is the same, but my_name is different, for each of these records. So we could do that by adding more than one line in each hostname file, but it's also a simple matter to set this up in SQL.

When we want to have statd remember things like multiple addresses for the same hostname, or whether the remote is a client or server, we will need to make more adjustments to the files.

Quoted 7 lines

As we get more and more new requirements, why lock ourselves into the current on-disk format? Using statd means we can store new fields and new records without any backwards-compatibility issues. It's all handled by the database code. So, we can think about the high level problem of getting statd to behave correctly rather than worry about the details of exactly how we are going to get the next data item stored in our current files in a backward compatible way.

Quoted 1 line

Again. This is a legacy filesystem. Why are we adding requirements?

Maybe you should ask the people who are requesting NFS/IPv6 from Red Hat and other distributors. Or ask yourself why we would add an engine to allow NFSv2 to shift authentication flavors without remounting, since NFSv2 is a legacy filesystem. Or ask why you think we should add support to statd to recognize the difference between remote clients and servers, if this is a legacy filesystem.

I don't object to any of those work items, but I do have trouble with you dropping in and saying "why diddle a legacy file system" when clearly that is not a show stopper in these other cases.

Quoted 24 lines
Quoted 24 lines
Quoted 7 lines
Quoted 3 lines

Certainly we could code this up ourselves, but what's the benefit to doing that when we have a perfectly good data storage engine available?

Why change something that works???? Rewriting from scratch is _NOT_ the
Linux way, and has usually bitten us hard when we've done it.

Because we are adding a bunch of new feature requirements. Internationalized domain names, multi-homed host support, IPv6 and TI- RPC, fast boot times, keeping better track of remote host addresses, keeping track of which remotes are clients and which are servers, and support for sending notifications via TCP all require significant modifications to this code base.

At some point you have to look at the code you have, and decide it's simply not going to be adequate, going forward.

Quoted 1 line

The 2.6.19 rewrite of the kernel mount code springs to mind...

One can just as easily argue that we've been bitten hard precisely because we've let things rot, or because we have inadequate testing for these components.

Quoted 6 lines
Quoted 3 lines

Another red herring, and especially annoying because you've known I was rewriting statd for months. Only now, when I'm done, do you say "rewriting is not the Linux way."

I have _NEVER_ agreed to a rewrite of the storage formats. You sprang this crap on me a month ago, and I made my feelings quite clear then.

"Rewriting is not the Linux way" is not the same as saying you don't want to change storage formats. Don't change the subject.

The idea that "the Linux way" is the best and only way is ridiculous on its face, anyway. I mean, what do you expect when we have no requirements and specification process, no formal testing, C coding style conventions based on 20-year old coding practices, a hit-or-miss review process that relies more on reviewers' personal preferences than any kind of standards, no static code analysis tools, no defect metrics or bug meta-analysis tools, kernel debuggers are verboten, a combative mailing list environment, and parts of our knowledge base and team history are lost every time a developer leaves (in this case, Olaf and Neil)? It's no wonder we never change anything unless absolutely necessary!

You told me to implement IPv6 support in statd. Now you are spitting on what I worked out without any guidance from you because you're too busy working on IETF standards and NFSv4.1 to bother discussing "legacy" code, other than to say "ewe!". Just how should I react to that, pray tell?

Clearly you do not want to admit that even "minimal" IPv6 support is a significant effort, especially given how far behind the Linux RPC and NLM/NSM implementations are. Yelling at me, throwing a bunch of generic objections up, and calling my code "crap" is not going to make the problem any simpler.

Cooperating with me to get your way and giving specific and constructive criticism will go a lot farther than your aggressive and disrespectful attitude and obnoxious tone. I will happily sit down and discuss this with you in a rational tone, but I will no longer tolerate your unwarranted harassment on a public mailing list.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#21 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 7 lines · Chuck Lever
Quoted 6 lines · Trond Myklebust
Quoted 4 lines · Chuck Lever

On Sep 9, 2009, at 2:39 PM, Trond Myklebust wrote:
The old statd still exists in nfs-utils. The new statd is an entirely separate component. Distributions can continue to use the old statd as long as they want. This is a red herring.

Quoted 6 lines
Quoted 2 lines

Bullshit. If they are adding IPv6 support, then they will have to upgrade at some point.

I don't see a problem with a distribution upgrade using old statd and a fresh install using new statd. You have to install a lot of new components to get NFS/IPv6 support.

What new components that are not already being installed??

Quoted 5 lines

It's not like the only thing that needs to change is statd. People will install a new distribution to get IPv6 support. With so many simple ways to install from scratch, the days of someone upgrading just a few pieces of an old system to get a new feature, especially one as extensive as NFS/IPv6, are long gone.

I'm not sure how people could only install bits and pieces of nfs-utils... Even 'make install' in the git tree installs everything...

Quoted 6 lines

And you have never clearly answered why it wouldn't be enough to add a little code to convert the current on-disk format to sqlite3 when upgrading to the new statd, if upgradability is truly an important requirement. Possibly this is because it eliminates the only real technical objection you have to using sqlite3 here.

The issue I would have with using sqlite3 is it would add yet another requirement on nfs-utils... I really don't know how big sqlite3 and/or sqlite3-devel (possibly needed for builds) packages are but it just one more thing will be need for nfs-utils to function...

Quoted 20 lines
Quoted 15 lines
Quoted 5 lines
Quoted 4 lines

Simplicity is another reason. WTF do we need a full SQL database, when all we want to do is store 2 pieces of data (a hostname and a cookie)? It isn't as if this has been a major problem for us previously.

Quoted 9 lines

Because we are not storing just a hostname and a cookie. We are storing several different data items for each host, and we need to search over the records, and provide uniqueness constraints, and handle data conversion (for binary data like the cookie, for string data like the hostname, and for integers, like the prog/vers/proc tuple). We need to store them durably on persistent storage to have some protection against crashes. These are all things that an embedded database can do well, and that we therefore don't have to code ourselves.

Quoted 4 lines

Speaking of red herrings. Why are we adding all this crap?

This is a legacy filesystem! We shouldn't not be rewriting NLM/NSM from scratch, just add minimal support for IPv6.

Quoted 10 lines

You and Bruce brought up a number of work items related to statd, including having distinct statd behavior for remotes who are clients and remotes who are servers. Tom Talpey suggested we needed to send multiple SM_NOTIFY requests to each host, and use TCP to do it when possible, and you even specifically encouraged me to read his connectathon presentation on this. If Asian countries are driving the IPv6 requirement, why wouldn't they want IDN support as well? Interoperable NFS/IPv6 support requires TI-RPC. Plus, NFS/IPv6 practically requires multi-homed NLM/NSM support -- see Alex's RFC draft for details on that.

So a database is needed to accomplish all this?

Quoted 7 lines

Let me also point out that old statd is already broken in a number of ways, and I certainly haven't heard a lot of complaints about it. Our client NLM has sent "0" as our NSM state number for years, for example. Thus I hardly think there is a lot of risk in making changes here. It can only get better.

I can agree with you here...

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

IPv6 is used in Asia, where they almost certainly need to use non- ASCII characters in their hostnames. Internationalized domain names are stored in double-wide character sets. To provide reliable support for IDNs in statd, we will have to guarantee somehow that we can store an IDN as a file name (if we want to stay with the current scheme), no matter what file system is used for /var.

So, what's stopping us? These are POSIX filesystems. They can store any filename as long as it doesn't contain '/' or '\0'.

Quoted 9 lines

IDNs are UTF16. /var therefore has to support UTF16 filenames; either byte in a double-byte character can be '/' or '\0'. That means the underlying fs implementation has to support UTF16 (FAT32 anyone?), and the system's locale has to be configured correctly. If we decide not to depend on the file system to support UTF16 filenames, then statd has to be intelligent enough to figure out how to deal with converting UTF16 hostnames before storing them as filenames. Then, we have to teach matchhostname() and friends how to deal with double-byte character strings...

Has this been a problem in the past? How are other implementations dealing with this? Have they gone to use a db as well?

Quoted 10 lines

Or we just tell sqlite3 that this is a double-byte character string, and let it handle the collation and on-disk storage details for us.

The point is, this is yet another detail we have to either worry about and open code in statd, or we can simply rely on what's already provided in sqlite3. No one, repeat NO ONE, is arguing that you can't implement these features without sqlite3. My argument is that we quickly bury a whole bunch of details if we use sqlite3, and can then focus on larger issues. That's the prime goal of software layering with libraries.

What kind of performance hit will there be (if any)? The nice thing about a file is you only have to read it once in to a cache verses doing a number of queries... or can one also cache queries?

Quoted 8 lines

We can open code any or all of statd. In fact the current statd open codes RPC request creation in socket buffers rather than using glibc's RPC API, and I think we agree that is not an optimal solution. The question is: should we duplicate code and bugs by open coding statd's RPC and data storage? Or should we pretend to be modern software engineers, and use widely used and known good code that other people have written already to handle these details?

I'm all for using moving forward with "modern software" but, as a common theme with me, I'm always worried about becoming needlessly complicated or over engineering... which might be the case with having statd use a db...

steved.

#22 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 8 lines · Steve Dickson
Quoted 7 lines · Chuck Lever
Quoted 6 lines · Trond Myklebust
Quoted 4 lines · Chuck Lever

On Sep 9, 2009, at 2:39 PM, Trond Myklebust wrote:
The old statd still exists in nfs-utils. The new statd is an entirely separate component. Distributions can continue to use the old statd as long as they want. This is a red herring.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

Bullshit. If they are adding IPv6 support, then they will have to upgrade at some point.

I don't see a problem with a distribution upgrade using old statd and a fresh install using new statd. You have to install a lot of new components to get NFS/IPv6 support.

What new components that are not already being installed??

You need a kernel that can do NFS/IPv6, you need to install rpcbind and libtirpc, you need the new mount command, you need all the user space network pieces to manage IPv6, you need to consider firewall and address distribution on your local network, and you need statd and mountd/exportfs to get NFS/IPv6 support.

Configuring a system for IPv6 support can also be nontrivial, and not something people will do on a whim.

I didn't mean to imply that some of these components are not already installed. My point is that the required changes for NFS/IPv6 are wide spread, and that most people would opt for installing a new OS on their systems to get these features, rather than upgrade all of these items piecemeal.

Quoted 9 lines
Quoted 5 lines

And you have never clearly answered why it wouldn't be enough to add a little code to convert the current on-disk format to sqlite3 when upgrading to the new statd, if upgradability is truly an important requirement. Possibly this is because it eliminates the only real technical objection you have to using sqlite3 here.

The issue I would have with using sqlite3 is it would add yet another requirement on nfs-utils... I really don't know how big sqlite3 and/or sqlite3-devel (possibly needed for builds) packages are but it just one more thing will be need for nfs-utils to function...

sqlite3.org provides a single source file version of sqlite3 that is licensed and designed explicitly for folks to include in their own code, without the need for linking a library. You can even disable a number of build time options to reduce object size.

This means that the libsqlite3 and libsqlite3-devel packages would not be required on either the build system or the end system, and it eliminates the issue of whether libsqlite3.so can be moved to /lib.

Quoted 29 lines
Quoted 19 lines
Quoted 14 lines
Quoted 4 lines
Quoted 3 lines

Simplicity is another reason. WTF do we need a full SQL database, when all we want to do is store 2 pieces of data (a hostname and a cookie)? It isn't as if this has been a major problem for us previously.

Quoted 9 lines

Because we are not storing just a hostname and a cookie. We are storing several different data items for each host, and we need to search over the records, and provide uniqueness constraints, and handle data conversion (for binary data like the cookie, for string data like the hostname, and for integers, like the prog/vers/proc tuple). We need to store them durably on persistent storage to have some protection against crashes. These are all things that an embedded database can do well, and that we therefore don't have to code ourselves.

Quoted 4 lines

Speaking of red herrings. Why are we adding all this crap?

This is a legacy filesystem! We shouldn't not be rewriting NLM/NSM from scratch, just add minimal support for IPv6.

Quoted 10 lines

You and Bruce brought up a number of work items related to statd, including having distinct statd behavior for remotes who are clients and remotes who are servers. Tom Talpey suggested we needed to send multiple SM_NOTIFY requests to each host, and use TCP to do it when possible, and you even specifically encouraged me to read his connectathon presentation on this. If Asian countries are driving the IPv6 requirement, why wouldn't they want IDN support as well? Interoperable NFS/IPv6 support requires TI-RPC. Plus, NFS/IPv6 practically requires multi-homed NLM/NSM support -- see Alex's RFC draft for details on that.

Quoted 1 line

So a database is needed to accomplish all this?

No, a database is not specifically required.

However, libsqlite3 is a library that contains all of the elements -- durable on-disk storage, proper data conversion for binary blobs, single- and double-width character strings, integers, the ability to constrain record uniqueness, the ability to add new data items easily to each record, and a facility for collating and searching the host records.

sqlite3 is an embedded database, meaning the implementation is purposely smaller than a full SQL database, and is designed explicitly to have zero database administration requirements. sqlite3 is designed for managing data for long-running network daemons, and it is widely used for that purpose.

If there is some other pre-existing code that can do this, I'm open to considering it.

Quoted 27 lines
Quoted 6 lines

Let me also point out that old statd is already broken in a number of ways, and I certainly haven't heard a lot of complaints about it. Our client NLM has sent "0" as our NSM state number for years, for example. Thus I hardly think there is a lot of risk in making changes here. It can only get better.

I can agree with you here...

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

IPv6 is used in Asia, where they almost certainly need to use non- ASCII characters in their hostnames. Internationalized domain names are stored in double-wide character sets. To provide reliable support for IDNs in statd, we will have to guarantee somehow that we can store an IDN as a file name (if we want to stay with the current scheme), no matter what file system is used for /var.

So, what's stopping us? These are POSIX filesystems. They can store any filename as long as it doesn't contain '/' or '\0'.

Quoted 9 lines

IDNs are UTF16. /var therefore has to support UTF16 filenames; either byte in a double-byte character can be '/' or '\0'. That means the underlying fs implementation has to support UTF16 (FAT32 anyone?), and the system's locale has to be configured correctly. If we decide not to depend on the file system to support UTF16 filenames, then statd has to be intelligent enough to figure out how to deal with converting UTF16 hostnames before storing them as filenames. Then, we have to teach matchhostname() and friends how to deal with double-byte character strings...

Quoted 2 lines

Has this been a problem in the past? How are other implementations dealing with this? Have they gone to use a db as well?

No, IDNs are recent, but it is reasonable to think that internationalized domain names is a feature that would appeal to the same folks who are driving the IPv6 requirement. This is not a hard requirement, but it is one reason why statd's current on-disk format is not adequate.

Yes, I understand that there are some statd implementations that use a database rather than flat files. statd is nothing if not exactly a mechanism for storing structured data across system crashes. That's exactly what databases are for.

Quoted 9 lines
Quoted 9 lines

Or we just tell sqlite3 that this is a double-byte character string, and let it handle the collation and on-disk storage details for us.

The point is, this is yet another detail we have to either worry about and open code in statd, or we can simply rely on what's already provided in sqlite3. No one, repeat NO ONE, is arguing that you can't implement these features without sqlite3. My argument is that we quickly bury a whole bunch of details if we use sqlite3, and can then focus on larger issues. That's the prime goal of software layering with libraries.

Quoted 3 lines

What kind of performance hit will there be (if any)? The nice thing about a file is you only have to read it once in to a cache verses doing a number of queries... or can one also cache queries?

sqlite3's performance for the statd application would actually be better than what we have today.

Naturally the database is cached in memory, making queries as fast as memory reads. The better performance comes with record insertion and deletion. Today statd does a file create and then an O_SYNC write to that file. This requires synchronous metadata updates to the file system to create the new file and create a new directory entry for it. If the directory becomes large, creating a new directory entry becomes even slower. Likewise for record deletion, multiple synchronous metadata updates are required to remove the directory entry and the file containing the host record.

With sqlite3 (or any database style solution) record insertion and deletion can usually be handled with a single O_SYNC write to the database file.

You could argue that using sqlite3 means more CPU and memory consumption. Perhaps, but that's a less onerous resource requirement than synchronous disk activity, in my view.

Quoted 11 lines
Quoted 7 lines

We can open code any or all of statd. In fact the current statd open codes RPC request creation in socket buffers rather than using glibc's RPC API, and I think we agree that is not an optimal solution. The question is: should we duplicate code and bugs by open coding statd's RPC and data storage? Or should we pretend to be modern software engineers, and use widely used and known good code that other people have written already to handle these details?

I'm all for using moving forward with "modern software" but, as a common theme with me, I'm always worried about becoming needlessly complicated or over engineering... which might be the case with having statd use a db...

Consider what would happen if we open coded all of the details of on-disk storage and record searching into statd itself. I think something like sqlite3 is a better and less complex solution than open coding because all these details are moved out of statd into a pre-existing library, thus making statd itself architecturally simpler, and therefore easier to understand and maintain.

The one weakness here is the dependence on SQL. That makes the statd code uglier and more complex than I would like, and is something I want to address.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#23 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

In reply to Chuck Lever
Quoted 11 lines · Chuck Lever

The idea that "the Linux way" is the best and only way is ridiculous on its face, anyway. I mean, what do you expect when we have no requirements and specification process, no formal testing, C coding style conventions based on 20-year old coding practices, a hit-or-miss review process that relies more on reviewers' personal preferences than any kind of standards, no static code analysis tools, no defect metrics or bug meta-analysis tools, kernel debuggers are verboten, a combative mailing list environment, and parts of our knowledge base and team history are lost every time a developer leaves (in this case, Olaf and Neil)? It's no wonder we never change anything unless absolutely necessary!

And yet is largely works!
I could summarise a lot of your points by observing that the community values people over process. I really think that is the right place to put value, because people are richer and more flexible than process.

I agree that combative mailing lists are a problem, but even there, I believe most of the aggression is more perceived than real, and that a graceful, humble, polite attitude can have a positive-feedback effect too.

Yes, there are lots of practices that might improve things that we don't have standardised. But one practice we do have that has proven very effective is incremental refinement. It can be hard to understand what order to make changes until after you have made them, but once you understand what you want to do, going back and doing it in logical order really is very effective. It makes it easier for others to review, it makes it easy for you to review yourself. It means less controversial bits can be included quickly leaving room for the more controversial bits to be discussed in isolation.

I think that the switch from portmap to rpcbind was a bad idea, and I think that a wholesale replacement of statd is probably a bad idea too. It might seem like the easiest way to get something useful working, but you'll probably be paying the price for years as little regression turn up because you didn't completely understand the original statd (and face it, who does?)

As for the use of sql-lite ... I must admit that I wouldn't choose it. Maybe it is a good idea. If it is, you probably need to merge that change early with a clear argument and tools to make it manageable (e.g. a developer will want to tool to be able to look inside the database easily and make changes, without having to know sql). It is much easier to discuss one thing at a time on these combative mailing lists ;-)

NeilBrown

P.S.
And we do have static code analysis tools. Both 'gcc' and 'sparse' fit that description.

#24 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 13 lines · NeilBrown
Quoted 11 lines · Chuck Lever

The idea that "the Linux way" is the best and only way is ridiculous on its face, anyway. I mean, what do you expect when we have no requirements and specification process, no formal testing, C coding style conventions based on 20-year old coding practices, a hit-or-miss review process that relies more on reviewers' personal preferences than any kind of standards, no static code analysis tools, no defect metrics or bug meta-analysis tools, kernel debuggers are verboten, a combative mailing list environment, and parts of our knowledge base and team history are lost every time a developer leaves (in this case, Olaf and Neil)? It's no wonder we never change anything unless absolutely necessary!

Quoted 4 lines

And yet is largely works!
I could summarise a lot of your points by observing that the community values people over process. I really think that is the right place to put value, because people are richer and more flexible than process.

Agreed, but there are risks to that approach as well, which are largely ignored by the Linux community. The last point in my list is probably the biggest risk: when people leave, we are stuck with decades-old code that no-one understands. Cf: statd.

My point is that many of the items I mentioned above are expressly designed to allow quicker, less risky change, precisely to decrease the amount of time and effort to get new features into our code. Yet we turn our back on all of them in favor of an antique "don't touch that!" policy. "Don't touch that!" is not a reasonable argument against replacing components that need to be replaced.

Quoted 4 lines

I agree that combative mailing lists are a problem, but even there, I believe most of the aggression is more perceived than real, and that a graceful, humble, polite attitude can have a positive-feedback effect too.

Years ago I believed that, but I have seen much evidence to the contrary in this community. More often such an attitude is entirely ignored, or treated as an invitation for abuse, especially by people who have no interest in politeness. This kind of approach has no effect on the leaders in the Linux community, who set an example of extreme rudeness and belligerence.

I've made an effort to stop arguing small points, and to make observations and not argue. I still get e-mail full of "crap" this and "bullshit" that and "NACK!" with little explanation.

Quoted 9 lines

Yes, there are lots of practices that might improve things that we don't have standardised. But one practice we do have that has proven very effective is incremental refinement. It can be hard to understand what order to make changes until after you have made them, but once you understand what you want to do, going back and doing it in logical order really is very effective. It makes it easier for others to review, it makes it easy for you to review yourself. It means less controversial bits can be included quickly leaving room for the more controversial bits to be discussed in isolation.

I am a fan of incremental refinement, and I use that approach as often as I can. There are some things that incremental refinement cannot do, however.

Quoted 6 lines

I think that the switch from portmap to rpcbind was a bad idea, and I think that a wholesale replacement of statd is probably a bad idea too. It might seem like the easiest way to get something useful working, but you'll probably be paying the price for years as little regression turn up because you didn't completely understand the original statd (and face it, who does?)

Yes, but _why_ is it a bad idea? All I hear is "this is a bad idea" and "you could do it some other way" but these are qualitative, not quantitative arguments. They are religious statements, not specific technical criticisms.

This leaves me with the impression that folks are responding out of a fear of the unknown, and not out of a considered technical opinion. If sqlite3 is outside of people's comfort zone, that's OK. Please let's be honest about it instead of slinging mud and throwing up a bunch of generic arguments that no one can rebut.

Quoted 7 lines

As for the use of sql-lite ... I must admit that I wouldn't choose it. Maybe it is a good idea. If it is, you probably need to merge that change early with a clear argument and tools to make it manageable (e.g. a developer will want to tool to be able to look inside the database easily and make changes, without having to know sql). It is much easier to discuss one thing at a time on these combative mailing lists ;-)

That's a fine and constructive comment, thanks.

There is already a tool for managing the data in the database: 'sqlite3' the executable, which can be used in shell scripts. There are also sqlite3 libraries for Python and Perl and C. This is really not very different from POSIX file system calls and using 'cat'. SQL is not difficult to learn, and a one or two page recipe document is easy to provide. I certainly have not used any advanced features of SQL to implement new statd.

Given the complexity of the change, it makes it much easier to argue against sqlite3, however, if it is separated from the set of changes that motivate its use. Bruce, for example, has stated to me specifically that he prefers having such changes and their motivational requirements included in the same patch. I regularly code for several different maintainers, and each one has his own preferences, often contradicting other maintainers. I don't think regular maintainers have any idea how confusing and challenging this is.

Note, however, I am not married to the specifics of sqlite3. What I am attached to is the ideas that the current system is inadequate for the kinds of features we want to add to statd, and that statd should not worry about the details of data storage, or search and management of the host records, because we have other tools that are better at this. If there is another solution that provides a durable and flexible way to store and search host records and offload the details to pre-existing code, I'm open. However, sqlite3 is the most widely used embedded database on the planet, and is eminently suitable for this task.

Quoted 2 lines

And we do have static code analysis tools. Both 'gcc' and 'sparse' fit that description.

Yes, gcc can provide some static analysis, if the correct options are specified, and care is taken to eliminate the noise of false positives. There is a prevailing attitude, however, that this is a worthless endeavor. Witness the amount of noise that comes out when you build a Red Hat kernel or the nfs-utils package. Witness also the sarcasm of Linus who repeatedly chides folks for not running sparse regularly.

Additionally, gcc is not the best tool for this job, given the often oblique way it calls out errors and warnings. There are purify, fortify, and splint, just to name three, that are standard analysis tools we don't even consider.

My comment goes more to the point that static analysis is not considered of any value.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#25 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 21 lines · On Thursday September 10, [email protected] wrote:
Quoted 15 lines · NeilBrown
Quoted 5 lines · Chuck Lever

The idea that "the Linux way" is the best and only way is ridiculous on its face, anyway. I mean, what do you expect when we have no requirements and specification process, no formal testing, C coding style conventions based on 20-year old coding practices, a hit-or- miss

Quoted 8 lines

review process that relies more on reviewers' personal preferences than any kind of standards, no static code analysis tools, no defect metrics or bug meta-analysis tools, kernel debuggers are verboten, a combative mailing list environment, and parts of our knowledge base and team history are lost every time a developer leaves (in this case,
Olaf and Neil)? It's no wonder we never change anything unless absolutely necessary!

Quoted 4 lines

And yet is largely works!
I could summarise a lot of your points by observing that the community values people over process. I really think that is the right place to put value, because people are richer and more flexible than process.

Quoted 5 lines

Agreed, but there are risks to that approach as well, which are largely ignored by the Linux community. The last point in my list is probably the biggest risk: when people leave, we are stuck with decades-old code that no-one understands. Cf: statd.

Much of my understanding of statd is embedded in the code in nfs-utils, and in the change log (which is much better since we changed to git).
By discarding all that and creating a new project you risk losing that legacy.

There can certainly be a case of discarding and starting again. I did that with the support tools for raid (raidtools is no-more, mdadm reigns :-). But this is not a decision to be taken lightly and is not without it's costs. The fact that raidtools was by that time essentially an unmaintained orphan made some of those costs unavoidable in my case.

You obviously could do the same thing - you don't need anyone's
permission: create a new project called "statd" and make it whatever you want and hope that distributors will pick it up. If your statd supports IPv6, and the nfs-utils one does not, then there is a good chance that distros will pick it up as people like to see that big tick next to "IPv6 support".

Maintaining and developing such a thing long enough to establish a self-sustaining community would be a big effort. You would need to compare that with the effort of taking the incremental approach and revising the code in nfs-utils and getting those revisions accepted. That would probably have a higher development cost, but a lower maintenance cost. It is very hard to know up front which cost is lower.

But you will leave one day. How can you best make sure that you leave something that others can maintain????

Quoted 6 lines

My point is that many of the items I mentioned above are expressly designed to allow quicker, less risky change, precisely to decrease the amount of time and effort to get new features into our code. Yet we turn our back on all of them in favor of an antique "don't touch that!" policy. "Don't touch that!" is not a reasonable argument against replacing components that need to be replaced.

The only "Don't touch that" which I am aware of relates to interfaces, particularly with established code.
In the case of statd, the files in sm/ and sm.bak/ are a well established interface. Exactly how much is dependant on it is hard to say. Not much formal code I expect but maybe some obscure scripts and lots of sysadmin knowledge.

Can you run them both in parallel?? i.e. have a database with all the data, but also store it in the files (if the hostname can be represented in ASCII)... It is hard to guess how easy that would be and how worthwhile it would be. And it doesn't answer the question of whether sqlite is stable enough.

Quoted 7 lines
Quoted 6 lines

I agree that combative mailing lists are a problem, but even there, I believe most of the aggression is more perceived than real, and that a graceful, humble, polite attitude can have a positive-feedback effect
too.

Quoted 6 lines

Years ago I believed that, but I have seen much evidence to the contrary in this community. More often such an attitude is entirely ignored, or treated as an invitation for abuse, especially by people who have no interest in politeness. This kind of approach has no effect on the leaders in the Linux community, who set an example of extreme rudeness and belligerence.

That last comment is interesting. At the 2007 kernel summit (the last one I was at) the topic of mailing list etiquette was discussed and there seemed to be agreement that we, the leaders (I guess the kernel summit attendees are the closest we have to group leadership) have a role in damping down the fire, not building it up. My feeling is that most high profile people do quite well, but maybe I am too forgiving.

Quoted 4 lines

I've made an effort to stop arguing small points, and to make observations and not argue. I still get e-mail full of "crap" this and "bullshit" that and "NACK!" with little explanation.

I certainly agree that sort of response is best not sent. And I must confess to a recent experience (on a different list) where I gave up
due to similar behaviour. That is partly why I decided to join in
this discussion (though I'm not sure if I'm being helpful yet).

Quoted 24 lines
Quoted 12 lines

Yes, there are lots of practices that might improve things that we don't
have standardised. But one practice we do have that has proven very effective is incremental refinement. It can be hard to understand what
order to make changes until after you have made them, but once you understand what you want to do, going back and doing it in logical order really is very effective. It makes it easier for others to review, it makes it easy for you to review yourself. It means less controversial bits can be included quickly leaving room for the more controversial bits to be discussed in isolation.

I am a fan of incremental refinement, and I use that approach as often as I can. There are some things that incremental refinement cannot do, however.

Quoted 6 lines

I think that the switch from portmap to rpcbind was a bad idea, and I think that a wholesale replacement of statd is probably a bad idea too. It might seem like the easiest way to get something useful working, but you'll probably be paying the price for years as little regression turn up because you didn't completely understand the original statd (and face it, who does?)

Quoted 4 lines

Yes, but _why_ is it a bad idea? All I hear is "this is a bad idea" and "you could do it some other way" but these are qualitative, not quantitative arguments. They are religious statements, not specific technical criticisms.

It is a bad idea because it doesn't have the legacy of testing and refinement. Almost as soon as we started using it bugs were found - or at least differences in behaviour to portmap (something about the privilege level required to register a binding I think).

Now I admit that no one put their hand up to add IPv6 support to portmap, arguably it could have been a worse idea to stay with portmap as it meant no IPv6. But changing was still a bad idea.

Had we (had the man power to) incrementally enhance portmap we would have had a much more reviewable process, and a bisectable result which would allow regression to be isolated more directly.

Quoted 26 lines

This leaves me with the impression that folks are responding out of a fear of the unknown, and not out of a considered technical opinion. If sqlite3 is outside of people's comfort zone, that's OK. Please let's be honest about it instead of slinging mud and throwing up a bunch of generic arguments that no one can rebut.

Quoted 8 lines

As for the use of sql-lite ... I must admit that I wouldn't choose it. Maybe it is a good idea. If it is, you probably need to merge that change early with a clear argument and tools to make it manageable
(e.g. a developer will want to tool to be able to look inside the database easily and make changes, without having to know sql). It is much easier to discuss one thing at a time on these combative mailing lists ;-)

That's a fine and constructive comment, thanks.

There is already a tool for managing the data in the database:
'sqlite3' the executable, which can be used in shell scripts. There are also sqlite3 libraries for Python and Perl and C. This is really not very different from POSIX file system calls and using 'cat'. SQL is not difficult to learn, and a one or two page recipe document is easy to provide. I certainly have not used any advanced features of SQL to implement new statd.

Quoted 9 lines

Given the complexity of the change, it makes it much easier to argue against sqlite3, however, if it is separated from the set of changes that motivate its use. Bruce, for example, has stated to me specifically that he prefers having such changes and their motivational requirements included in the same patch. I regularly code for several different maintainers, and each one has his own preferences, often contradicting other maintainers. I don't think regular maintainers have any idea how confusing and challenging this is.

:-)

I think there has to be room for balance here. Certainly it is best to use new functionality as soon as it is added so the use case is clear. But I can imagine that converting from a files database to a sqlite database would be several patches in itself. And then if you want to start adding 'search' or 'utf-16' functionality, that would be more patches still. Combining all that in to one big patch just to keep the change with the motivation seems unlikely to be a win from anyone's perspective.
Certainly having a real big changelog comment early on that explains the value of sqlite would be essential, and that patch would not be merged until the subsequent ones were reviewed.

Quoted 22 lines

Note, however, I am not married to the specifics of sqlite3. What I am attached to is the ideas that the current system is inadequate for the kinds of features we want to add to statd, and that statd should not worry about the details of data storage, or search and management of the host records, because we have other tools that are better at this. If there is another solution that provides a durable and flexible way to store and search host records and offload the details to pre-existing code, I'm open. However, sqlite3 is the most widely used embedded database on the planet, and is eminently suitable for this task.

Quoted 2 lines

And we do have static code analysis tools. Both 'gcc' and 'sparse' fit that description.

Yes, gcc can provide some static analysis, if the correct options are specified, and care is taken to eliminate the noise of false positives. There is a prevailing attitude, however, that this is a worthless endeavor. Witness the amount of noise that comes out when you build a Red Hat kernel or the nfs-utils package. Witness also the sarcasm of Linus who repeatedly chides folks for not running sparse regularly.

Quoted 7 lines

Additionally, gcc is not the best tool for this job, given the often oblique way it calls out errors and warnings. There are purify, fortify, and splint, just to name three, that are standard analysis tools we don't even consider.

My comment goes more to the point that static analysis is not considered of any value.

You might be right....

I see there be a "law of diminishing returns" here.
sparse (aka "make C=1") has almost never shown me anything interesting. So while there is a (small) cost in running it, there is almost no perceived value.

For my own C projects I always compile with "-Wall -Werror" to remove the cost of running it and to artificially increase the value.

I just tried "make C=1" on a couple of bits of kernel code and it did provide some vaguely interesting things that I'll probably fix. But there was a lot of noise like:
warning: potentially expensive pointer subtraction
which I don't think I want to fix, but don't know how to silence the warning for just that case. It would be nice of "C=1" was a default
and either the warning, or sparse, were fixed. That might increase
the perception that this sort of thing was of value.

Some time ago Greg Banks went on a pursuit of warnings in nfs-utils and got rid of all of them -- except those generated by rpcgen. Have more been introduced?

NeilBrown

A message that is not in the archive (sent off-list or not yet received): 19117.60405.793389.323010-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org

#27 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 4 lines · Neil Brown

Some time ago Greg Banks went on a pursuit of warnings in nfs-utils and got rid of all of them -- except those generated by rpcgen. Have more been introduced?

The only other warnings I'm struggling with are the
warning: dereferencing pointer 'local_addr' does break strict-aliasing rules

in sm-notify.c, which seem to be a bunch of noise... But I will like to get ride of them... Other than that, there are no warnings that I know of...

steved.

#28 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

In reply to Neil Brown
Quoted 5 lines · Neil Brown
Quoted 1 line · On Thursday September 10, [email protected] wrote:

On Sep 10, 2009, at 4:44 AM, NeilBrown wrote:

But you will leave one day. How can you best make sure that you leave something that others can maintain????

By writing code that is self-explanatory, providing lots of comments in the code, adding to the git log (as you suggested) and writing expansive man pages that describe the interfaces in as clear a manner as possible. The review process is also part of that effort.

There is also the possibility of mentoring others, as FreeBSD does, and providing extensive written documentation and specifications in wikis. Agile methodologies suggest that rewriting as a regular practice is a good way for a team to retain familiarity with a code base. Having a full test suite that can be used to verify the behavior of new or existing code is also a way to codify requirements and create an institutional memory of regressions, as well as to insulate users from regressions in new code.

Quoted 7 lines
Quoted 6 lines

My point is that many of the items I mentioned above are expressly designed to allow quicker, less risky change, precisely to decrease the amount of time and effort to get new features into our code. Yet we turn our back on all of them in favor of an antique "don't touch that!" policy. "Don't touch that!" is not a reasonable argument against replacing components that need to be replaced.

Quoted 6 lines

The only "Don't touch that" which I am aware of relates to interfaces, particularly with established code.
In the case of statd, the files in sm/ and sm.bak/ are a well established interface. Exactly how much is dependant on it is hard to say. Not much formal code I expect but maybe some obscure scripts and lots of sysadmin knowledge.

There is no documentation I'm aware of of statd's on-disk format as a formal interface. I have had some recent conversations with Lon about this, to handle any dependencies his clustering scripts may have, and he didn't throw up any flags. He told me that all we needed was to provide a mechanism to access this data from a shell script, which we would have in 'sqlite3' the executable.

So this is a new requirement (to me, anyway). If these files constitute a formal interface, how can statd be modified to store additional data or new data types in these files? Am I allowed to put IPv6 presentation addresses in these files in place of IPv4 addresses? Am I allowed to add new fields? Not rhetorical questions... really... how should I go about doing this and testing the result? You seem to be suggesting that the sm/* files can't be used for the kind of features we want to add.

Quoted 5 lines

Can you run them both in parallel?? i.e. have a database with all the data, but also store it in the files (if the hostname can be represented in ASCII)... It is hard to guess how easy that would be and how worthwhile it would be. And it doesn't answer the question of whether sqlite is stable enough.

Is it even a good thing to freeze the sm/* files as a formal interface, or should we go about providing a real documented programming interface for this, and migrate to it? There is a real risk to maintaining undocumented interfaces like this, and that is that we can't make any change to this code without a significant possibility of breaking something.

Quoted 17 lines
Quoted 11 lines
Quoted 6 lines

I think that the switch from portmap to rpcbind was a bad idea, and I think that a wholesale replacement of statd is probably a bad idea too. It might seem like the easiest way to get something useful working, but you'll probably be paying the price for years as little regression turn up because you didn't completely understand the original statd (and face it, who does?)

Yes, but _why_ is it a bad idea? All I hear is "this is a bad idea" and "you could do it some other way" but these are qualitative, not quantitative arguments. They are religious statements, not specific technical criticisms.

It is a bad idea because it doesn't have the legacy of testing and refinement. Almost as soon as we started using it bugs were found - or at least differences in behaviour to portmap (something about the privilege level required to register a binding I think).

Quoted 7 lines

Now I admit that no one put their hand up to add IPv6 support to portmap, arguably it could have been a worse idea to stay with portmap as it meant no IPv6. But changing was still a bad idea.

Had we (had the man power to) incrementally enhance portmap we would have had a much more reviewable process, and a bisectable result which would allow regression to be isolated more directly.

What we have now is an inherited body of code (with its own history of incremental improvement) that is shared with many other operating systems, which improves our ability to interoperate with them, and includes bug fixes that have been made to it over the years.

I think we would have had some bugs and regressions pursuing either path. There are well-understood ways to manage these risks, either way.

But this is a sidebar.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#29 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

In reply to Chuck Lever
Quoted 14 lines · Chuck Lever
Quoted 5 lines · NeilBrown

I agree that combative mailing lists are a problem, but even there, I believe most of the aggression is more perceived than real, and that a graceful, humble, polite attitude can have a positive-feedback effect
too.

Years ago I believed that, but I have seen much evidence to the contrary in this community. More often such an attitude is entirely ignored, or treated as an invitation for abuse, especially by people who have no interest in politeness. This kind of approach has no effect on the leaders in the Linux community, who set an example of extreme rudeness and belligerence.

Quoted 3 lines

I've made an effort to stop arguing small points, and to make observations and not argue. I still get e-mail full of "crap" this and "bullshit" that and "NACK!" with little explanation.

As you said above, you've been part of the community for years. It is not as if you haven't learned by now that a review might turn up issues that may give rise to a NACK, and that you need to be open to changing your code should this happen.
If you need more information about what needs to be changed, then you know to ask.

That hasn't been your approach in this case, though, and the responses you got were a direct consequence of that approach.
You tried reversing the burden of proof as to why we should change an established interface instead of supplying adequate evidence justifying that change.
When problems were pointed out to you (e.g. backward compatibility) your response was to deny they existed instead of proposing a change to your code.
Finally, you tried changing the thread into a discussion about mean rude people obstructing you and failing to give you adequate guidance.

The problems with the code remain, and you will need to change it in order to make it acceptable. The question I haven't seen you asking, and that you should have be asking from the very start is "what would be the minimal set of changes?".

I'm quite willing to discuss that with you.

Trond

A message that is not in the archive (sent off-list or not yet received): 1252936467.6866.54.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#31 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 4 lines

The problems with the code remain, and you will need to change it in order to make it acceptable. The question I haven't seen you asking, and that you should have be asking from the very start is "what would be the minimal set of changes?".

I too would like to see what are minimal change set would be... Just to see how much a db is really needed... If at all..

Now with that said... I've taken on the task of how to deal with pNFS exports and it might make sense that a db is need to deal with that complexity... So I'm not totally against the idea of introducing a db to nfs-utils I just think we need to do it for the right reason....

steved.

#32 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

In reply to Trond Myklebust
Quoted 15 lines · Trond Myklebust
Quoted 14 lines · Chuck Lever
Quoted 5 lines · NeilBrown

I agree that combative mailing lists are a problem, but even there, I believe most of the aggression is more perceived than real, and that a graceful, humble, polite attitude can have a positive-feedback effect
too.

Years ago I believed that, but I have seen much evidence to the contrary in this community. More often such an attitude is entirely ignored, or treated as an invitation for abuse, especially by people who have no interest in politeness. This kind of approach has no effect on the leaders in the Linux community, who set an example of extreme rudeness and belligerence.

Quoted 8 lines
Quoted 3 lines

I've made an effort to stop arguing small points, and to make observations and not argue. I still get e-mail full of "crap" this and "bullshit" that and "NACK!" with little explanation.

As you said above, you've been part of the community for years. It is not as if you haven't learned by now that a review might turn up issues that may give rise to a NACK, and that you need to be open to changing your code should this happen.

The tone of your vetoes is unnecessarily aggressive, and often the comments are entirely negative. A mention of the pieces of new statd that you liked, for example, would have been welcome, and even useful for this conversation.

Naturally, you are free to disagree with me as much as you like. But I wish you could be more constructive about it. This is not a contest... we're supposed to be working together to improve the code base.

Quoted 8 lines

If you need more information about what needs to be changed, then you know to ask.

That hasn't been your approach in this case, though, and the responses you got were a direct consequence of that approach.
You tried reversing the burden of proof as to why we should change an established interface instead of supplying adequate evidence justifying that change.

I think it's reasonable to ask for evidence on both sides.

Quoted 3 lines

When problems were pointed out to you (e.g. backward compatibility) your response was to deny they existed instead of proposing a change to your code.

The problem you described seemed to stem from a basic disagreement about how statd with IPv6 would be deployed -- an issue that neither of us (being upstream developers and not distributors) can resolve. Your objection still doesn't make sense to me. But see below.

Quoted 2 lines

Finally, you tried changing the thread into a discussion about mean rude people obstructing you and failing to give you adequate guidance.

I was responding to Neil's comments, not changing the subject.

In any event, since all of the NFS maintainers have now passed their judgement, it's clear that I will have to withdraw new statd, and proceed with a re-write that uses the existing on-disk format. I've never claimed that sqlite3 is a _requirement_ to solve these issues, but only that some changes would be necessary to the on-disk format, and that a database seemed an appropriate improvement.

No maintainers agree with that, so I will rework it.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

A message that is not in the archive (sent off-list or not yet received): 9eae93545189a6be6eebe0460b860fc7.squirrel-eq65iwfR9nKIECXXMXunQA@public.gmane.org

#34 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 3 lines · NeilBrown

I could summarise a lot of your points by observing that the community values people over process. I really think that is the right place to put value, because people are richer and more flexible than process.

What a refreshing point of view. I hope it catches on.

Quoted 2 lines

As for the use of sql-lite ... I must admit that I wouldn't choose it.

A number of other projects support multiple backends. Maybe that makes sense here.

-Ben

#35 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

In reply to Chuck Lever
Quoted 9 lines · Chuck Lever

IDNs are UTF16. /var therefore has to support UTF16 filenames; either byte in a double-byte character can be '/' or '\0'. That means the underlying fs implementation has to support UTF16 (FAT32 anyone?), and the system's locale has to be configured correctly. If we decide not to depend on the file system to support UTF16 filenames, then statd has to be intelligent enough to figure out how to deal with converting UTF16 hostnames before storing them as filenames. Then, we have to teach matchhostname() and friends how to deal with double-byte character strings...

Googling around.... Is this accurate?:

http://en.wikipedia.org/wiki/Internationalized_domain_name

That makes it sound like domain names are staying ascii, and they're just adding something on top to allow encoding unicode using ascii, which may optionally be used by applications.

--b.

#36 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 11 lines · J. Bruce Fields
Quoted 9 lines · Chuck Lever

IDNs are UTF16. /var therefore has to support UTF16 filenames; either byte in a double-byte character can be '/' or '\0'. That means the underlying fs implementation has to support UTF16 (FAT32 anyone?), and the system's locale has to be configured correctly. If we decide not to depend on the file system to support UTF16 filenames, then statd has to be intelligent enough to figure out how to deal with converting UTF16 hostnames before storing them as filenames. Then, we have to teach matchhostname() and friends how to deal with double-byte character strings...

Quoted 7 lines

Googling around.... Is this accurate?:

http://en.wikipedia.org/wiki/Internationalized_domain_name

That makes it sound like domain names are staying ascii, and they're just adding something on top to allow encoding unicode using ascii, which may optionally be used by applications.

There is a mechanism that provides an ASCII-ized version of domain names that may contain non-ASCII characters, expressly for applications that need to perform DNS queries but can't be easily converted to handle double-byte character strings. This can be adapted for statd, though I'm not sure if the converted ASCII version of such names specifically exclude '/'.

Internationalized domain names themselves are still expressed in UTF16, as far as I understand it.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#37 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 31 lines · Chuck Lever
Quoted 15 lines · J. Bruce Fields
Quoted 5 lines · Chuck Lever

IDNs are UTF16. /var therefore has to support UTF16 filenames; either
byte in a double-byte character can be '/' or '\0'. That means the underlying fs implementation has to support UTF16 (FAT32 anyone?), and

Quoted 8 lines

the system's locale has to be configured correctly. If we decide not to
depend on the file system to support UTF16 filenames, then statd has to
be intelligent enough to figure out how to deal with converting UTF16 hostnames before storing them as filenames. Then, we have to teach matchhostname() and friends how to deal with double-byte character strings...

Quoted 7 lines

Googling around.... Is this accurate?:

http://en.wikipedia.org/wiki/Internationalized_domain_name

That makes it sound like domain names are staying ascii, and they're just adding something on top to allow encoding unicode using ascii, which may optionally be used by applications.

There is a mechanism that provides an ASCII-ized version of domain names that may contain non-ASCII characters, expressly for applications that need to perform DNS queries but can't be easily converted to handle double-byte character strings. This can be adapted for statd, though I'm not sure if the converted ASCII version of such names specifically exclude '/'.

Quoted 4 lines

Internationalized domain names themselves are still expressed in UTF16, as far as I understand it.

From a quick skim of http://www.ietf.org/rfc/rfc3490.txt, it appears to

me that protocols (at the very least, any preexisting protocols) are all expected to use the ascii representation on the wire, and that the translation to unicode is meant by use for applications.

So in our case we'd continue to expect ascii domain names on the wire, and I believe that's also what we should store in any database. But if someone were to write a gui administrative interface to that data, for example, they might choose to use idna for display.

--b.

#38 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 9 lines · J. Bruce Fields

From a quick skim of http://www.ietf.org/rfc/rfc3490.txt, it appears to me that protocols (at the very least, any preexisting protocols) are all expected to use the ascii representation on the wire, and that the translation to unicode is meant by use for applications.

So in our case we'd continue to expect ascii domain names on the wire, and I believe that's also what we should store in any database. But if someone were to write a gui administrative interface to that data, for example, they might choose to use idna for display.

Right. There should be no need to translate anything. Just store as is and use as is...

Trond

#39 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

In reply to J. Bruce Fields
Quoted 40 lines · J. Bruce Fields
Quoted 31 lines · Chuck Lever
Quoted 15 lines · J. Bruce Fields
Quoted 5 lines · Chuck Lever

IDNs are UTF16. /var therefore has to support UTF16 filenames; either
byte in a double-byte character can be '/' or '\0'. That means the underlying fs implementation has to support UTF16 (FAT32 anyone?), and

Quoted 8 lines

the system's locale has to be configured correctly. If we decide not to
depend on the file system to support UTF16 filenames, then statd has to
be intelligent enough to figure out how to deal with converting UTF16 hostnames before storing them as filenames. Then, we have to teach matchhostname() and friends how to deal with double-byte character strings...

Quoted 7 lines

Googling around.... Is this accurate?:

http://en.wikipedia.org/wiki/Internationalized_domain_name

That makes it sound like domain names are staying ascii, and they're just adding something on top to allow encoding unicode using ascii, which may optionally be used by applications.

There is a mechanism that provides an ASCII-ized version of domain names that may contain non-ASCII characters, expressly for applications that need to perform DNS queries but can't be easily converted to handle double-byte character strings. This can be adapted for statd, though I'm not sure if the converted ASCII version of such names specifically exclude '/'.

Quoted 2 lines

Internationalized domain names themselves are still expressed in UTF16, as far as I understand it.

From a quick skim of http://www.ietf.org/rfc/rfc3490.txt, it appears to me that protocols (at the very least, any preexisting protocols) are all expected to use the ascii representation on the wire, and that the translation to unicode is meant by use for applications.

Quoted 4 lines

So in our case we'd continue to expect ascii domain names on the wire, and I believe that's also what we should store in any database. But if someone were to write a gui administrative interface to that data, for example, they might choose to use idna for display.

That's a reasonable and specific objection to my claim that our current host record storage format is inadequate to support IDNA. I've also confirmed that ToAscii with the UseSTD3ASCIIRules flag set is not supposed to generate a domain label string with a '/' in it. My remaining concern here is that we could possibly see hostnames that are too long to be stored in directory entries of some file systems, especially considering that the ASCII-fied Unicode names will be longer than typical ASCII names we normally encounter today.

What about multi-homed host support? The same mon_name can be used with more than one my_name, for multi-homed hosts. Using the current on-disk scheme, statd turns that SM_MON request into a no-op. So additional records for the same hostname can't be stored, or we have to resort to adding multiple lines in the same file. This is possible to do with just POSIX file system calls, but it does add complexity to manage several lines in each hostname file without increasing the risk of corruption if a file update (especially the deletion of one record in the middle) is interrupted.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#40 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 55 lines · Chuck Lever
Quoted 48 lines · J. Bruce Fields
Quoted 36 lines · Chuck Lever
Quoted 17 lines · J. Bruce Fields
Quoted 7 lines · Chuck Lever

IDNs are UTF16. /var therefore has to support UTF16 filenames; either
byte in a double-byte character can be '/' or '\0'. That means the underlying fs implementation has to support UTF16 (FAT32 anyone?), and
the system's locale has to be configured correctly. If we decide not to

Quoted 8 lines

depend on the file system to support UTF16 filenames, then statd has
to
be intelligent enough to figure out how to deal with converting UTF16
hostnames before storing them as filenames. Then, we have to teach matchhostname() and friends how to deal with double-byte character strings...

Quoted 7 lines

Googling around.... Is this accurate?:

http://en.wikipedia.org/wiki/Internationalized_domain_name

That makes it sound like domain names are staying ascii, and they're just adding something on top to allow encoding unicode using ascii, which may optionally be used by applications.

There is a mechanism that provides an ASCII-ized version of domain names
that may contain non-ASCII characters, expressly for applications that
need to perform DNS queries but can't be easily converted to handle double-byte character strings. This can be adapted for statd, though I'm
not sure if the converted ASCII version of such names specifically exclude '/'.

Quoted 3 lines

Internationalized domain names themselves are still expressed in UTF16,
as far as I understand it.

From a quick skim of http://www.ietf.org/rfc/rfc3490.txt, it appears to
me that protocols (at the very least, any preexisting protocols) are all
expected to use the ascii representation on the wire, and that the translation to unicode is meant by use for applications.

Quoted 5 lines

So in our case we'd continue to expect ascii domain names on the wire, and I believe that's also what we should store in any database. But if
someone were to write a gui administrative interface to that data, for example, they might choose to use idna for display.

Quoted 8 lines

That's a reasonable and specific objection to my claim that our current host record storage format is inadequate to support IDNA. I've also confirmed that ToAscii with the UseSTD3ASCIIRules flag set is not supposed to generate a domain label string with a '/' in it. My remaining concern here is that we could possibly see hostnames that are too long to be stored in directory entries of some file systems, especially considering that the ASCII-fied Unicode names will be longer than typical ASCII names we normally encounter today.

Googling around some more.... the normal limits for dns appear to be 63 bytes per component, and 255 for the whole string, and those limits are still in force on the output of that mapping. I suspect this isn't a huge deal.

Quoted 3 lines

What about multi-homed host support? The same mon_name can be used with more than one my_name, for multi-homed hosts. Using the current on-disk scheme, statd turns that SM_MON request into a no-op.

I don't know what we're supposed to do in that case. You want to store them all so you can send notifies to them all on reboot?

--b.

Quoted 14 lines

So additional
records for the same hostname can't be stored, or we have to resort to adding multiple lines in the same file. This is possible to do with just POSIX file system calls, but it does add complexity to manage several lines in each hostname file without increasing the risk of corruption if a file update (especially the deletion of one record in the middle) is interrupted.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#41 lore raw

Re: [PATCH 1/4] nfs-utils: introduce new statd implementation (1st part)

Quoted 65 lines · J. Bruce Fields
Quoted 55 lines · Chuck Lever
Quoted 48 lines · J. Bruce Fields
Quoted 36 lines · Chuck Lever
Quoted 17 lines · J. Bruce Fields
Quoted 7 lines · Chuck Lever

IDNs are UTF16. /var therefore has to support UTF16 filenames; either
byte in a double-byte character can be '/' or '\0'. That means the underlying fs implementation has to support UTF16 (FAT32 anyone?), and
the system's locale has to be configured correctly. If we decide not to

Quoted 8 lines

depend on the file system to support UTF16 filenames, then statd has
to
be intelligent enough to figure out how to deal with converting UTF16
hostnames before storing them as filenames. Then, we have to teach matchhostname() and friends how to deal with double-byte character strings...

Quoted 7 lines

Googling around.... Is this accurate?:

http://en.wikipedia.org/wiki/Internationalized_domain_name

That makes it sound like domain names are staying ascii, and they're just adding something on top to allow encoding unicode using ascii, which may optionally be used by applications.

There is a mechanism that provides an ASCII-ized version of domain names
that may contain non-ASCII characters, expressly for applications that
need to perform DNS queries but can't be easily converted to handle double-byte character strings. This can be adapted for statd, though I'm
not sure if the converted ASCII version of such names specifically exclude '/'.

Quoted 3 lines

Internationalized domain names themselves are still expressed in UTF16,
as far as I understand it.

From a quick skim of http://www.ietf.org/rfc/rfc3490.txt, it appears to
me that protocols (at the very least, any preexisting protocols) are all
expected to use the ascii representation on the wire, and that the translation to unicode is meant by use for applications.

Quoted 5 lines

So in our case we'd continue to expect ascii domain names on the wire, and I believe that's also what we should store in any database. But if
someone were to write a gui administrative interface to that data, for example, they might choose to use idna for display.

Quoted 8 lines

That's a reasonable and specific objection to my claim that our current host record storage format is inadequate to support IDNA. I've also confirmed that ToAscii with the UseSTD3ASCIIRules flag set is not supposed to generate a domain label string with a '/' in it. My remaining concern here is that we could possibly see hostnames that are too long to be stored in directory entries of some file systems, especially considering that the ASCII-fied Unicode names will be longer than typical ASCII names we normally encounter today.

Quoted 4 lines

Googling around some more.... the normal limits for dns appear to be 63 bytes per component, and 255 for the whole string, and those limits are still in force on the output of that mapping. I suspect this isn't a huge deal.

I bring this up because NI_MAXHOST, declared in /usr/include/netdb.h, is 1025, not 255. You are probably correct, practically speaking.

Quoted 6 lines
Quoted 3 lines

What about multi-homed host support? The same mon_name can be used with more than one my_name, for multi-homed hosts. Using the current on-disk scheme, statd turns that SM_MON request into a no-op.

I don't know what we're supposed to do in that case. You want to store them all so you can send notifies to them all on reboot?

Something like that. Basically I think we want to send SM_NOTIFY to the monitored host _from_ every registered my_name we have for that mon_name. The kernel will probably use a separate nsm_host for each one of these, so statd should probably keep track of each of the cookies as well.

Remembering which my_names have been used is important because the remote often uses the sender's name or IP address to identify which monitored host has rebooted.

We could throw up our hands and just keep track of all the my_names that were used during the last reboot, and notify each mon_name from all of those. That doesn't help with remembering the cookies, though, and makes sending all notifications at reboot take longer.

Quoted 8 lines
Quoted 8 lines

So additional
records for the same hostname can't be stored, or we have to resort to adding multiple lines in the same file. This is possible to do with just POSIX file system calls, but it does add complexity to manage several lines in each hostname file without increasing the risk of corruption if a file update (especially the deletion of one record in the middle) is interrupted.

Quoted 6 lines
Quoted 6 lines

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#42 lore raw

PATCH 2/4 nfs-utils: introduce new statd implementation (2nd part)

In reply to a message not in the archive

Finish introducing source files for new statd.

---

 utils/new-statd/smncall.c |  774 +++++++++++++++++++++++++++++++++++++++++
 utils/new-statd/svc.c     |  841 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 1615 insertions(+), 0 deletions(-)
 create mode 100644 utils/new-statd/smncall.c
 create mode 100644 utils/new-statd/svc.c
utils/new-statd/smncall.c new file +774 -0
new file mode 100644
index 0000000..e6aba47
@@ -0,0 +1,774 @@
+/*
+ * Copyright 2009 Oracle.  All rights reserved.
+ *
+ * This file is part of nfs-utils.
+ *
+ * nfs-utils is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * nfs-utils is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with nfs-utils.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+/*
+ * NSM for Linux.
+ */
+
+/*
+ * Send reboot notifications to remote peers on our notify list.
+ */
+
+#ifdef HAVE_CONFIG_H
+#include <config.h>
+#endif
+
+#include <unistd.h>
+#include <signal.h>
+#include <fcntl.h>
+#include <errno.h>
+
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <sys/wait.h>
+
+#include <netinet/in.h>
+#include <arpa/inet.h>
+
+#include "statd.h"
+#include "nfsrpc.h"
+
+static const char *statd_pgmtbl[] = {
+       "status",
+       NULL,
+};
+
+static void
+statd_delete_notified_host(const char *mon_name, const char *my_name)
+{
+	sqlite3_stmt *stmt;
+	sqlite3 *db;
+	char *sql;
+	bool_t rc;
+
+	db = statd_open_db(SQLITE_OPEN_READWRITE);
+	if (db == NULL)
+		return;
+
+	sql = sqlite3_mprintf("DELETE FROM " STATD_NOTIFY_TABLENAME
+				" WHERE mon_name='%q' and my_name='%q'",
+					mon_name, my_name);
+	if (sql == NULL) {
+		xlog(L_ERROR, "Failed to generate DELETE FROM statement", __func__);
+		goto out_close;
+	}
+
+	rc = statd_prepare_stmt(db, &stmt, sql);
+	sqlite3_free(sql);
+	if (!rc)
+		goto out_close;
+
+	switch (sqlite3_step(stmt)) {
+	case SQLITE_DONE:
+		if (sqlite3_changes(db) == 0)
+			xlog(L_ERROR, "Mon_name '%s' not found "
+					"in the notify list", mon_name);
+		else
+			xlog(D_GENERAL, "Mon_name '%s' removed from notify list",
+					mon_name);
+		break;
+	default:
+		xlog(L_ERROR, "Failed to delete row for mon_name '%s': %s",
+				mon_name, sqlite3_errmsg(db));
+	}
+
+	statd_finalize_stmt(stmt);
+
+out_close:
+	statd_close_db(db);
+}
+
+/*
+ * Returns 1 if the table already exists, or was created; otherwise
+ * zero is returned if an error occurred.
+ */
+static bool_t
+statd_create_notify_table(sqlite3 *db)
+{
+	sqlite3_stmt *stmt;
+	bool_t result;
+	int rc;
+
+	result = FALSE;
+
+	rc = sqlite3_prepare_v2(db, "CREATE TABLE " STATD_NOTIFY_TABLENAME
+					" (mon_name TEXT NOT NULL,"
+					" my_name TEXT NOT NULL,"
+					" protocol TEXT NOT NULL,"
+					" UNIQUE(mon_name, my_name));",
+						-1, &stmt, NULL);
+	switch (rc) {
+	case SQLITE_OK:
+		rc = sqlite3_step(stmt);
+		switch (rc) {
+		case SQLITE_DONE:
+			result = TRUE;
+			xlog(D_GENERAL, "Table '" STATD_NOTIFY_TABLENAME
+					"' created successfully");
+			break;
+		default:
+			xlog(L_ERROR, "Failed to create table '"
+					STATD_NOTIFY_TABLENAME "': %s",
+					sqlite3_errmsg(db));
+		}
+
+		statd_finalize_stmt(stmt);
+		break;
+	case SQLITE_ERROR:
+		result = TRUE;
+		xlog(D_GENERAL, "Table '" STATD_NOTIFY_TABLENAME
+				"' already exists");
+		break;
+	default:
+		xlog(L_ERROR, "Failed to compile SQL: %s",
+				sqlite3_errmsg(db));
+		xlog(L_ERROR, "SQL: %s");
+		break;
+	}
+
+	return result;
+}
+
+static bool_t
+statd_merge_notify_list(sqlite3 *db)
+{
+	char *err_msg;
+
+	if (sqlite3_exec(db,
+		"INSERT OR REPLACE INTO " STATD_NOTIFY_TABLENAME
+		" SELECT mon_name,my_name,protocol FROM " STATD_MONITOR_TABLENAME ";",
+					NULL, 0, &err_msg) != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to merge notify list: %s", err_msg);
+		sqlite3_free(err_msg);
+		return FALSE;
+	}
+	return TRUE;
+}
+
+static bool_t
+statd_empty_monitor_list(sqlite3 *db)
+{
+	char *err_msg;
+
+	err_msg = NULL;
+	if (sqlite3_exec(db, "DELETE FROM " STATD_MONITOR_TABLENAME ,
+					NULL, 0, &err_msg) != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to empty the monitor list: %s",
+					sqlite3_errmsg(db));
+		sqlite3_free(err_msg);
+		return FALSE;
+	}
+
+	xlog(D_GENERAL, "All monitor list entries were deleted");
+	return TRUE;
+}
+
+/*
+ * Returns 1 if monitor list was retired.  Returns zero if there
+ * were no hosts to notify.  Otherwise -1 is returned if some
+ * error occurred.
+ */
+static int
+statd_retire_monitor_list(void)
+{
+	sqlite3 *db;
+	int result;
+
+	result = 0;
+
+	db = statd_open_db(SQLITE_OPEN_READWRITE);
+	if (db == NULL)
+		return result;
+
+	if (!statd_begin_transaction(db))
+		goto out_close;
+
+	if (!statd_update_nsm_state(db))
+		goto out_rollback;
+
+	if (!statd_create_notify_table(db))
+		goto out_rollback;
+
+	if (!statd_merge_notify_list(db))
+		goto out_rollback;
+
+	if (!statd_empty_monitor_list(db))
+		goto out_rollback;
+
+	statd_end_transaction(db);
+	result = 1;
+
+	xlog(D_CALL, "Retired monitor list");
+
+out_close:
+	statd_close_db(db);
+	return result;
+
+out_rollback:
+	statd_rollback_transaction(db);
+	goto out_close;
+}
+
+/*
+ * Ensure IP addresses and hostnames are mapped to a working address.
+ * If @hostname is NULL, generate an appropriate ANYADDR address instead.
+ *
+ * You might think the bind address is loop invariant (ie the same for
+ * all notified peers).  Unfortunately the DNS lookup here will fail if
+ * sm-notify is started before networking is up on the local host.  So
+ * we have to make statd_verify_bindaddr() part of the retry loop.
+ */
+static struct addrinfo *
+statd_verify_bindaddr(const char *hostname, const uint16_t port,
+		const int family, const int transport)
+{
+	struct addrinfo gai_hint = {
+		.ai_flags	= AI_NUMERICSERV,
+		.ai_family	= family,
+		.ai_protocol	= transport,
+	};
+	struct addrinfo	*gai_results;
+	char buf[8];
+	int error;
+
+	buf[0] = '\0';
+	if (port != 0)
+		(void)snprintf(buf, sizeof(buf), "%u", port);
+	if (hostname == NULL)
+		gai_hint.ai_flags = AI_PASSIVE;
+	error = getaddrinfo(hostname, buf, &gai_hint, &gai_results);
+	switch (error) {
+	case 0:
+		return gai_results;
+	case EAI_NONAME:
+		if (hostname != NULL)
+			xlog(L_ERROR, "The supplied source address \"%s\" "
+				"is invalid", hostname);
+		else
+			/* Shouldn't ever happen, but just in case... */
+			xlog(L_ERROR, "EAI_NONAME for a NULL hostname?");
+		break;
+	case EAI_SYSTEM:
+		xlog(L_ERROR, "While determining bind address: %m");
+		break;
+	default:
+		xlog(L_ERROR, "While determining bind address: %s",
+			gai_strerror(error));
+	}
+
+	return NULL;
+}
+
+static int
+statd_socket(const struct addrinfo *gai_results)
+{
+	int sock;
+
+	sock = socket(gai_results->ai_family, gai_results->ai_socktype, 0);
+	if (sock < 0) {
+		xlog(L_ERROR, "Failed to create RPC socket: %m");
+		return -1;
+	}
+
+	fcntl(sock, F_SETFL, O_NONBLOCK);
+	if (gai_results->ai_family == AF_INET6) {
+		const int zero = 0;
+		/* We want replies from both IPv4 and IPv6 remotes, please */
+		setsockopt(sock, SOL_IPV6, IPV6_V6ONLY,
+				(char *)&zero, sizeof(zero));
+	}
+
+	return sock;
+}
+
+static int
+statd_bind(int sock, struct addrinfo *gai_results)
+{
+	struct sockaddr *sap = gai_results->ai_addr;
+	socklen_t salen = gai_results->ai_addrlen;
+
+	/* port was set by getaddrinfo(3) */
+	if (bind(sock, sap, salen) == -1) {
+		xlog(L_ERROR, "Failed to bind RPC socket: %m");
+		close(sock);
+		sock = -1;
+	}
+
+	return sock;
+}
+
+/*
+ * @bindaddr can be NULL; in this case, an appropriate ANYADDR is used.
+ * @srcport can be zero; in this case, an appropriate reserved port
+ * is used.
+ *
+ * Returns a bound socket file descriptor, or -1 if an error occurs.
+ */
+static int
+statd_create_socket(const char *bindaddr, const uint16_t srcport,
+			const int family, const int protocol)
+{
+	struct addrinfo *gai_results;
+	char *transport;
+	int sock;
+
+	switch (protocol) {
+	case IPPROTO_UDP:
+		transport = "datagram";
+		break;
+	case IPPROTO_TCP:
+		transport = "stream";
+		break;
+	default:
+		transport = "unknown";
+	}
+	xlog(D_CALL, "Creating %s socket with bindaddr '%s' and port %u",
+			transport, bindaddr, srcport);
+
+	gai_results = statd_verify_bindaddr(bindaddr, srcport, family, protocol);
+	if (gai_results == NULL)
+		return -1;
+
+	sock = statd_socket(gai_results);
+	if (sock < 0)
+		return -1;
+
+	sock = statd_bind(sock, gai_results);
+
+	freeaddrinfo(gai_results);
+	return sock;
+}
+
+/*
+ * @timeout is modified to contain the time remaining (i.e. time
+ * provided minus time elasped).
+ *
+ * Returns zero on success, or returns -1 on error.  errno is
+ * set to reflect the nature of the error.
+ */
+static int
+statd_connect_socket(const int sock, const struct sockaddr *sap,
+			const socklen_t salen, struct timeval *timeout)
+{
+	int save, result;
+	fd_set rset;
+
+	result = -1;
+
+	save = fcntl(sock, F_GETFL, 0);
+	if (save == -1) {
+		xlog(L_ERROR, "Failed to get socket flags: %m");
+		goto out;
+	}
+
+	if (fcntl(sock, F_SETFL, save | O_NONBLOCK) == -1) {
+		xlog(L_ERROR, "Failed to set socket flags: %m");
+		goto out;
+	}
+
+	if (connect(sock, sap, salen) == -1) {
+		switch (errno) {
+		case EINPROGRESS:
+			break;
+		case EINVAL:
+			xlog(L_ERROR, "Invalid bind address");
+			goto out_restore;
+		default:
+			xlog(L_ERROR, "Failed to connect socket: %m");
+			goto out_restore;
+		}
+	} else {
+		result = 0;
+		goto out_restore;
+	}
+
+	FD_ZERO(&rset);
+	FD_SET(sock, &rset);
+
+	result = select(sock + 1, NULL, &rset, NULL, timeout);
+	if (result <= 0) {
+		if (result == 0)
+			errno = ETIMEDOUT;
+		xlog(L_ERROR, "Failed to select: %m");
+		result = -1;
+		goto out_restore;
+	}
+
+	if (FD_ISSET(sock, &rset)) {
+		socklen_t len = (socklen_t)sizeof(result);
+
+		if (getsockopt(sock, SOL_SOCKET, SO_ERROR, &result, &len) == -1) {
+			xlog(L_ERROR, "Failed to get socket options: %m");
+			result = -1;
+			goto out_restore;
+		}
+
+		if (result != 0) {
+			xlog(L_ERROR, "Invalid socket options: %s",
+					strerror(result));
+			errno = result;
+			result = -1;
+		}
+	}
+
+out_restore:
+	(void)fcntl(sock, F_SETFL, save);
+
+out:
+	return result;
+}
+
+/*
+ * clnt_create_timed(3) could do all this for us, but we have to
+ * allow admins to set the socket's bind address and source port.
+ *
+ * @bindaddr is supposedly necessary because Linux's statd can
+ * scrape the source address of incoming SM_NOTIFY requests to
+ * figure out if it knows about this host.
+ */
+static CLIENT *
+statd_create_clnt(struct sockaddr *sap, const socklen_t salen,
+		const char *bindaddr, const unsigned short srcport,
+		const unsigned short protocol, struct timeval *timeout)
+{
+	CLIENT *clnt = NULL;
+	struct netbuf nbuf = {
+		.maxlen	= salen,
+		.len	= salen,
+		.buf	= sap,
+	};
+	rpcprot_t program;
+	uint16_t port;
+	int sock;
+
+	/*
+	 * Chicken and egg... rpcb_getaddr(3t) could give us the
+	 * address and port of the remote, but we would need to pass
+	 * it an nconf.  To get an nconf, we need the transport
+	 * protocol and the address family.  But we don't know until
+	 * we have the remote's address what address family to use.
+	 *
+	 * Lame.
+	 */
+	program = nfs_getrpcbyname(SM_PROG, statd_pgmtbl);
+	port = nfs_getport(nbuf.buf, nbuf.len, program, SM_VERS, protocol);
+	if (port == 0) {
+		xlog(L_WARNING, "Failed to get server's port: %s",
+				clnt_sperrno(rpc_createerr.cf_stat));
+		return NULL;
+	}
+	nfs_set_port(sap, port);
+
+	/*
+	 * ...and we can't create our transport socket until we know
+	 * which address family we need to use to contact the remote.
+	 */
+	sock = statd_create_socket(bindaddr, srcport,
+				sap->sa_family, protocol);
+	if (sock == -1)
+		return NULL;
+
+	if (protocol == IPPROTO_TCP)
+		if (statd_connect_socket(sock, nbuf.buf, nbuf.len,
+							timeout) == -1) {
+			close(sock);
+			return NULL;
+		}
+
+	clnt = clnt_tli_create(sock, NULL, &nbuf, program, SM_VERS,
+					RPCSMALLMSGSIZE, RPCSMALLMSGSIZE);
+	if (clnt == NULL) {
+		xlog(L_WARNING, "Failed to create RPC client: %s",
+				clnt_sperrno(rpc_createerr.cf_stat));
+		return NULL;
+	}
+
+	CLNT_CONTROL(clnt, CLSET_FD_CLOSE, NULL);
+	return clnt;
+}
+
+static unsigned int
+statd_sm_notify_call(struct sockaddr *sap, const socklen_t salen,
+		const char *bindaddr, const unsigned short srcport,
+		const unsigned short protocol,
+		const char *my_name, const int my_state)
+{
+	struct timeval timeout = { SMN_TIMEOUT, 0 };
+	static char hostbuf[NI_MAXHOST];
+	struct stat_chge new_status;
+	enum clnt_stat status;
+	CLIENT *clnt;
+
+	getnameinfo(sap, salen, hostbuf, sizeof(hostbuf), NULL, 0, NI_NUMERICHOST);
+	xlog(D_CALL, "Attempting to send notification to %s", hostbuf);
+
+	nfs_clear_rpc_createerr();
+	clnt = statd_create_clnt(sap, salen, bindaddr, srcport,
+						protocol, &timeout);
+	if (clnt == NULL)
+		return 0;
+
+	new_status.state = my_state;
+	new_status.mon_name = strdup(my_name);
+	status = clnt_call(clnt, SM_NOTIFY,
+				(xdrproc_t)xdr_stat_chge, (caddr_t)&new_status,
+				(xdrproc_t)xdr_void, NULL, timeout);
+	free(new_status.mon_name);
+	clnt_destroy(clnt);
+
+	if (status != RPC_SUCCESS) {
+		xlog(D_CALL, "Notification RPC failed: %s", clnt_sperrno(status));
+		return 0;
+	}
+
+	xlog(D_CALL, "Notification RPC succeeded");
+	return 1;
+}
+
+/*
+ * Attempt to send to each of this host's known IP addresses.
+ * This is a weak attempt to send from each interface and/or
+ * protocol family on the local host.
+ *
+ * We might want to beef this up by looking at what network
+ * interfaces exist on the system and trying to send from
+ * each of them.  Unfortunately in the world of DHCP and
+ * NetworkManager, we can't count on the local network
+ * configuration to be stable, especially at boot time when
+ * this function is typically invoked..
+ *
+ * If we get at least one through, consider it a success and
+ * return TRUE.
+ */
+static bool_t
+statd_notify_host(const char *hostname, const char *bindaddr,
+		const unsigned short srcport, const unsigned short protocol,
+		const char *my_name, const int my_state)
+{
+	struct addrinfo gai_hint = {
+#ifdef IPV6_SUPPORTED
+		.ai_family	= AF_UNSPEC,
+		.ai_flags	= AI_ADDRCONFIG,
+#else	/* !IPV6_SUPPORTED */
+		.ai_family	= AF_INET,
+#endif	/* !IPV6_SUPPORTED */
+	};
+	struct addrinfo	*gai_results, *ai;
+	char canon_name[NI_MAXHOST];
+	unsigned int count;
+
+	gai_hint.ai_protocol = protocol;
+	gai_results = statd_get_address_list(canon_name, &gai_hint);
+	if (gai_results == NULL)
+		return FALSE;
+
+	count = 0;
+	ai = gai_results;
+	while (ai != NULL) {
+		count += statd_sm_notify_call(gai_results->ai_addr,
+				gai_results->ai_addrlen, bindaddr, srcport,
+				protocol, my_name, my_state);
+		ai = ai->ai_next;
+	}
+	freeaddrinfo(gai_results);
+
+	if (count == 0) {
+		xlog(D_CALL, "Failed to notify Host %s", hostname);
+		return FALSE;
+	}
+
+	xlog(D_CALL, "Host %s notified successfully", hostname);
+	return TRUE;
+}
+
+/*
+ * Keep trying until we succeed or our timer expires
+ */
+static void
+statd_notification_child(const char *mon_name, const char *bindaddr,
+		const unsigned short srcport, const char *transport,
+		const char *my_name, const int my_state,
+		const time_t timeout)
+{
+	const struct sigaction statd_create_sigaction = {
+		.sa_handler	= SIG_IGN,
+	};
+	unsigned int sleep_time = SMN_TIMEOUT;
+	unsigned short protocol;
+
+	(void)setpriority(PRIO_PROCESS, 0, 19);
+
+	if (strcmp(transport, "tcp") == 0) {
+		/* Don't bother our parent if the other end
+		 * closes our TCP connection. */
+		(void)sigaction(SIGPIPE, &statd_create_sigaction, NULL);
+		protocol = IPPROTO_TCP;
+	} else if (strcmp(transport, "udp") == 0) {
+		protocol = IPPROTO_UDP;
+	} else {
+		xlog(L_ERROR, "Unknown transport protocol for notification");
+		return;
+	}
+
+	for (;;) {
+		if (statd_notify_host(mon_name, bindaddr, srcport,
+						protocol, my_name, my_state)) {
+			statd_delete_notified_host(mon_name, my_name);
+			break;
+		}
+
+		if (timeout != 0 && timeout < time(NULL)) {
+			xlog(L_WARNING, "Could not notify %s; giving up",
+				mon_name);
+			break;
+		}
+
+		(void)sleep(sleep_time);
+
+		sleep_time <<= 2;
+		if (sleep_time > 60)
+			sleep_time = 60;
+	}
+}
+
+/*
+ * Hosts that can't be contacted now shouldn't block notification
+ * of active hosts that may be farther down the notify list.  So
+ * fork off all notifications.
+ */
+static void
+statd_fork_and_notify(const char *mon_name, const char *bindaddr,
+		const unsigned short srcport, const char *transport,
+		const char *my_name, const int my_state,
+		const unsigned long max_retry_minutes)
+{
+	time_t timeout = 0;
+
+	if (max_retry_minutes != 0)
+		timeout = time(NULL) + max_retry_minutes * 60;
+
+	switch (fork()) {
+	case -1:
+		xlog(L_ERROR, "Failed to fork notification process: %m");
+		break;
+	case 0:
+		/* Child. */
+		statd_notification_child(mon_name, bindaddr, srcport,
+					transport, my_name, my_state, timeout);
+		_exit(EXIT_SUCCESS);
+	default:
+		/* Parent. */
+		usleep(100000);
+		break;
+	}
+}
+
+/*
+ * Returns TRUE if all listed notifications have been scheduled;
+ * otherwise FALSE is returned.
+ */
+static bool_t
+statd_iterate_notify_list(const char *bindaddr, const unsigned short srcport,
+		const unsigned long max_retry_minutes)
+{
+	int i, nrow, ncolumn, my_state;
+	char **resultp;
+	char *err_msg;
+	bool_t result;
+	sqlite3 *db;
+
+	result = FALSE;
+	my_state = statd_get_nsm_state();
+
+	db = statd_open_db(SQLITE_OPEN_READONLY);
+	if (db == NULL)
+		return result;
+
+	err_msg = NULL;
+	if (sqlite3_get_table(db, "SELECT mon_name,my_name,protocol FROM "
+			STATD_NOTIFY_TABLENAME ";",
+			&resultp, &nrow, &ncolumn, &err_msg) != SQLITE_OK) {
+		xlog(L_ERROR, "Failed to get '" STATD_NOTIFY_TABLENAME
+				"' table: %s", err_msg);
+		sqlite3_free(err_msg);
+		goto out_close;
+	}
+
+	if (ncolumn == 0) {
+		xlog(D_GENERAL, "No hosts to notify");
+		result = TRUE;
+		goto out_free_table;
+	}
+	if (ncolumn != 3) {
+		xlog(L_ERROR, "Incorrect column count (%d) in SELECT result",
+				ncolumn);
+		goto out_free_table;
+	}
+
+	xlog(D_CALL, "Notification NSM state: %d", my_state);
+
+	for (i = ncolumn; i < (nrow + 1) * ncolumn ; i += ncolumn)
+		statd_fork_and_notify(resultp[i],
+				bindaddr, srcport, resultp[i + 2],
+				resultp[i + 1], my_state, max_retry_minutes);
+	result = TRUE;
+
+out_free_table:
+	sqlite3_free_table(resultp);
+
+out_close:
+	statd_close_db(db);
+	return result;
+}
+
+/**
+ * statd_notify - send SM_NOTIFY requests
+ * @bindaddr: C string containing DNS name or presentation address
+ *	of socket's source address
+ * @srcport: source port to use when sending reboot notifications
+ * @max_retry_minutes: minutes after which to stop attempting to contact peer
+ *
+ * Copy the on-disk monitored host list to a backup directory,
+ * update the local NSM state number, and fork off notification
+ * processes.
+ *
+ * Returns TRUE if notificaitions were started successfully,
+ * or FALSE if an error occurred.
+ */
+bool_t
+statd_notify(const char *bindaddr, const unsigned short srcport,
+		const unsigned long max_retry_minutes)
+{
+	const struct sigaction statd_sigchld_action = {
+		.sa_handler	= SIG_DFL,
+		.sa_flags	= SA_NOCLDSTOP | SA_NOCLDWAIT,
+	};
+
+	(void)sigaction(SIGCHLD, &statd_sigchld_action, NULL);
+
+	switch (statd_retire_monitor_list()) {
+	case 0:
+		return TRUE;
+	case 1:
+		return statd_iterate_notify_list(bindaddr, srcport,
+							max_retry_minutes);
+	default:
+		return FALSE;
+	}
+}
utils/new-statd/svc.c new file +841 -0

841 lines not on this page · raw message

#43 lore raw

PATCH 3/4 nfs-utils: introduce new statd implementation (3rd part)

In reply to a message not in the archive

Introduce auxiliary files (Makefiles, etc.) and update actools to support utils/new-statd.

---

 .gitignore                  |    4 +
 Makefile.am                 |    4 -
 aclocal/libcap.m4           |   15 ++
 aclocal/libsqlite3.m4       |   33 ++++
 aclocal/libtirpc.m4         |    2 
 configure.ac                |   12 ++
 utils/Makefile.am           |    7 +
 utils/new-statd/Makefile.am |  102 ++++++++++++++
 utils/new-statd/main.c      |  319 +++++++++++++++++++++++++++++++++++++++++++
 utils/new-statd/sm-notify.c |  155 +++++++++++++++++++++
 utils/new-statd/sm_inter.x  |  107 ++++++++++++++
 utils/new-statd/start-statd |   12 ++
 utils/new-statd/statd.h     |  135 ++++++++++++++++++
 utils/statd/Makefile.am     |    9 +
 14 files changed, 911 insertions(+), 5 deletions(-)
 create mode 100644 aclocal/libcap.m4
 create mode 100644 aclocal/libsqlite3.m4
 create mode 100644 utils/new-statd/Makefile.am
 create mode 100644 utils/new-statd/main.c
 create mode 100644 utils/new-statd/sm-notify.c
 create mode 100644 utils/new-statd/sm_inter.x
 create mode 100644 utils/new-statd/start-statd
 create mode 100644 utils/new-statd/statd.h
.gitignore +4 -0
index 632609e..30e9408 100644
@@ -49,6 +49,8 @@ utils/rquotad/rquota.h
 utils/rquotad/rquota_xdr.c
 utils/showmount/showmount
 utils/statd/statd
+utils/new-statd/statd
+utils/new-statd/sm-notify
 tools/locktest/testlk
 tools/getiversion/getiversion
 support/export/mount.h
@@ -59,6 +61,8 @@ utils/statd/sm_inter.h
 utils/statd/sm_inter_clnt.c
 utils/statd/sm_inter_svc.c
 utils/statd/sm_inter_xdr.c
+utils/new-statd/sm_inter.h
+utils/new-statd/sm_inter_xdr.c
 # cscope database files
 cscope.*
 # generic editor backup et al
Makefile.am +0 -4
index b3a6e91..d45dbde 100644
@@ -54,10 +54,6 @@ install-data-hook:
 	touch $(DESTDIR)$(statedir)/xtab; chmod 644 $(DESTDIR)$(statedir)/xtab
 	touch $(DESTDIR)$(statedir)/etab; chmod 644 $(DESTDIR)$(statedir)/etab
 	touch $(DESTDIR)$(statedir)/rmtab; chmod 644 $(DESTDIR)$(statedir)/rmtab
-	mkdir -p $(DESTDIR)$(statedir)/sm $(DESTDIR)$(statedir)/sm.bak
-	touch $(DESTDIR)$(statedir)/state
-	chmod go-rwx $(DESTDIR)$(statedir)/sm $(DESTDIR)$(statedir)/sm.bak $(DESTDIR)$(statedir)/state
-	-chown $(statduser) $(DESTDIR)$(statedir)/sm $(DESTDIR)$(statedir)/sm.bak $(DESTDIR)$(statedir)/state
 
 uninstall-hook:
 	rm $(DESTDIR)$(statedir)/xtab
aclocal/libcap.m4 new file +15 -0
new file mode 100644
index 0000000..eabe507
@@ -0,0 +1,15 @@
+dnl Checks for libcap.so
+dnl
+AC_DEFUN([AC_LIBCAP], [
+
+  dnl look for prctl
+  AC_CHECK_FUNC([prctl], , )
+
+  dnl look for the library; do not add to LIBS if found
+  AC_CHECK_LIB([cap], [cap_get_proc], [LIBCAP=-lcap], ,)
+  AC_SUBST(LIBCAP)
+
+  AC_CHECK_HEADERS([sys/capability.h], ,
+                   [AC_MSG_ERROR([libcap headers not found.])])
+
+])dnl
aclocal/libsqlite3.m4 new file +33 -0
new file mode 100644
index 0000000..73d1e46
@@ -0,0 +1,33 @@
+dnl Checks for matching sqlite3 header and library, and
+dnl sufficient sqlite3 version.
+dnl
+AC_DEFUN([AC_SQLITE3_VERS], [
+  AC_CHECK_HEADERS([sqlite3.h], ,)
+
+  dnl look for the library; do not add to LIBS if found
+  AC_CHECK_LIB([sqlite3], [sqlite3_libversion_number], [LIBSQLITE=-lsqlite3], ,)
+  AC_SUBST(LIBSQLITE)
+
+  AC_MSG_CHECKING(for suitable sqlite3 version)
+
+  AC_CACHE_VAL([libsqlite3_cv_is_recent],
+   [
+    saved_LIBS="$LIBS"
+    LIBS=-lsqlite3
+    AC_TRY_RUN([
+	#include <stdio.h>
+	#include <sqlite3.h>
+	int main()
+	{
+		int vers = sqlite3_libversion_number();
+
+		return vers != SQLITE_VERSION_NUMBER ||
+			vers < 3003000;
+	}
+       ], [libsqlite3_cv_is_recent=yes], [libsqlite3_cv_is_recent=no],
+       [libsqlite3_cv_is_recent=unknown])
+    LIBS="$saved_LIBS"])
+
+  AC_MSG_RESULT($libsqlite3_cv_is_recent)
+  AM_CONDITIONAL(CONFIG_SQLITE3, [test "$libsqlite3_cv_is_recent" = "yes"])
+])dnl
aclocal/libtirpc.m4 +2 -0
index 9f0fde0..a20894c 100644
@@ -37,4 +37,6 @@ AC_DEFUN([AC_LIBTIRPC], [
 
   fi
 
+  AM_CONDITIONAL([CONFIG_TIRPC], [test "$enable_tirpc" != "no"])
+
 ])dnl
configure.ac +12 -0
index e0ca70e..c9ef819 100644
@@ -139,6 +139,9 @@ AC_ARG_ENABLE(ipv6,
 dnl Check for TI-RPC library and headers
 AC_LIBTIRPC
 
+dnl Check for -lcap
+AC_LIBCAP
+
 # Check whether user wants TCP wrappers support
 AC_TCP_WRAPPERS
 
@@ -256,6 +259,14 @@ fi
 dnl Check for IPv6 support
 AC_IPV6
 
+dnl Check for sqlite3
+AC_SQLITE3_VERS
+
+dnl Build new-statd if TI-RPC and a recent libsqlite3 is present
+if test "$enable_tirpc" = yes; then
+  AM_CONDITIONAL(CONFIG_NEWSTATD, [test "$libsqlite3_cv_is_recent" = "yes"])
+fi
+
 dnl *************************************************************
 dnl Check for headers
 dnl *************************************************************
@@ -389,6 +400,7 @@ AC_CONFIG_FILES([
 	utils/nfsd/Makefile
 	utils/nfsstat/Makefile
 	utils/showmount/Makefile
+	utils/new-statd/Makefile
 	utils/statd/Makefile])
 AC_OUTPUT
 
utils/Makefile.am +6 -1
index 8665183..3acae4c 100644
@@ -2,6 +2,12 @@
 
 OPTDIRS =
 
+if CONFIG_NEWSTATD
+OPTDIRS += new-statd
+else
+OPTDIRS += statd
+endif
+
 if CONFIG_NFSV4
 OPTDIRS += idmapd
 endif
@@ -20,7 +26,6 @@ SUBDIRS = \
 	nfsd \
 	nfsstat \
 	showmount \
-	statd \
 	$(OPTDIRS)
 
 MAINTAINERCLEANFILES = Makefile.in
utils/new-statd/Makefile.am new file +102 -0

102 lines not on this page · raw message

utils/new-statd/main.c new file +319 -0

319 lines not on this page · raw message

utils/new-statd/sm-notify.c new file +155 -0

155 lines not on this page · raw message

utils/new-statd/sm_inter.x new file +107 -0

107 lines not on this page · raw message

utils/new-statd/start-statd new file +12 -0
new file mode 100644
index 0000000..da0f65f
@@ -0,0 +1,12 @@
+#!/bin/sh -p
+#
+# Sample start-statd script
+#
+# mount.nfs execs this script when mounting a filesystem with locking
+# enabled, but when statd does not seem to be running
+#
+# It should run run statd with whatever flags are appropriate for
+# this site.
+#
+PATH=/sbin:/usr/sbin
+exec rpc.statd
utils/new-statd/statd.h new file +135 -0

135 lines not on this page · raw message

utils/statd/Makefile.am +9 -0
index 8a3ba4e..080bce1 100644
@@ -75,6 +75,15 @@ uninstall-hook:
 	    [ $$p = sm-notify ] || rm -f $(RPCPREFIX)$(KPREFIX)$$p$(EXEEXT) ;\
 	  done)
 
+install-data-hook:
+	if [ ! -d $(DESTDIR)$(statedir) ]; then mkdir -p $(DESTDIR)$(statedir); fi
+	mkdir -p $(DESTDIR)$(statedir)/sm $(DESTDIR)$(statedir)/sm.bak
+	touch $(DESTDIR)$(statedir)/state
+	chmod go-rwx $(DESTDIR)$(statedir)/sm $(DESTDIR)$(statedir)/sm.bak $(DESTDIR)$(statedir)/state
+	-chown $(statduser) $(DESTDIR)$(statedir)/sm $(DESTDIR)$(statedir)/sm.bak $(DESTDIR)$(statedir)/state
+
+uninstall-hook:
+	rm $(DESTDIR)$(statedir)/state
 
 # XXX This makes some assumptions about what automake does.
 # XXX But there is no install-man-hook or install-man-local.
#44 lore raw

PATCH 4/4 nfs-utils: introduce new statd implementation (4th part)

In reply to a message not in the archive

Introduce new man pages for new statd and sm-notify.

---

 utils/new-statd/sm-notify.man |  286 ++++++++++++++++++++++++++++++
 utils/new-statd/statd.man     |  388 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 674 insertions(+), 0 deletions(-)
 create mode 100644 utils/new-statd/sm-notify.man
 create mode 100644 utils/new-statd/statd.man
utils/new-statd/sm-notify.man new file +286 -0
new file mode 100644
index 0000000..b38acf8
@@ -0,0 +1,286 @@
+.\"@(#)sm-notify.8"
+.\"
+.\" Copyright 2009 Oracle.  All rights reserved.
+.\"
+.TH SM-NOTIFY 8 "1 July 2009
+.SH NAME
+sm-notify \- send reboot notification to NFS peers
+.SH SYNOPSIS
+.BI "/usr/sbin/sm-notify [-dfh] [-m " minutes "] [-n " name "] [-o " port "] [-P " path "]
+.SH DESCRIPTION
+Locks held on files are not part of persistent file system state.
+Lock state is thus lost when a host reboots.
+.PP
+NFS file locking must also detect when lock state is lost because a remote
+host has rebooted.
+After an NFS client reboots, an NFS server must release all locks held by
+applications that were running on that client.
+After a server reboots, a client must remind the
+server of locks held by applications running on that client.
+.PP
+For NFS version 2 and version 3, a network protocol known as the
+.I Network Status Monitor
+protocol (or NSM for short)
+is used to notify NFS peers of reboots.
+On Linux, two separate user-space components constitute the NSM service:
+.TP
+.B sm-notify
+A helper program that notifies NFS peers after the local system reboots
+.TP
+.B rpc.statd
+A daemon that listens for reboot notifications from other hosts, and
+manages the list of hosts to be notified when the local system reboots
+.PP
+When a system reboots, system initialization starts
+.BR rpc.statd ,
+which then typically invokes
+.B sm-notify
+automatically.
+The local NFS lock manager indicates to
+.B rpc.statd
+which remote hosts to monitor.
+.SS NSM operation
+The first file locking interaction between an NFS client and server causes
+the NFS lock managers on both peers to contact their local NSM service to
+store information about their respective remote peers.
+.PP
+On Linux,
+the lock manager contacts
+.BR rpc.statd ,
+which records on persistent storage information about each monitored NFS peer.
+This information describes how to contact a remote peer
+in case the local system reboots, and how to notify the local
+lockd when a monitored peer indicates to
+.B rpc.statd
+it has rebooted.
+.PP
+An NFS client sends a hostname in each file lock request to NFS servers
+that can be used to call the client back.
+This hostname is known as the client's
+.IR caller_name .
+An NFS server can use this hostname to send asynchronous GRANT
+calls to a client, and
+it can also use this hostname when notifying clients of reboots.
+An NFS server may use this hostname,
+or it may simply scrape the client's network address
+from the underlying transport and use that instead,
+when calling a client back.
+.PP
+The hostname or network address of the client is handed to
+.B rpc.statd
+during the first locking interaction between an NFS client and server.
+This is known as the monitored peer's
+.IR mon_name .
+In addition, the local lockd tells
+.B rpc.statd
+what it thinks its own hostname is.
+This hostname is known as
+.IR my_name .
+.PP
+NFS clients do not actually know what an NFS server's
+.I mon_name
+might be, so they usually record the hostname and/or
+network address used to mount the server.
+.SS Reboot notification
+After a reboot,
+.B sm-notify
+reads the list of monitored peers from persistent storage and
+sends an SM_NOTIFY request
+to the NSM service on each listed remote peer,
+using each
+.I mon_name
+as the destination, and the
+.I my_name
+string to identify which host has rebooted.
+.PP
+After receiving an SM_NOTIFY request from a remote peer,
+.B rpc.statd
+contacts its local lock manager,
+which initiates appropriate lock recovery.
+.B rpc.statd
+does not forward all reboot notifications it receives to the local lockd,
+however.
+It matches incoming SM_NOTIFY requests by their
+.I mon_name
+or network address to one or more peers on its own monitor list.
+If
+.B rpc.statd
+does not find a peer on its monitor list that matches
+an incoming SM_NOTIFY request, the request is ignored.
+.PP
+In addition, each peer has its own
+.IR "state number" ,
+which is a 32-bit integer that is bumped after each reboot.
+This state number is communicated between two peers so that they
+can distinguish between actual reboots and replayed notifications.
+.PP
+The value of state number is managed on Linux by
+.BR rpc.statd " and " sm-notify .
+The NSM state number is sent by NFS clients in each NLM LOCK request.
+The client does not discover the server's state number until the
+server sends an SM_NOTIFY request.
+.PP
+After reboot notifications have been
+sent to each peer on the list of monitored hosts,
+.B rpc.statd
+clears the monitor list, since, as noted above,
+lock state does not persist across reboots.
+Part of NFS lock recovery is rediscovering
+which peers need to be monitored again.
+.SH OPTIONS
+.TP
+.BR -d , " --debug
+Keeps
+.B sm-notify
+in the foreground while sending reboot notifications,
+and causes it to generate extra messages
+so that notification progress may be observed directly.
+.TP
+.BR -h , " --help
+Displays version and usage information on
+.IR stderr .
+.TP
+.BR -f , " --force
+Disables detection of system reboots, forcing notification to occur.
+If this option is not specified,
+.B sm-notify
+bumps the NSM state number, clears
+.BR rpc.statd 's
+monitor list, and
+sends reboot notifications only once after each system reboot;
+subsequent invocations of
+.B sm-notify
+become no-ops until the system is rebooted.
+.TP
+.BI "\-m, " "" "\-\-max-retry " retry-time
+Specifies the length of time, in minutes, to continue retrying
+notifications to unresponsive hosts.
+If this option is not specified,
+.B sm-notify
+tries to send notifications for 15 minutes.
+.IP
+Specifying a value of 0 causes
+.B sm-notify
+to continue sending notifications to unresponsive peers
+until it is manually killed.
+.TP
+.BI "\-n, " "" "\-\-name " ipaddr " | " hostname
+Specifies the source network address or hostname from which
+to send reboot notification requests.
+If this option is not specified,
+.B sm-notify
+uses an appropriate ANYADDR as the transport bind address,
+and sends the local lockd's
+.I caller_name
+as the
+.I mon_name
+argument to SM_NOTIFY.
+.IP
+This option can be useful in multi-homed configurations where
+the remote requires notification from a specific network address.
+.TP
+.BI "\-o, " "" "\-\-notify\-port " port
+Specifies the source port number for the network socket used to
+send reboot notifications.
+If this option is not specified, a randomly chosen ephemeral port is used.
+.IP
+This option can be used to traverse a firewall between client and server.
+.TP
+.BI "\-P, " "" \-\-state\-directory\-path " pathname
+Specifies a pathname to the parent directory
+on the local system where NSM state information is kept.
+If this option is not specified,
+.B sm-notify
+uses
+.I /var/lib/nfs/statd
+by default.
+.IP
+After starting,
+.B sm-notify
+attempts to set its effective UID and GID to the owner
+and group of this directory.
+.SH SECURITY
+The
+.B sm-notify
+program must be started as root to acquire privileges needed
+to access the state information database.
+It drops root privileges
+as soon as it starts up to reduce the risk of a privilege escalation attack.
+During normal operation,
+the effective user ID it chooses is the owner of the state directory.
+This allows it to continue to access files in that directory after it
+has dropped its root privileges.
+To control which user ID
+.B rpc.statd
+chooses, simply use
+.BR chown (1)
+to set the owner of
+the state directory.
+.SH NOTES
+Lock recovery after a reboot is critical to maintaining data integrity
+and preventing unnecessary application hangs.
+.PP
+To help
+.B rpc.statd
+match SM_NOTIFY requests to NLM requests, a number of best practices
+should be observed, including:
+.IP
+The UTS nodename of your systems should match the DNS names that NFS
+peers use to contact them
+.IP
+The nodenames should all be fully qualified domain names
+.IP
+The forward and reverse DNS mapping of the nodenames should be
+consistent
+.IP
+The hostname the client uses to contact the server should match the
+server's mon_name in SM_NOTIFY requests it sends
+.PP
+The use of network addresses for either argument should be avoided when
+interoperating with non-Linux NFS implementations.
+.PP
+Unmounting an NFS file system does not necessarily stop
+either the NFS client or server from monitoring each other.
+Both may continue monitoring each other for a time in case subsequent
+NFS traffic between the two results in fresh mounts and additional
+file locking.
+.PP
+On Linux, if the
+.B lockd
+kernel module is unloaded during normal operation,
+all remote NFS peers are unmonitored.
+This can happen on an NFS client, for example,
+if an automounter removes all NFS mount
+points due to inactivity.
+.SS IPv6 and TI-RPC support
+This version of
+.B sm-notify
+supports IPv6 networking.
+RPC over IPv6 requires TI-RPC, implemented on Linux via
+.BR libtirpc .
+.B libtirpc
+is required to be available on the local system for this version of
+.BR sm-notify .
+.PP
+.B sm-notify
+will choose an appropriate IPv4 or IPv6 transport
+based on the network address returned by DNS for each remote peer.
+It should be fully compatible with remote systems that do not support TI-RPC
+or IPv6.
+.SH FILES
+.TP 2.5i
+.I /var/lib/nfs/statd/statdb
+default NSM state information database
+.SH SEE ALSO
+.BR rpc.statd (8),
+.BR nfs (5),
+.BR uname (2),
+.BR hostname (7)
+.PP
+RFC 1094 - "NFS: Network File System Protocol Specification"
+.br
+RFC 1813 - "NFS Version 3 Protocol Specification"
+.br
+OpenGroup Protocols for Interworking: XNFS, Version 3W - Chapter 11
+.SH AUTHORS
+Chuck Lever <[email protected]>
utils/new-statd/statd.man new file +388 -0

388 lines not on this page · raw message

Acked-by
#45 lore raw
In reply to a message not in the archive
Quoted 15 lines · On Wed, 05 Aug 2009 10:45:30 -0400 Chuck Lever <[email protected]> wrote:

Apparently there is a 100KB per-email size limit imposed by vger.kernel.org. I've broken up the large statd patch, and postponed the nfs(8) man page patches to another day.

Yesterday it was decided that:

[PATCH 3/5] statd: change default state dir to /var/lib/nfs/statd

was overkill. That one has been dropped from this series.

The first two of the present patches introduce the .c files under utils/new-statd/ . The next patch adds actools infrastructure for building the new statd and sm-notify executables. The last patch adds two new man pages for statd and sm-notify.

Quoted 8 lines

The old statd remains in utils/statd. If sqlite3 and TI-RPC is present on the build system, and TI-RPC is enabled during configuration, the new statd is built and installed instead of the old one.

See the description in the first patch for more details on the features of the new implementation.

Nice work, Chuck.

Rewriting statd from scratch is a pretty bold step, but I think it's warranted.

I've been testing this new implementation over several iterations and the recent ones have worked well for me so far. I've also walked through the code and it all seems to make sense.

A message that is not in the archive (sent off-list or not yet received): 20090805131554.67eb15bc-9yPaYZwiELC+kQycOl6kW4xkIHaj4LzF@public.gmane.org

#47 lore raw
Quoted 29 lines · Jeff Layton
Quoted 15 lines · On Wed, 05 Aug 2009 10:45:30 -0400 Chuck Lever <[email protected]> wrote:

Apparently there is a 100KB per-email size limit imposed by vger.kernel.org. I've broken up the large statd patch, and postponed the nfs(8) man page patches to another day.

Yesterday it was decided that:

[PATCH 3/5] statd: change default state dir to /var/lib/nfs/statd

was overkill. That one has been dropped from this series.

The first two of the present patches introduce the .c files under utils/new-statd/ . The next patch adds actools infrastructure for building the new statd and sm-notify executables. The last patch adds two new man pages for statd and sm-notify.

Quoted 8 lines

The old statd remains in utils/statd. If sqlite3 and TI-RPC is present on the build system, and TI-RPC is enabled during configuration, the new statd is built and installed instead of the old one.

See the description in the first patch for more details on the features of the new implementation.

Nice work, Chuck.

Quoted 8 lines

Rewriting statd from scratch is a pretty bold step, but I think it's warranted.

I've been testing this new implementation over several iterations and the recent ones have worked well for me so far. I've also walked through the code and it all seems to make sense.

Independent testing was an important piece of this work. Thanks to Jeff and Sankar Bose at NetApp for their assistance so far.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com