Skip to contentSkip to conversation

Automatic NFSv4 state revocation on filesystem unmount

PATCH Superseded Chuck Lever 8 participants 30 messages Feb 24 – Mar 4

Subsystems (4)

v3 of 9 Superseded Latest is v9 →

lore
Share & tools
lore

Chuck Lever

author maintainer
#0 lore raw

v3 cover Automatic NFSv4 state revocation on filesystem unmount

From: Chuck Lever <[email protected]>

When an NFS server exports a filesystem and clients hold NFSv4 state (opens, locks, delegations), unmounting the underlying filesystem fails with EBUSY. The /proc/fs/nfsd/unlock_fs interface exists for administrators to manually revoke state before retrying the unmount, but this approach has significant operational drawbacks.

Manual intervention breaks automation workflows. Containerized NFS servers, orchestration systems, and unattended maintenance scripts cannot reliably unmount exported filesystems without implementing custom logic to detect the failure and invoke unlock_fs. System administrators managing many exports face tedious, error-prone procedures when decommissioning storage.

This series enables the NFS server to detect filesystem unmount events and automatically revoke associated state. The mechanism registers with a new SRCU notifier chain in VFS that fires during mount teardown, after processing stuck children but before fsnotify_vfsmount_delete(), while SB_ACTIVE is still set. When a filesystem is unmounted, all NFSv4 opens, locks, and delegations referencing it are revoked, async COPY operations are cancelled with NFS4ERR_ADMIN_REVOKED sent to clients, NLM locks are released, and cached file handles are closed.

With automatic revocation, unmount operations complete without administrator intervention once the brief state cleanup finishes. Clients receive immediate notification of state loss through standard NFSv4 error codes, allowing applications to handle the situation appropriately rather than encountering silent failures.

Based on v7.0-rc1

---

Changes since v2:
- Replace fs_pin with an SRCU umount notifier chain in VFS
- Merge the pending COPY cancellation patch
- Replace xa_cmpxchg() with xa_insert()
- Use cancel_work_sync() instead of flush_workqueue()
- Remove rcu_barrier()
- Correct misleading claims in kdoc comments and commit messages

Changes since v1:
- Explain why drop_client() is being renamed
- Finish implementing revocation on umount
- Rename pin_insert_group
- Clarified log output and code comments
- Hold nfsd_mutex while closing nfsd_files

Chuck Lever (3):
fs: add umount notifier chain for filesystem unmount notification
nfsd: revoke NFSv4 state when filesystem is unmounted
nfsd: close cached files on filesystem unmount

 fs/namespace.c        |  69 ++++++++++
 fs/nfsd/Makefile      |   2 +-
 fs/nfsd/filecache.c   |  45 +++++++
 fs/nfsd/filecache.h   |   1 +
 fs/nfsd/netns.h       |   5 +
 fs/nfsd/nfs4state.c   |  29 +++++
 fs/nfsd/nfsctl.c      |  10 +-
 fs/nfsd/sb_watch.c    | 283 ++++++++++++++++++++++++++++++++++++++++++
 fs/nfsd/state.h       |   7 ++
 include/linux/mount.h |   4 +
 10 files changed, 452 insertions(+), 3 deletions(-)
 create mode 100644 fs/nfsd/sb_watch.c
Signature · 1 line
2.53.0
author maintainer
#1 lore raw

PATCH v3 1/3 fs: add umount notifier chain for filesystem unmount notification

From: Chuck Lever <[email protected]>

Kernel subsystems occasionally need notification when a filesystem is unmounted. Until now, the only mechanism available is the fs_pin infrastructure, which has limited adoption (only BSD process accounting uses it) and VFS maintainers consider it deprecated.

Add an SRCU notifier chain that fires during mount teardown, following the pattern established by lease_notifier_chain in fs/locks.c. The notifier fires after processing stuck children but before fsnotify_vfsmount_delete(), at which point SB_ACTIVE is still set and the superblock remains fully accessible.

The SRCU notifier type is chosen because:
- Unmount is relatively infrequent, so the overhead of SRCU registration and unregistration is acceptable
- Callbacks run in process context and may sleep
- No cache bounces occur during chain traversal

NFSD requires this mechanism to revoke NFSv4 state (opens, locks, delegations) and release cached file handles when a filesystem is unmounted, avoiding EBUSY errors that occur when client state pins the mount.

---
 fs/namespace.c        | 69 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/mount.h |  4 +++
 2 files changed, 73 insertions(+)
fs/namespace.c +69 -0
index ebe19ded293a..269e007e9312 100644
@@ -34,6 +34,7 @@
 #include <linux/mnt_idmapping.h>
 #include <linux/pidfs.h>
 #include <linux/nstree.h>
+#include <linux/notifier.h>
 
 #include "pnode.h"
 #include "internal.h"
@@ -73,6 +74,70 @@ static u64 event;
 static DEFINE_XARRAY_FLAGS(mnt_id_xa, XA_FLAGS_ALLOC);
 static DEFINE_IDA(mnt_group_ida);
 
+/*
+ * Kernel subsystems can register to be notified when a filesystem is
+ * unmounted. This is used by (e.g.) nfsd to revoke state associated
+ * with files on the filesystem being unmounted.
+ */
+static struct srcu_notifier_head umount_notifier_chain;
+
+/**
+ * umount_register_notifier - register for unmount notifications
+ * @nb: notifier_block to register
+ *
+ * Registers a notifier to be called when any filesystem is
+ * unmounted. The callback is invoked after stuck children are
+ * processed but before fsnotify_vfsmount_delete(), while SB_ACTIVE
+ * is still set and the superblock remains fully accessible.
+ *
+ * Callback signature:
+ *   int (*callback)(struct notifier_block *nb,
+ *                   unsigned long val, void *data)
+ *
+ *   @val:  always 0 (reserved for future extension)
+ *   @data: struct super_block * for the unmounting filesystem
+ *
+ * Callbacks run in process context and may sleep. Return
+ * NOTIFY_DONE from the callback; return values are ignored and
+ * cannot prevent unmount. Callbacks must handle their own error
+ * recovery internally.
+ *
+ * The notification fires once per mount instance. Bind mounts of
+ * the same filesystem trigger multiple callbacks with the same
+ * super_block pointer; callbacks must handle duplicate
+ * notifications idempotently.
+ *
+ * The super_block pointer is valid only for the duration of the
+ * callback. Callbacks must not retain this pointer for
+ * asynchronous use; to access the filesystem after the callback
+ * returns, acquire a separate reference (e.g., via an open file)
+ * during callback execution.
+ *
+ * Returns: 0 on success, negative error code on failure.
+ */
+int umount_register_notifier(struct notifier_block *nb)
+{
+	return srcu_notifier_chain_register(&umount_notifier_chain, nb);
+}
+EXPORT_SYMBOL_GPL(umount_register_notifier);
+
+/**
+ * umount_unregister_notifier - unregister an unmount notifier
+ * @nb: notifier_block to unregister
+ *
+ * Unregisters a previously registered notifier. This function may
+ * block due to SRCU synchronization.
+ *
+ * Must not be called from within a notifier callback; doing so
+ * causes deadlock. Must be called before module unload if the
+ * notifier_block resides in module memory.
+ */
+void umount_unregister_notifier(struct notifier_block *nb)
+{
+	srcu_notifier_chain_unregister(&umount_notifier_chain, nb);
+}
+EXPORT_SYMBOL_GPL(umount_unregister_notifier);
+
 /* Don't allow confusion with old 32bit mount ID */
 #define MNT_UNIQUE_ID_OFFSET (1ULL << 31)
 static u64 mnt_id_ctr = MNT_UNIQUE_ID_OFFSET;
@@ -1307,6 +1372,8 @@ static void cleanup_mnt(struct mount *mnt)
 		hlist_del(&m->mnt_umount);
 		mntput(&m->mnt);
 	}
+	/* Notify registrants before superblock deactivation */
+	srcu_notifier_call_chain(&umount_notifier_chain, 0, mnt->mnt.mnt_sb);
 	fsnotify_vfsmount_delete(&mnt->mnt);
 	dput(mnt->mnt.mnt_root);
 	deactivate_super(mnt->mnt.mnt_sb);
@@ -6189,6 +6256,8 @@ void __init mnt_init(void)
 {
 	int err;
 
+	srcu_init_notifier_head(&umount_notifier_chain);
+
 	mnt_cache = kmem_cache_create("mnt_cache", sizeof(struct mount),
 			0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL);
 
include/linux/mount.h +4 -0
index acfe7ef86a1b..9a46ab40dffd 100644
@@ -21,6 +21,7 @@ struct file_system_type;
 struct fs_context;
 struct file;
 struct path;
+struct notifier_block;
 
 enum mount_flags {
 	MNT_NOSUID	= 0x01,
@@ -109,4 +110,7 @@ extern void kern_unmount_array(struct vfsmount *mnt[], unsigned int num);
 
 extern int cifs_root_data(char **dev, char **opts);
 
+int umount_register_notifier(struct notifier_block *nb);
+void umount_unregister_notifier(struct notifier_block *nb);
+
 #endif /* _LINUX_MOUNT_H */
Signature · 1 line
2.53.0
#2 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 7 lines · Chuck Lever

From: Chuck Lever <[email protected]>

Kernel subsystems occasionally need notification when a filesystem is unmounted. Until now, the only mechanism available is the fs_pin infrastructure, which has limited adoption (only BSD process accounting uses it) and VFS maintainers consider it deprecated.

Quoted 5 lines

Add an SRCU notifier chain that fires during mount teardown, following the pattern established by lease_notifier_chain in fs/locks.c. The notifier fires after processing stuck children but before fsnotify_vfsmount_delete(), at which point SB_ACTIVE is still set and the superblock remains fully accessible.

What I don't understand is why you need this per-mount especially because you say above "when a filesystem is mounted. Could you explain this in some more details, please?

Also this should take namespaces into account somehow, right? As Al correctly observed anything that does CLONE_NEWNS and inherits your mountable will generate notifications. Like, if systemd spawns services, if a container runtime start, if someone uses unshare you'll get absolutely flooded with events. I'm pretty sure that is not what you want and that is defo not what the VFS should do...

Another thing: These ad-hoc notifiers are horrific. So I'm pitching another idea and I hope that Jan and Amir can tell me that this is doable...

Can we extend fsnotify so that it's possible for a filesystem to register "internal watches" on relevant objects such as mounts and superblocks and get notified and execute blocking stuff if needed.

Then we don't have to add another set of custom notification mechanisms but have it available in a single subsystem and uniformely available.

Quoted 142 lines

The SRCU notifier type is chosen because:
- Unmount is relatively infrequent, so the overhead of SRCU registration and unregistration is acceptable
- Callbacks run in process context and may sleep
- No cache bounces occur during chain traversal

NFSD requires this mechanism to revoke NFSv4 state (opens, locks, delegations) and release cached file handles when a filesystem is unmounted, avoiding EBUSY errors that occur when client state pins the mount.

---
 fs/namespace.c        | 69 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/mount.h |  4 +++
 2 files changed, 73 insertions(+)
fs/namespace.c +69 -0
index ebe19ded293a..269e007e9312 100644
@@ -34,6 +34,7 @@
 #include <linux/mnt_idmapping.h>
 #include <linux/pidfs.h>
 #include <linux/nstree.h>
+#include <linux/notifier.h>
 
 #include "pnode.h"
 #include "internal.h"
@@ -73,6 +74,70 @@ static u64 event;
 static DEFINE_XARRAY_FLAGS(mnt_id_xa, XA_FLAGS_ALLOC);
 static DEFINE_IDA(mnt_group_ida);
 
+/*
+ * Kernel subsystems can register to be notified when a filesystem is
+ * unmounted. This is used by (e.g.) nfsd to revoke state associated
+ * with files on the filesystem being unmounted.
+ */
+static struct srcu_notifier_head umount_notifier_chain;
+
+/**
+ * umount_register_notifier - register for unmount notifications
+ * @nb: notifier_block to register
+ *
+ * Registers a notifier to be called when any filesystem is
+ * unmounted. The callback is invoked after stuck children are
+ * processed but before fsnotify_vfsmount_delete(), while SB_ACTIVE
+ * is still set and the superblock remains fully accessible.
+ *
+ * Callback signature:
+ *   int (*callback)(struct notifier_block *nb,
+ *                   unsigned long val, void *data)
+ *
+ *   @val:  always 0 (reserved for future extension)
+ *   @data: struct super_block * for the unmounting filesystem
+ *
+ * Callbacks run in process context and may sleep. Return
+ * NOTIFY_DONE from the callback; return values are ignored and
+ * cannot prevent unmount. Callbacks must handle their own error
+ * recovery internally.
+ *
+ * The notification fires once per mount instance. Bind mounts of
+ * the same filesystem trigger multiple callbacks with the same
+ * super_block pointer; callbacks must handle duplicate
+ * notifications idempotently.
+ *
+ * The super_block pointer is valid only for the duration of the
+ * callback. Callbacks must not retain this pointer for
+ * asynchronous use; to access the filesystem after the callback
+ * returns, acquire a separate reference (e.g., via an open file)
+ * during callback execution.
+ *
+ * Returns: 0 on success, negative error code on failure.
+ */
+int umount_register_notifier(struct notifier_block *nb)
+{
+	return srcu_notifier_chain_register(&umount_notifier_chain, nb);
+}
+EXPORT_SYMBOL_GPL(umount_register_notifier);
+
+/**
+ * umount_unregister_notifier - unregister an unmount notifier
+ * @nb: notifier_block to unregister
+ *
+ * Unregisters a previously registered notifier. This function may
+ * block due to SRCU synchronization.
+ *
+ * Must not be called from within a notifier callback; doing so
+ * causes deadlock. Must be called before module unload if the
+ * notifier_block resides in module memory.
+ */
+void umount_unregister_notifier(struct notifier_block *nb)
+{
+	srcu_notifier_chain_unregister(&umount_notifier_chain, nb);
+}
+EXPORT_SYMBOL_GPL(umount_unregister_notifier);
+
 /* Don't allow confusion with old 32bit mount ID */
 #define MNT_UNIQUE_ID_OFFSET (1ULL << 31)
 static u64 mnt_id_ctr = MNT_UNIQUE_ID_OFFSET;
@@ -1307,6 +1372,8 @@ static void cleanup_mnt(struct mount *mnt)
 		hlist_del(&m->mnt_umount);
 		mntput(&m->mnt);
 	}
+	/* Notify registrants before superblock deactivation */
+	srcu_notifier_call_chain(&umount_notifier_chain, 0, mnt->mnt.mnt_sb);
 	fsnotify_vfsmount_delete(&mnt->mnt);
 	dput(mnt->mnt.mnt_root);
 	deactivate_super(mnt->mnt.mnt_sb);
@@ -6189,6 +6256,8 @@ void __init mnt_init(void)
 {
 	int err;
 
+	srcu_init_notifier_head(&umount_notifier_chain);
+
 	mnt_cache = kmem_cache_create("mnt_cache", sizeof(struct mount),
 			0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT, NULL);
 
include/linux/mount.h +4 -0
index acfe7ef86a1b..9a46ab40dffd 100644
@@ -21,6 +21,7 @@ struct file_system_type;
 struct fs_context;
 struct file;
 struct path;
+struct notifier_block;
 
 enum mount_flags {
 	MNT_NOSUID	= 0x01,
@@ -109,4 +110,7 @@ extern void kern_unmount_array(struct vfsmount *mnt[], unsigned int num);
 
 extern int cifs_root_data(char **dev, char **opts);
 
+int umount_register_notifier(struct notifier_block *nb);
+void umount_unregister_notifier(struct notifier_block *nb);
+
 #endif /* _LINUX_MOUNT_H */
Signature · 1 line
2.53.0
#3 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 9 lines · Christian Brauner
Quoted 7 lines · Chuck Lever

From: Chuck Lever <[email protected]>

Kernel subsystems occasionally need notification when a filesystem is unmounted. Until now, the only mechanism available is the fs_pin infrastructure, which has limited adoption (only BSD process accounting uses it) and VFS maintainers consider it deprecated.

Quoted 5 lines
Quoted 5 lines

Add an SRCU notifier chain that fires during mount teardown, following the pattern established by lease_notifier_chain in fs/locks.c. The notifier fires after processing stuck children but before fsnotify_vfsmount_delete(), at which point SB_ACTIVE is still set and the superblock remains fully accessible.

Did you see commit 74bd284537b34 ("fsnotify: Shutdown fsnotify before destroying sb's dcache")?

Does it make the fsnotify_sb_delete() hook an appropriate place for this cleanup?

We could send an FS_UNMOUNT event on sb, the same way as we send it on inode in fsnotify_unmount_inodes().

Quoted 5 lines

What I don't understand is why you need this per-mount especially because you say above "when a filesystem is mounted. Could you explain this in some more details, please?

The confusing thing is that FS_UNMOUNT/IN_UNMOUNT are sent for inotify when the sb is destroyed, not when the mount is unmounted.

If we wanted we could also send FS_UNMOUNT in fsnotify_vfsmount_delete(), but that would be too confusing.

I think the only reason that we did not add fanotify support for FAN_UNMOUNT is this name confusion, but there could be other reasons which I don't remember.

Quoted 7 lines

Also this should take namespaces into account somehow, right? As Al correctly observed anything that does CLONE_NEWNS and inherits your mountable will generate notifications. Like, if systemd spawns services, if a container runtime start, if someone uses unshare you'll get absolutely flooded with events. I'm pretty sure that is not what you want and that is defo not what the VFS should do...

Quoted 8 lines

Another thing: These ad-hoc notifiers are horrific. So I'm pitching another idea and I hope that Jan and Amir can tell me that this is doable...

Can we extend fsnotify so that it's possible for a filesystem to register "internal watches" on relevant objects such as mounts and superblocks and get notified and execute blocking stuff if needed.

You mean like nfsd_file_fsnotify_group? ;)

Quoted 3 lines

Then we don't have to add another set of custom notification mechanisms but have it available in a single subsystem and uniformely available.

I don't see a problem with nfsd registering for FS_UNMOUNT event on sb (once we add it).

As a matter of fact, I think that nfsd can already add an inode mark on the export root path for FS_UNMOUNT event.

Thanks,
Amir.

author maintainer
#4 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 34 lines · Amir Goldstein
Quoted 9 lines · Christian Brauner
Quoted 7 lines · Chuck Lever

From: Chuck Lever <[email protected]>

Kernel subsystems occasionally need notification when a filesystem is unmounted. Until now, the only mechanism available is the fs_pin infrastructure, which has limited adoption (only BSD process accounting uses it) and VFS maintainers consider it deprecated.

Quoted 5 lines
Quoted 5 lines

Add an SRCU notifier chain that fires during mount teardown, following the pattern established by lease_notifier_chain in fs/locks.c. The notifier fires after processing stuck children but before fsnotify_vfsmount_delete(), at which point SB_ACTIVE is still set and the superblock remains fully accessible.

Did you see commit 74bd284537b34 ("fsnotify: Shutdown fsnotify before destroying sb's dcache")?

Does it make the fsnotify_sb_delete() hook an appropriate place for this cleanup?

We could send an FS_UNMOUNT event on sb, the same way as we send it on inode in fsnotify_unmount_inodes().

Quoted 5 lines

What I don't understand is why you need this per-mount especially because you say above "when a filesystem is mounted. Could you explain this in some more details, please?

The confusing thing is that FS_UNMOUNT/IN_UNMOUNT are sent for inotify when the sb is destroyed, not when the mount is unmounted.

Quoted 13 lines

If we wanted we could also send FS_UNMOUNT in fsnotify_vfsmount_delete(), but that would be too confusing.

I think the only reason that we did not add fanotify support for FAN_UNMOUNT is this name confusion, but there could be other reasons which I don't remember.

Quoted 6 lines

Also this should take namespaces into account somehow, right? As Al correctly observed anything that does CLONE_NEWNS and inherits your mountable will generate notifications. Like, if systemd spawns services, if a container runtime start, if someone uses unshare you'll get absolutely flooded with events. I'm pretty sure that is not what you want and that is defo not what the VFS should do...

I agree with Al's earlier comment and have added some protection there for the next revision of the series.

Quoted 15 lines
Quoted 8 lines

Another thing: These ad-hoc notifiers are horrific. So I'm pitching another idea and I hope that Jan and Amir can tell me that this is doable...

Can we extend fsnotify so that it's possible for a filesystem to register "internal watches" on relevant objects such as mounts and superblocks and get notified and execute blocking stuff if needed.

You mean like nfsd_file_fsnotify_group? ;)

Quoted 3 lines

Then we don't have to add another set of custom notification mechanisms but have it available in a single subsystem and uniformely available.

Quoted 5 lines

I don't see a problem with nfsd registering for FS_UNMOUNT event on sb (once we add it).

As a matter of fact, I think that nfsd can already add an inode mark on the export root path for FS_UNMOUNT event.

There isn't much required here aside from getting a synchronous notice that the final file system unmount is going on. I'm happy to try whatever mechanism VFS maintainers are most comfortable with.

Signature · 1 line
Chuck Lever
maintainer
#5 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 20 lines · Chuck Lever
Quoted 19 lines · Amir Goldstein
Quoted 8 lines · Christian Brauner

Another thing: These ad-hoc notifiers are horrific. So I'm pitching another idea and I hope that Jan and Amir can tell me that this is doable...

Can we extend fsnotify so that it's possible for a filesystem to register "internal watches" on relevant objects such as mounts and superblocks and get notified and execute blocking stuff if needed.

You mean like nfsd_file_fsnotify_group? ;)

Quoted 3 lines

Then we don't have to add another set of custom notification mechanisms but have it available in a single subsystem and uniformely available.

I don't see a problem with nfsd registering for FS_UNMOUNT event on sb (once we add it).

Quoted 6 lines
Quoted 2 lines

As a matter of fact, I think that nfsd can already add an inode mark on the export root path for FS_UNMOUNT event.

There isn't much required here aside from getting a synchronous notice that the final file system unmount is going on. I'm happy to try whatever mechanism VFS maintainers are most comfortable with.

Yeah, then as Amir writes placing a mark with FS_UNMOUNT event on the export root path and handling the event in nfsd_file_fsnotify_handle_event() should do what you need?

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
author maintainer
#6 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 24 lines · Jan Kara
Quoted 23 lines · Chuck Lever
Quoted 21 lines · Amir Goldstein
Quoted 8 lines · Christian Brauner

Another thing: These ad-hoc notifiers are horrific. So I'm pitching another idea and I hope that Jan and Amir can tell me that this is doable...

Can we extend fsnotify so that it's possible for a filesystem to register "internal watches" on relevant objects such as mounts and superblocks and get notified and execute blocking stuff if needed.

You mean like nfsd_file_fsnotify_group? ;)

Quoted 3 lines

Then we don't have to add another set of custom notification mechanisms but have it available in a single subsystem and uniformely available.

I don't see a problem with nfsd registering for FS_UNMOUNT event on sb (once we add it).

As a matter of fact, I think that nfsd can already add an inode mark on the export root path for FS_UNMOUNT event.

Quoted 7 lines
Quoted 3 lines

There isn't much required here aside from getting a synchronous notice that the final file system unmount is going on. I'm happy to try whatever mechanism VFS maintainers are most comfortable with.

Yeah, then as Amir writes placing a mark with FS_UNMOUNT event on the export root path and handling the event in nfsd_file_fsnotify_handle_event() should do what you need?

Turns out FS_UNMOUNT doesn't do what I need.

1/3 here has a fatal flaw: the SRCU notifier does not fire until all files on the mount are closed. The problem is that NFSD holds files open when there is outstanding NFSv4 state. So the SRCU notifier will never fire, on umount, to release that state.

FS_UNMOUNT notifiers have the same issue.

They fire from fsnotify_sb_delete() inside generic_shutdown_super(), which runs inside deactivate_locked_super(), which runs when s_active drops to 0. That requires all mounts to be freed, which requires all NFSD files to be closed: the same problem.

For any notification approach to actually do what is needed, it needs to fire during do_umount(), before propagate_mount_busy(). Something like:

do_umount(mnt):
<- NEW: notify subsystems, allow them to release file refs
retval = propagate_mount_busy(mnt, 2) // now passes
umount_tree(mnt, ...)

This is what Christian's "internal watches... execute blocking stuff" would need to enable. The existing fsnotify plumbing (groups, marks, event dispatch) provides the infrastructure, but a new notification hook in do_umount() is required — neither fsnotify_vfsmount_delete() nor fsnotify_sb_delete() fires early enough.

But a hook in do_umount() fires for every mount namespace teardown, not just admin-initiated unmounts. NFSD's callback would need to filter (e.g., only act when it's the last mount of a superblock that NFSD is exporting).

This is why I originally went with fs_pin. Not saying the series should go back to that, but this is the basic requirement: NFSD needs notification of a umount request while files are still open on that mount, so that it can revoke the NFSv4 state and close those files.

Signature · 1 line
Chuck Lever
#7 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 67 lines · Chuck Lever
Quoted 24 lines · Jan Kara
Quoted 23 lines · Chuck Lever
Quoted 21 lines · Amir Goldstein
Quoted 8 lines · Christian Brauner

Another thing: These ad-hoc notifiers are horrific. So I'm pitching another idea and I hope that Jan and Amir can tell me that this is doable...

Can we extend fsnotify so that it's possible for a filesystem to register "internal watches" on relevant objects such as mounts and superblocks and get notified and execute blocking stuff if needed.

You mean like nfsd_file_fsnotify_group? ;)

Quoted 3 lines

Then we don't have to add another set of custom notification mechanisms but have it available in a single subsystem and uniformely available.

I don't see a problem with nfsd registering for FS_UNMOUNT event on sb (once we add it).

As a matter of fact, I think that nfsd can already add an inode mark on the export root path for FS_UNMOUNT event.

Quoted 7 lines
Quoted 3 lines

There isn't much required here aside from getting a synchronous notice that the final file system unmount is going on. I'm happy to try whatever mechanism VFS maintainers are most comfortable with.

Yeah, then as Amir writes placing a mark with FS_UNMOUNT event on the export root path and handling the event in nfsd_file_fsnotify_handle_event() should do what you need?

Turns out FS_UNMOUNT doesn't do what I need.

1/3 here has a fatal flaw: the SRCU notifier does not fire until all files on the mount are closed. The problem is that NFSD holds files open when there is outstanding NFSv4 state. So the SRCU notifier will never fire, on umount, to release that state.

FS_UNMOUNT notifiers have the same issue.

They fire from fsnotify_sb_delete() inside generic_shutdown_super(), which runs inside deactivate_locked_super(), which runs when s_active drops to 0. That requires all mounts to be freed, which requires all NFSD files to be closed: the same problem.

For any notification approach to actually do what is needed, it needs to fire during do_umount(), before propagate_mount_busy(). Something like:

do_umount(mnt):
<- NEW: notify subsystems, allow them to release file refs
retval = propagate_mount_busy(mnt, 2) // now passes
umount_tree(mnt, ...)

This is what Christian's "internal watches... execute blocking stuff" would need to enable. The existing fsnotify plumbing (groups, marks, event dispatch) provides the infrastructure, but a new notification hook in do_umount() is required — neither fsnotify_vfsmount_delete() nor fsnotify_sb_delete() fires early enough.

But a hook in do_umount() fires for every mount namespace teardown, not just admin-initiated unmounts. NFSD's callback would need to filter (e.g., only act when it's the last mount of a superblock that NFSD is exporting).

Quoted 5 lines

This is why I originally went with fs_pin. Not saying the series should go back to that, but this is the basic requirement: NFSD needs notification of a umount request while files are still open on that mount, so that it can revoke the NFSv4 state and close those files.

I understand the problem with FS_UNMOUNT, but I fail to understand the desired semantics, specifically the "the last mount of a superblock that NFSD is exporting".

One option is that nfsd will use a private mount clone for accessing files as overlayfs does and wait until all the other mounts are gone, but FWIW that has some user visible implications.

Then we can enable subscribing for FS_MNT_DETACH events on a super_block.
fanotify UAPI currently only allows subscribing them on mntns, but allowing internal users to subscribe on all the unmounts of a sb should be as simple as below (famous last word).

Thanks,
Amir.

fs/notify/fsnotify.c +0 -0
index 9995de1710e59..0abe16db3636c 100644
@@ -695,6 +695,7 @@ void fsnotify_mnt(__u32 mask, struct mnt_namespace

*ns, struct vfsmount *mnt)

 {
        struct fsnotify_mnt data = {
                .ns = ns,
+               .sb = mnt->mnt_sb,
                .mnt_id = real_mount(mnt)->mnt_id_unique,
        };
include/linux/fsnotify_backend.h +0 -0
index 95985400d3d8e..c21fae333f0dc 100644
@@ -332,6 +332,7 @@ static inline const struct path

*file_range_path(const struct file_range *range)

 struct fsnotify_mnt {
        const struct mnt_namespace *ns;
+       struct super_block *sb;
        u64 mnt_id;

};

@@ -395,6 +396,8 @@ static inline struct super_block *fsnotify_data_sb(const void *data,

                return file_range_path(data)->dentry->d_sb;
        case FSNOTIFY_EVENT_ERROR:
                return ((struct fs_error_report *) data)->sb;
+       case FSNOTIFY_EVENT_MNT:
+               return ((struct fsnotify_mnt *)data)->sb;
        default:
                return NULL;
        }
author maintainer
#8 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 74 lines · Amir Goldstein
Quoted 67 lines · Chuck Lever
Quoted 24 lines · Jan Kara
Quoted 23 lines · Chuck Lever
Quoted 21 lines · Amir Goldstein
Quoted 8 lines · Christian Brauner

Another thing: These ad-hoc notifiers are horrific. So I'm pitching another idea and I hope that Jan and Amir can tell me that this is doable...

Can we extend fsnotify so that it's possible for a filesystem to register "internal watches" on relevant objects such as mounts and superblocks and get notified and execute blocking stuff if needed.

You mean like nfsd_file_fsnotify_group? ;)

Quoted 3 lines

Then we don't have to add another set of custom notification mechanisms but have it available in a single subsystem and uniformely available.

I don't see a problem with nfsd registering for FS_UNMOUNT event on sb (once we add it).

As a matter of fact, I think that nfsd can already add an inode mark on the export root path for FS_UNMOUNT event.

Quoted 7 lines
Quoted 3 lines

There isn't much required here aside from getting a synchronous notice that the final file system unmount is going on. I'm happy to try whatever mechanism VFS maintainers are most comfortable with.

Yeah, then as Amir writes placing a mark with FS_UNMOUNT event on the export root path and handling the event in nfsd_file_fsnotify_handle_event() should do what you need?

Turns out FS_UNMOUNT doesn't do what I need.

1/3 here has a fatal flaw: the SRCU notifier does not fire until all files on the mount are closed. The problem is that NFSD holds files open when there is outstanding NFSv4 state. So the SRCU notifier will never fire, on umount, to release that state.

FS_UNMOUNT notifiers have the same issue.

They fire from fsnotify_sb_delete() inside generic_shutdown_super(), which runs inside deactivate_locked_super(), which runs when s_active drops to 0. That requires all mounts to be freed, which requires all NFSD files to be closed: the same problem.

For any notification approach to actually do what is needed, it needs to fire during do_umount(), before propagate_mount_busy(). Something like:

do_umount(mnt):
<- NEW: notify subsystems, allow them to release file refs
retval = propagate_mount_busy(mnt, 2) // now passes
umount_tree(mnt, ...)

This is what Christian's "internal watches... execute blocking stuff" would need to enable. The existing fsnotify plumbing (groups, marks, event dispatch) provides the infrastructure, but a new notification hook in do_umount() is required — neither fsnotify_vfsmount_delete() nor fsnotify_sb_delete() fires early enough.

But a hook in do_umount() fires for every mount namespace teardown, not just admin-initiated unmounts. NFSD's callback would need to filter (e.g., only act when it's the last mount of a superblock that NFSD is exporting).

Quoted 5 lines

This is why I originally went with fs_pin. Not saying the series should go back to that, but this is the basic requirement: NFSD needs notification of a umount request while files are still open on that mount, so that it can revoke the NFSv4 state and close those files.

Quoted 3 lines

I understand the problem with FS_UNMOUNT, but I fail to understand the desired semantics, specifically the "the last mount of a superblock that NFSD is exporting".

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

The discussion here has added some interesting corner cases: NFSD can export bind mounts (portions of a local physical file system); unprivileged users can create and umount file systems using "share".

The goal is to make umount behavior more deterministic without having to insert additional adminstrative steps.

Signature · 1 line
Chuck Lever
#9 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 78 lines · Chuck Lever
Quoted 74 lines · Amir Goldstein
Quoted 72 lines · Chuck Lever
Quoted 24 lines · Jan Kara
Quoted 23 lines · Chuck Lever
Quoted 21 lines · Amir Goldstein
Quoted 8 lines · Christian Brauner

Another thing: These ad-hoc notifiers are horrific. So I'm pitching another idea and I hope that Jan and Amir can tell me that this is doable...

Can we extend fsnotify so that it's possible for a filesystem to register "internal watches" on relevant objects such as mounts and superblocks and get notified and execute blocking stuff if needed.

You mean like nfsd_file_fsnotify_group? ;)

Quoted 3 lines

Then we don't have to add another set of custom notification mechanisms but have it available in a single subsystem and uniformely available.

I don't see a problem with nfsd registering for FS_UNMOUNT event on sb (once we add it).

As a matter of fact, I think that nfsd can already add an inode mark on the export root path for FS_UNMOUNT event.

Quoted 7 lines
Quoted 3 lines

There isn't much required here aside from getting a synchronous notice that the final file system unmount is going on. I'm happy to try whatever mechanism VFS maintainers are most comfortable with.

Yeah, then as Amir writes placing a mark with FS_UNMOUNT event on the export root path and handling the event in nfsd_file_fsnotify_handle_event() should do what you need?

Turns out FS_UNMOUNT doesn't do what I need.

1/3 here has a fatal flaw: the SRCU notifier does not fire until all files on the mount are closed. The problem is that NFSD holds files open when there is outstanding NFSv4 state. So the SRCU notifier will never fire, on umount, to release that state.

FS_UNMOUNT notifiers have the same issue.

They fire from fsnotify_sb_delete() inside generic_shutdown_super(), which runs inside deactivate_locked_super(), which runs when s_active drops to 0. That requires all mounts to be freed, which requires all NFSD files to be closed: the same problem.

For any notification approach to actually do what is needed, it needs to fire during do_umount(), before propagate_mount_busy(). Something like:

do_umount(mnt):
<- NEW: notify subsystems, allow them to release file refs
retval = propagate_mount_busy(mnt, 2) // now passes
umount_tree(mnt, ...)

This is what Christian's "internal watches... execute blocking stuff" would need to enable. The existing fsnotify plumbing (groups, marks, event dispatch) provides the infrastructure, but a new notification hook in do_umount() is required — neither fsnotify_vfsmount_delete() nor fsnotify_sb_delete() fires early enough.

But a hook in do_umount() fires for every mount namespace teardown, not just admin-initiated unmounts. NFSD's callback would need to filter (e.g., only act when it's the last mount of a superblock that NFSD is exporting).

This is why I originally went with fs_pin. Not saying the series should go back to that, but this is the basic requirement: NFSD needs notification of a umount request while files are still open on that mount, so that it can revoke the NFSv4 state and close those files.

Quoted 8 lines
Quoted 3 lines

I understand the problem with FS_UNMOUNT, but I fail to understand the desired semantics, specifically the "the last mount of a superblock that NFSD is exporting".

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Quoted 5 lines

The discussion here has added some interesting corner cases: NFSD can export bind mounts (portions of a local physical file system); unprivileged users can create and umount file systems using "share".

The basic question is whether nfsd is exporting a mount, a filesystem or something in between (i.e. a subtree of a filesystem).

AFAIK, the current implementation is that nfsd is actually exporting one specific mount, so changing the properties of this mount (e.g. readonly) would affect the exported share.

If nfsd wanted to export a subtree of a filesystem it could in theory use a cloned private mount as overlayfs does, but then things like audit logs may not have a full path if the original path used to export the subtree was unmounted.

Quoted 3 lines

The goal is to make umount behavior more deterministic without having to insert additional adminstrative steps.

I would say that if svc_export_put() succeeded, umount should be able to work, but I realize that with automounts, the exports are not actually administered from userspace all the way (?), so perhaps the problem needs to be spelled out in more detail.

Thanks,
Amir.

author maintainer
#10 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 10 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 7 lines
Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Quoted 10 lines
Quoted 3 lines

The discussion here has added some interesting corner cases: NFSD can export bind mounts (portions of a local physical file system); unprivileged users can create and umount file systems using "share".

The basic question is whether nfsd is exporting a mount, a filesystem or something in between (i.e. a subtree of a filesystem).

AFAIK, the current implementation is that nfsd is actually exporting one specific mount, so changing the properties of this mount (e.g. readonly) would affect the exported share.

AIUI NFSD can export starting at any arbitrary directory that appears in the mount namespace. It does not have to start at the local file system's root directory. But even so, outstanding NFSv4 delegations on a narrow portion of a mounted file system will still pin that mount.

Signature · 1 line
Chuck Lever
#11 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 17 lines · Chuck Lever
Quoted 15 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Quoted 7 lines
Quoted 2 lines

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

NeilBrown

author maintainer
#12 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 21 lines · NeilBrown
Quoted 20 lines · Chuck Lever
Quoted 17 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Quoted 8 lines
Quoted 4 lines

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

That meets my needs, but should be passed by the linux-nfs@ review committee.

-F could probably just use the existing "unlock filesystem" API after it does the unexport.

Signature · 1 line
Chuck Lever
maintainer
#13 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 30 lines · Chuck Lever
Quoted 20 lines · NeilBrown
Quoted 19 lines · Chuck Lever
Quoted 17 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Quoted 8 lines
Quoted 4 lines

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

Quoted 5 lines

That meets my needs, but should be passed by the linux-nfs@ review committee.

-F could probably just use the existing "unlock filesystem" API after it does the unexport.

If this option flies, then I guess it is the most sensible variant. If it doesn't work for some reason, then something like ->umount_begin sb callback could be twisted (may possibly need some extension) to provide the needed notification? At least in my naive understanding it was created for usecases like this...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#14 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 37 lines · Jan Kara
Quoted 30 lines · Chuck Lever
Quoted 20 lines · NeilBrown
Quoted 19 lines · Chuck Lever
Quoted 17 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Quoted 8 lines
Quoted 4 lines

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

Quoted 5 lines

That meets my needs, but should be passed by the linux-nfs@ review committee.

-F could probably just use the existing "unlock filesystem" API after it does the unexport.

Quoted 7 lines

If this option flies, then I guess it is the most sensible variant. If it doesn't work for some reason, then something like ->umount_begin sb callback could be twisted (may possibly need some extension) to provide the needed notification? At least in my naive understanding it was created for usecases like this...

Honza

umount_begin is a superblock op that only occurs when MNT_FORCE is set. In this case though, we really want something that calls back into nfsd, rather than to the fs being unmounted.

You could just wire up a bunch of umount_begin() operations but that seems rather nasty. Maybe you could add some sort of callback that nfsd could register that runs just before umount_begin does?

That would be preferable from a UI standpoint than dealing with a new exportfs option, IMO. MNT_FORCE could then (in theory) properly allow for state teardown in nfsd that way, which seems rather natural.

Signature · 1 line
Jeff Layton <[email protected]>
maintainer
#15 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 44 lines · Jeff Layton
Quoted 43 lines · Jan Kara
Quoted 30 lines · Chuck Lever
Quoted 20 lines · NeilBrown
Quoted 19 lines · Chuck Lever
Quoted 17 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Quoted 8 lines
Quoted 4 lines

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

Quoted 5 lines

That meets my needs, but should be passed by the linux-nfs@ review committee.

-F could probably just use the existing "unlock filesystem" API after it does the unexport.

If this option flies, then I guess it is the most sensible variant. If it doesn't work for some reason, then something like ->umount_begin sb callback could be twisted (may possibly need some extension) to provide the needed notification? At least in my naive understanding it was created for usecases like this...

Quoted 5 lines
Quoted 1 line

Honza

umount_begin is a superblock op that only occurs when MNT_FORCE is set. In this case though, we really want something that calls back into nfsd, rather than to the fs being unmounted.

I see OK.

Quoted 3 lines

You could just wire up a bunch of umount_begin() operations but that seems rather nasty. Maybe you could add some sort of callback that nfsd could register that runs just before umount_begin does?

Thinking about this more - Chuck was also writing about the problem of needing to shutdown the state only when this is the last unmount of a superblock but until we grab namespace_lock(), that's impossible to tell in a race-free manner? And how about lazy unmounts? There it would seem to be extra hard to determine when NFS needs to drop it's delegations since you need to figure out whether all file references are NFS internal only? It all seems like a notification from VFS isn't the right place to solve this issue...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
maintainer
#16 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 57 lines · Jan Kara
Quoted 44 lines · Jeff Layton
Quoted 43 lines · Jan Kara
Quoted 30 lines · Chuck Lever
Quoted 20 lines · NeilBrown
Quoted 19 lines · Chuck Lever
Quoted 17 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Quoted 8 lines
Quoted 4 lines

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

Quoted 5 lines

That meets my needs, but should be passed by the linux-nfs@ review committee.

-F could probably just use the existing "unlock filesystem" API after it does the unexport.

If this option flies, then I guess it is the most sensible variant. If it doesn't work for some reason, then something like ->umount_begin sb callback could be twisted (may possibly need some extension) to provide the needed notification? At least in my naive understanding it was created for usecases like this...

Quoted 5 lines
Quoted 1 line

Honza

umount_begin is a superblock op that only occurs when MNT_FORCE is set. In this case though, we really want something that calls back into nfsd, rather than to the fs being unmounted.

I see OK.

Quoted 3 lines

You could just wire up a bunch of umount_begin() operations but that seems rather nasty. Maybe you could add some sort of callback that nfsd could register that runs just before umount_begin does?

Quoted 9 lines

Thinking about this more - Chuck was also writing about the problem of needing to shutdown the state only when this is the last unmount of a superblock but until we grab namespace_lock(), that's impossible to tell in a race-free manner? And how about lazy unmounts? There it would seem to be extra hard to determine when NFS needs to drop it's delegations since you need to figure out whether all file references are NFS internal only? It all seems like a notification from VFS isn't the right place to solve this issue...

The issue is that traditionally, "exportfs -u" is what unexports the filesystem and at that point you can (usually) unmount it. We'd ideally like to have a solution that doesn't create extra steps or change this, since there is already a lot of automation and muscle memory around these commands.

This method mostly works with v3, since there is no long term state (technically lockd can hold some, but that's only for file locking). With v4 that changed and nfsd holds files open for much longer.

We can't drop all the state when fs is unexported, as it's not uncommon for it to be reexported soon afterward, and we can't force a grace period at that point to allow reclaim.

Unmounting seems like the natural place for this. At the point where you're unmounting, there can be no more state and the admin's intent is clear. Tearing down nfsd state at that point seems pretty safe.

If we can't add some sort of hook to the umount path, then I'll understand, but it would be a nice to have for this use-case.

Signature · 1 line
Jeff Layton <[email protected]>
#17 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 82 lines · Jeff Layton
Quoted 57 lines · Jan Kara
Quoted 44 lines · Jeff Layton
Quoted 43 lines · Jan Kara
Quoted 30 lines · Chuck Lever
Quoted 20 lines · NeilBrown
Quoted 19 lines · Chuck Lever
Quoted 17 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Quoted 8 lines
Quoted 4 lines

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

Quoted 5 lines

That meets my needs, but should be passed by the linux-nfs@ review committee.

-F could probably just use the existing "unlock filesystem" API after it does the unexport.

If this option flies, then I guess it is the most sensible variant. If it doesn't work for some reason, then something like ->umount_begin sb callback could be twisted (may possibly need some extension) to provide the needed notification? At least in my naive understanding it was created for usecases like this...

Quoted 5 lines
Quoted 1 line

Honza

umount_begin is a superblock op that only occurs when MNT_FORCE is set. In this case though, we really want something that calls back into nfsd, rather than to the fs being unmounted.

I see OK.

Quoted 3 lines

You could just wire up a bunch of umount_begin() operations but that seems rather nasty. Maybe you could add some sort of callback that nfsd could register that runs just before umount_begin does?

Quoted 9 lines

Thinking about this more - Chuck was also writing about the problem of needing to shutdown the state only when this is the last unmount of a superblock but until we grab namespace_lock(), that's impossible to tell in a race-free manner? And how about lazy unmounts? There it would seem to be extra hard to determine when NFS needs to drop it's delegations since you need to figure out whether all file references are NFS internal only? It all seems like a notification from VFS isn't the right place to solve this issue...

The issue is that traditionally, "exportfs -u" is what unexports the filesystem and at that point you can (usually) unmount it. We'd ideally like to have a solution that doesn't create extra steps or change this, since there is already a lot of automation and muscle memory around these commands.

This method mostly works with v3, since there is no long term state (technically lockd can hold some, but that's only for file locking). With v4 that changed and nfsd holds files open for much longer.

We can't drop all the state when fs is unexported, as it's not uncommon for it to be reexported soon afterward, and we can't force a grace period at that point to allow reclaim.

Quoted 6 lines

Unmounting seems like the natural place for this. At the point where you're unmounting, there can be no more state and the admin's intent is clear. Tearing down nfsd state at that point seems pretty safe.

If we can't add some sort of hook to the umount path, then I'll understand, but it would be a nice to have for this use-case.

At first glance, umount seems like a natural place for a lot of things.

The locking and the guarantees that we have traditionally given to userspace make it a very convoluted codepath and I'm very hesitant to add more complexity in this part of the code.

Now I suggested the fsnotify mechanism because it's already there and if it is _reasonably_ easy to provide the notification that nfs needs to clean up whatever it needs to clean up than this is probably fine. What I absolutely don't want is to have another custom notification mechanism in the VFS layer.

But if we can solve this in userspace then it is absolutely the preferred variant and what we should do.

author maintainer
#18 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 96 lines · Christian Brauner
Quoted 82 lines · Jeff Layton
Quoted 57 lines · Jan Kara
Quoted 44 lines · Jeff Layton
Quoted 43 lines · Jan Kara
Quoted 30 lines · Chuck Lever
Quoted 20 lines · NeilBrown
Quoted 19 lines · Chuck Lever
Quoted 17 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Quoted 8 lines
Quoted 4 lines

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

Quoted 5 lines

That meets my needs, but should be passed by the linux-nfs@ review committee.

-F could probably just use the existing "unlock filesystem" API after it does the unexport.

If this option flies, then I guess it is the most sensible variant. If it doesn't work for some reason, then something like ->umount_begin sb callback could be twisted (may possibly need some extension) to provide the needed notification? At least in my naive understanding it was created for usecases like this...

Quoted 5 lines
Quoted 1 line

Honza

umount_begin is a superblock op that only occurs when MNT_FORCE is set. In this case though, we really want something that calls back into nfsd, rather than to the fs being unmounted.

I see OK.

Quoted 3 lines

You could just wire up a bunch of umount_begin() operations but that seems rather nasty. Maybe you could add some sort of callback that nfsd could register that runs just before umount_begin does?

Quoted 9 lines

Thinking about this more - Chuck was also writing about the problem of needing to shutdown the state only when this is the last unmount of a superblock but until we grab namespace_lock(), that's impossible to tell in a race-free manner? And how about lazy unmounts? There it would seem to be extra hard to determine when NFS needs to drop it's delegations since you need to figure out whether all file references are NFS internal only? It all seems like a notification from VFS isn't the right place to solve this issue...

The issue is that traditionally, "exportfs -u" is what unexports the filesystem and at that point you can (usually) unmount it. We'd ideally like to have a solution that doesn't create extra steps or change this, since there is already a lot of automation and muscle memory around these commands.

This method mostly works with v3, since there is no long term state (technically lockd can hold some, but that's only for file locking). With v4 that changed and nfsd holds files open for much longer.

We can't drop all the state when fs is unexported, as it's not uncommon for it to be reexported soon afterward, and we can't force a grace period at that point to allow reclaim.

Quoted 6 lines

Unmounting seems like the natural place for this. At the point where you're unmounting, there can be no more state and the admin's intent is clear. Tearing down nfsd state at that point seems pretty safe.

If we can't add some sort of hook to the umount path, then I'll understand, but it would be a nice to have for this use-case.

At first glance, umount seems like a natural place for a lot of things.

The locking and the guarantees that we have traditionally given to userspace make it a very convoluted codepath and I'm very hesitant to add more complexity in this part of the code.

Quoted 8 lines

Now I suggested the fsnotify mechanism because it's already there and if it is _reasonably_ easy to provide the notification that nfs needs to clean up whatever it needs to clean up than this is probably fine. What I absolutely don't want is to have another custom notification mechanism in the VFS layer.

But if we can solve this in userspace then it is absolutely the preferred variant and what we should do.

No problem with that line of reasoning. Right now it looks like we can do this with changes to exportfs, so I will pursue that.

Signature · 1 line
Chuck Lever
#19 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

In reply to Jan Kara
Quoted 57 lines · Jan Kara
Quoted 44 lines · Jeff Layton
Quoted 43 lines · Jan Kara
Quoted 30 lines · Chuck Lever
Quoted 20 lines · NeilBrown
Quoted 19 lines · Chuck Lever
Quoted 17 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Quoted 8 lines
Quoted 4 lines

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

Quoted 5 lines

That meets my needs, but should be passed by the linux-nfs@ review committee.

-F could probably just use the existing "unlock filesystem" API after it does the unexport.

If this option flies, then I guess it is the most sensible variant. If it doesn't work for some reason, then something like ->umount_begin sb callback could be twisted (may possibly need some extension) to provide the needed notification? At least in my naive understanding it was created for usecases like this...

Quoted 5 lines
Quoted 1 line

Honza

umount_begin is a superblock op that only occurs when MNT_FORCE is set. In this case though, we really want something that calls back into nfsd, rather than to the fs being unmounted.

I see OK.

Quoted 3 lines

You could just wire up a bunch of umount_begin() operations but that seems rather nasty. Maybe you could add some sort of callback that nfsd could register that runs just before umount_begin does?

Quoted 8 lines

Thinking about this more - Chuck was also writing about the problem of needing to shutdown the state only when this is the last unmount of a superblock but until we grab namespace_lock(), that's impossible to tell in a race-free manner? And how about lazy unmounts? There it would seem to be extra hard to determine when NFS needs to drop it's delegations since you need to figure out whether all file references are NFS internal only? It all seems like a notification from VFS isn't the right place to solve this issue...

It isn't clear to me that "last unmount" is the correct target.

The nfsd file cache (which I think is the main focus here - it holds delegated files etc) caches files. i.e. "struct file *". This identifies a dentry on a vfsmount.
I think it could be appropriate to drop nfsd state when the vfsmount is being considered for unmount - including lazy unmount - whether the superblock has other mounts or not.

An unmount attempt should fail if the vfsmount is still in use, including in use in the the nfsd export table. And if it is going to fail, then we don't want to drop nfsd state. But if it would succeed after nfsd state was dropped - then and only then do we want to drop nfsd state.

I think it would be messy to get this dependency correct.

i.e. I think that guessing, in the kernel, what the user wants is problematic. I think giving the user the tools to say exactly what they want is best.

NeilBrown

Quoted 6 lines

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#20 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

In reply to Chuck Lever
Quoted 27 lines · Chuck Lever
Quoted 26 lines · NeilBrown
Quoted 24 lines · Chuck Lever
Quoted 10 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 7 lines
Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Quoted 6 lines
Quoted 3 lines

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

That meets my needs, but should be passed by the linux-nfs@ review committee.

Discussed with the reporter. -F addresses the automation requirement, but users still expect "exportfs -u" to work the same way for NFSv3 and
NFSv4: "unexport" followed by "unmount" always works.

I am not remembering clearly why the linux-nfs folks though that NFSv4 delegations should stay in place after unexport. In my view, unexport should be a security boundary, stopping access to the files on the export.

But during a warm server reboot, do we want that behavior?

Quoted 2 lines

-F could probably just use the existing "unlock filesystem" API after it does the unexport.

Signature · 1 line
Chuck Lever
#21 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 35 lines · Chuck Lever
Quoted 27 lines · Chuck Lever
Quoted 26 lines · NeilBrown
Quoted 24 lines · Chuck Lever
Quoted 10 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 7 lines
Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Quoted 6 lines
Quoted 3 lines

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

That meets my needs, but should be passed by the linux-nfs@ review committee.

Quoted 8 lines

Discussed with the reporter. -F addresses the automation requirement, but users still expect "exportfs -u" to work the same way for NFSv3 and
NFSv4: "unexport" followed by "unmount" always works.

I am not remembering clearly why the linux-nfs folks though that NFSv4 delegations should stay in place after unexport. In my view, unexport should be a security boundary, stopping access to the files on the export.

At the time when the API was growing, delegations were barely an unhatched idea.

unexport may be a security boundary, but it is not so obvious that it is a state boundary.

The kernel is not directly involved in whether something is exported or not. That is under the control of mountd/exportfs. The kernel keeps a cache of info from there. So if you want to impose a state boundary, it really should involved mountd/exportfs.

There was once this idea floating around that policy didn't belong in the kernel.

NeilBrown

Quoted 10 lines

But during a warm server reboot, do we want that behavior?

Quoted 2 lines

-F could probably just use the existing "unlock filesystem" API after it does the unexport.

Signature · 1 line
Chuck Lever
#22 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 51 lines · NeilBrown
Quoted 35 lines · Chuck Lever
Quoted 27 lines · Chuck Lever
Quoted 26 lines · NeilBrown
Quoted 24 lines · Chuck Lever
Quoted 10 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 7 lines
Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Quoted 6 lines
Quoted 3 lines

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

That meets my needs, but should be passed by the linux-nfs@ review committee.

Quoted 8 lines

Discussed with the reporter. -F addresses the automation requirement, but users still expect "exportfs -u" to work the same way for NFSv3 and
NFSv4: "unexport" followed by "unmount" always works.

I am not remembering clearly why the linux-nfs folks though that NFSv4 delegations should stay in place after unexport. In my view, unexport should be a security boundary, stopping access to the files on the export.

At the time when the API was growing, delegations were barely an unhatched idea.

unexport may be a security boundary, but it is not so obvious that it is a state boundary.

Quoted 7 lines

The kernel is not directly involved in whether something is exported or not. That is under the control of mountd/exportfs. The kernel keeps a cache of info from there. So if you want to impose a state boundary, it really should involved mountd/exportfs.

There was once this idea floating around that policy didn't belong in the kernel.

I consider enabling unmount after unexport more "mechanism" than "policy", but not so much that I'm about to get religious about it. It appears that the expedient path forward would be teaching exportfs to do an "unlock filesystem" after it finishes unexporting, and leaving the kernel untouched.

The question now is whether exportfs should grow a command-line option to modulate this behavior:

- Some users consider the current situation as a regression -- unmount after unexport used to work seamlessly with NFSv3; still does; but not with NFSv4.

- Some users might consider changing the current unexport behavior as introducing a regression -- they rely on NFSv4 state continuing to exist after unexport. That behavior isn't documented anywhere, I suspect.

Thus I'm not sure exactly what change to exportfs is most appropriate.

Signature · 1 line
Chuck Lever
#23 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 66 lines · Chuck Lever
Quoted 51 lines · NeilBrown
Quoted 35 lines · Chuck Lever
Quoted 27 lines · Chuck Lever
Quoted 26 lines · NeilBrown
Quoted 24 lines · Chuck Lever
Quoted 10 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 7 lines
Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Quoted 6 lines
Quoted 3 lines

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

That meets my needs, but should be passed by the linux-nfs@ review committee.

Quoted 8 lines

Discussed with the reporter. -F addresses the automation requirement, but users still expect "exportfs -u" to work the same way for NFSv3 and
NFSv4: "unexport" followed by "unmount" always works.

I am not remembering clearly why the linux-nfs folks though that NFSv4 delegations should stay in place after unexport. In my view, unexport should be a security boundary, stopping access to the files on the export.

At the time when the API was growing, delegations were barely an unhatched idea.

unexport may be a security boundary, but it is not so obvious that it is a state boundary.

Quoted 7 lines

The kernel is not directly involved in whether something is exported or not. That is under the control of mountd/exportfs. The kernel keeps a cache of info from there. So if you want to impose a state boundary, it really should involved mountd/exportfs.

There was once this idea floating around that policy didn't belong in the kernel.

I consider enabling unmount after unexport more "mechanism" than "policy", but not so much that I'm about to get religious about it. It appears that the expedient path forward would be teaching exportfs to do an "unlock filesystem" after it finishes unexporting, and leaving the kernel untouched.

Quoted 6 lines

The question now is whether exportfs should grow a command-line option to modulate this behavior:

- Some users consider the current situation as a regression -- unmount after unexport used to work seamlessly with NFSv3; still does; but not with NFSv4.

They are of course welcome to keep using NFSv3 (and to not lock files) :-)

Quoted 7 lines

- Some users might consider changing the current unexport behavior as introducing a regression -- they rely on NFSv4 state continuing to exist after unexport. That behavior isn't documented anywhere, I suspect.

Thus I'm not sure exactly what change to exportfs is most appropriate.

I think any purging of the cache should happen at unexport time, not transparently when unmount is attempted as I think the ordering semantics there are complex.

And as the kernel doesn't know when something has been unexported, it must be exportfs which initiates the cache purge.

So the only interesting question I can see is:
do we mount "purge on unexport" the default, or do we require an explicit request (-F)?
A complexity here is that a given filesystem can be exported to different clients with different options, and different subtrees can be exported. If the cache-flush were to be the default, it would need to be on the last export of any path to the filesystem. This would need to include implicit exports via crossmnt. I think this would be hard to specify and document well.

So I think an explicit "flush cache" exportfs action is simplest and best.
Possibly:
exportfs -F /some/path
would unexport all exports which reference the same mountpoint, then would tell the kernel to drop all cached data for that mount.

Thanks,
NeilBrown

#24 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 87 lines · NeilBrown
Quoted 66 lines · Chuck Lever
Quoted 51 lines · NeilBrown
Quoted 35 lines · Chuck Lever
Quoted 27 lines · Chuck Lever
Quoted 26 lines · NeilBrown
Quoted 24 lines · Chuck Lever
Quoted 10 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 7 lines
Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Quoted 6 lines
Quoted 3 lines

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

That meets my needs, but should be passed by the linux-nfs@ review committee.

Quoted 8 lines

Discussed with the reporter. -F addresses the automation requirement, but users still expect "exportfs -u" to work the same way for NFSv3 and
NFSv4: "unexport" followed by "unmount" always works.

I am not remembering clearly why the linux-nfs folks though that NFSv4 delegations should stay in place after unexport. In my view, unexport should be a security boundary, stopping access to the files on the export.

At the time when the API was growing, delegations were barely an unhatched idea.

unexport may be a security boundary, but it is not so obvious that it is a state boundary.

Quoted 7 lines

The kernel is not directly involved in whether something is exported or not. That is under the control of mountd/exportfs. The kernel keeps a cache of info from there. So if you want to impose a state boundary, it really should involved mountd/exportfs.

There was once this idea floating around that policy didn't belong in the kernel.

I consider enabling unmount after unexport more "mechanism" than "policy", but not so much that I'm about to get religious about it. It appears that the expedient path forward would be teaching exportfs to do an "unlock filesystem" after it finishes unexporting, and leaving the kernel untouched.

Quoted 6 lines

The question now is whether exportfs should grow a command-line option to modulate this behavior:

- Some users consider the current situation as a regression -- unmount after unexport used to work seamlessly with NFSv3; still does; but not with NFSv4.

They are of course welcome to keep using NFSv3 (and to not lock files) :-)

Quoted 6 lines

- Some users might consider changing the current unexport behavior as introducing a regression -- they rely on NFSv4 state continuing to exist after unexport. That behavior isn't documented anywhere, I suspect.

Thus I'm not sure exactly what change to exportfs is most appropriate.

I think any purging of the cache should happen at unexport time, not transparently when unmount is attempted as I think the ordering semantics there are complex.

Quoted 6 lines

And as the kernel doesn't know when something has been unexported, it must be exportfs which initiates the cache purge.

So the only interesting question I can see is:
do we mount "purge on unexport" the default, or do we require an explicit request (-F)?

Yes, that's what I was trying to say above.

Quoted 6 lines

A complexity here is that a given filesystem can be exported to different clients with different options, and different subtrees can be exported. If the cache-flush were to be the default, it would need to be on the last export of any path to the filesystem. This would need to include implicit exports via crossmnt. I think this would be hard to specify and document well.

Is there nothing we can do to engineer the exportfs command to remove some of this complexity?

Quoted 6 lines

So I think an explicit "flush cache" exportfs action is simplest and best.
Possibly:
exportfs -F /some/path
would unexport all exports which reference the same mountpoint, then would tell the kernel to drop all cached data for that mount.

I passed along your original "-F" suggestion to the original reporter a few days ago, and it was not met with universal glee and a huzzah.

Although "-F" can be added to automation easily enough, their preference, based on their own users' experience, is that the fix should not require changes in user behavior.

Signature · 1 line
Chuck Lever
#25 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

Quoted 105 lines · Chuck Lever
Quoted 87 lines · NeilBrown
Quoted 66 lines · Chuck Lever
Quoted 51 lines · NeilBrown
Quoted 35 lines · Chuck Lever
Quoted 27 lines · Chuck Lever
Quoted 26 lines · NeilBrown
Quoted 24 lines · Chuck Lever
Quoted 10 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 7 lines
Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Quoted 6 lines
Quoted 3 lines

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

That meets my needs, but should be passed by the linux-nfs@ review committee.

Quoted 8 lines

Discussed with the reporter. -F addresses the automation requirement, but users still expect "exportfs -u" to work the same way for NFSv3 and
NFSv4: "unexport" followed by "unmount" always works.

I am not remembering clearly why the linux-nfs folks though that NFSv4 delegations should stay in place after unexport. In my view, unexport should be a security boundary, stopping access to the files on the export.

At the time when the API was growing, delegations were barely an unhatched idea.

unexport may be a security boundary, but it is not so obvious that it is a state boundary.

Quoted 7 lines

The kernel is not directly involved in whether something is exported or not. That is under the control of mountd/exportfs. The kernel keeps a cache of info from there. So if you want to impose a state boundary, it really should involved mountd/exportfs.

There was once this idea floating around that policy didn't belong in the kernel.

I consider enabling unmount after unexport more "mechanism" than "policy", but not so much that I'm about to get religious about it. It appears that the expedient path forward would be teaching exportfs to do an "unlock filesystem" after it finishes unexporting, and leaving the kernel untouched.

Quoted 6 lines

The question now is whether exportfs should grow a command-line option to modulate this behavior:

- Some users consider the current situation as a regression -- unmount after unexport used to work seamlessly with NFSv3; still does; but not with NFSv4.

They are of course welcome to keep using NFSv3 (and to not lock files) :-)

Quoted 6 lines

- Some users might consider changing the current unexport behavior as introducing a regression -- they rely on NFSv4 state continuing to exist after unexport. That behavior isn't documented anywhere, I suspect.

Thus I'm not sure exactly what change to exportfs is most appropriate.

I think any purging of the cache should happen at unexport time, not transparently when unmount is attempted as I think the ordering semantics there are complex.

Quoted 6 lines

And as the kernel doesn't know when something has been unexported, it must be exportfs which initiates the cache purge.

So the only interesting question I can see is:
do we mount "purge on unexport" the default, or do we require an explicit request (-F)?

Yes, that's what I was trying to say above.

Quoted 6 lines

A complexity here is that a given filesystem can be exported to different clients with different options, and different subtrees can be exported. If the cache-flush were to be the default, it would need to be on the last export of any path to the filesystem. This would need to include implicit exports via crossmnt. I think this would be hard to specify and document well.

Quoted 2 lines

Is there nothing we can do to engineer the exportfs command to remove some of this complexity?

An argument could certainly be made that the exportfs command attempts to be too general and consequently is too vague, and that a better interface could be defined that was more opinionated and less flexible, and so was easier to use without risk of complexity.

But that would be a different tool.
The complexity I see is not in the implementation, which could be re-engineers, but in the design/behaviour which cannot without user-visible change.

Quoted 9 lines
Quoted 8 lines

So I think an explicit "flush cache" exportfs action is simplest and best.
Possibly:
exportfs -F /some/path
would unexport all exports which reference the same mountpoint, then would tell the kernel to drop all cached data for that mount.

Quoted 6 lines

I passed along your original "-F" suggestion to the original reporter a few days ago, and it was not met with universal glee and a huzzah.

Although "-F" can be added to automation easily enough, their preference, based on their own users' experience, is that the fix should not require changes in user behavior.

I think their preference, while understandable, is naive.
They are doing something that is not documented as supported, is not actually supported, and has always had the possibility of failure. They have switched from v3 to v4 (possibly following a default) and now the failure is more likely...
Maybe the best fix without changes in user behaviour is to switch the default back to v3....

NeilBrown

Quoted 5 lines
Signature · 2 lines
-- 
Chuck Lever
#26 lore raw

Re: [PATCH v3 1/3] fs: add umount notifier chain for filesystem unmount notification

In reply to NeilBrown
Quoted 51 lines · NeilBrown
Quoted 35 lines · Chuck Lever
Quoted 27 lines · Chuck Lever
Quoted 26 lines · NeilBrown
Quoted 24 lines · Chuck Lever
Quoted 10 lines · Amir Goldstein
Quoted 4 lines · Chuck Lever

Perhaps that description nails down too much implementation detail, and it might be stale. A broader description is this user story:

"As a system administrator, I'd like to be able to unexport an NFSD

Doesn't "unexporting" involve communicating to nfsd? Meaning calling to svc_export_put() to path_put() the share root path?

Quoted 7 lines
Quoted 4 lines

share that is being accessed by NFSv4 clients, and then unmount it, reliably (for example, via automation). Currently the umount step hangs if there are still outstanding delegations granted to the NFSv4 clients."

Can't svc_export_put() be the trigger for nfsd to release all resources associated with this share?

Currently unexport does not revoke NFSv4 state. So, that would be a user-visible behavior change. I suggested that approach a few months ago to linux-nfs@ and there was push-back.

Quoted 6 lines
Quoted 3 lines

Could we add a "-F" or similar flag to "exportfs -u" which implements the desired semantic? i.e. asking nfsd to release all locks and close all state on the filesystem.

That meets my needs, but should be passed by the linux-nfs@ review committee.

Quoted 8 lines

Discussed with the reporter. -F addresses the automation requirement, but users still expect "exportfs -u" to work the same way for NFSv3 and
NFSv4: "unexport" followed by "unmount" always works.

I am not remembering clearly why the linux-nfs folks though that NFSv4 delegations should stay in place after unexport. In my view, unexport should be a security boundary, stopping access to the files on the export.

At the time when the API was growing, delegations were barely an unhatched idea.

unexport may be a security boundary, but it is not so obvious that it is a state boundary.

Quoted 7 lines

The kernel is not directly involved in whether something is exported or not. That is under the control of mountd/exportfs. The kernel keeps a cache of info from there. So if you want to impose a state boundary, it really should involved mountd/exportfs.

There was once this idea floating around that policy didn't belong in the kernel.

Very much agree.

author maintainer
#27 lore raw

PATCH v3 2/3 nfsd: revoke NFSv4 state when filesystem is unmounted

In reply to Chuck Lever

From: Chuck Lever <[email protected]>

When an NFS server's local filesystem is unmounted while NFS clients are still accessing it, NFSv4 state holds files open which pins the filesystem, preventing unmount.

Previously, administrators had to manually revoke state via /proc/fs/nfsd/unlock_fs before a formerly exported filesystem could be unmounted.

Register with the VFS umount notifier chain to detect filesystem unmounts and revoke NFSv4 state and NLM locks associated with that filesystem. An xarray in nfsd_net tracks per-superblock entries. When NFS state is created for a file on a given superblock, an entry is registered (idempotently) for that superblock. When the filesystem is unmounted, VFS invokes the notifier callback which queues work to:

- Cancel ongoing async COPY operations (nfsd4_cancel_copy_by_sb)
- Release NLM locks (nlmsvc_unlock_all_by_sb)
- Revoke NFSv4 state (nfsd4_revoke_states)

Each network namespace registers its own notifier_block, allowing the callback to directly access the correct nfsd_net via container_of().

The revocation work runs on a dedicated workqueue (nfsd_sb_wq) to avoid deadlocks since the VFS notifier callback should not block for extended periods. Synchronization between VFS unmount and NFSD shutdown uses xa_erase() atomicity: the path that successfully erases the xarray entry triggers work.

If state revocation takes an unexpectedly long time (e.g., when re-exporting an NFS mount whose backend server is unresponsive), periodic warnings are emitted every 30 seconds. The wait is
interruptible: if interrupted before work starts, cancel_work() removes the queued work and revocation runs directly in the unmount context; if work is already running, the callback returns and revocation continues in the background. Open files keep the superblock alive until revocation closes them.

---
 fs/nfsd/Makefile    |   2 +-
 fs/nfsd/netns.h     |   5 +
 fs/nfsd/nfs4state.c |  29 +++++
 fs/nfsd/nfsctl.c    |  10 +-
 fs/nfsd/sb_watch.c  | 273 ++++++++++++++++++++++++++++++++++++++++++++
 fs/nfsd/state.h     |   7 ++
 6 files changed, 323 insertions(+), 3 deletions(-)
 create mode 100644 fs/nfsd/sb_watch.c
fs/nfsd/Makefile +1 -1
index f0da4d69dc74..bf6146283165 100644
@@ -13,7 +13,7 @@ nfsd-y			+= trace.o
 nfsd-y 			+= nfssvc.o nfsctl.o nfsfh.o vfs.o \
 			   export.o auth.o lockd.o nfscache.o \
 			   stats.o filecache.o nfs3proc.o nfs3xdr.o \
-			   netlink.o
+			   netlink.o sb_watch.o
 nfsd-$(CONFIG_NFSD_V2) += nfsproc.o nfsxdr.o
 nfsd-$(CONFIG_NFSD_V2_ACL) += nfs2acl.o
 nfsd-$(CONFIG_NFSD_V3_ACL) += nfs3acl.o
fs/nfsd/netns.h +5 -0
index 9fa600602658..bc6004c85a4d 100644
@@ -13,6 +13,7 @@
 #include <linux/filelock.h>
 #include <linux/nfs4.h>
 #include <linux/percpu_counter.h>
+#include <linux/xarray.h>
 #include <linux/percpu-refcount.h>
 #include <linux/siphash.h>
 #include <linux/sunrpc/stats.h>
@@ -219,6 +220,10 @@ struct nfsd_net {
 	/* last time an admin-revoke happened for NFSv4.0 */
 	time64_t		nfs40_last_revoke;
 
+	/* Superblock watch for automatic state revocation on unmount */
+	struct xarray		nfsd_sb_watches;
+	struct notifier_block	nfsd_umount_notifier;
+
 #if IS_ENABLED(CONFIG_NFS_LOCALIO)
 	/* Local clients to be invalidated when net is shut down */
 	spinlock_t              local_clients_lock;
fs/nfsd/nfs4state.c +29 -0
index 6b9c399b89df..d35a578db1c0 100644
@@ -6463,6 +6463,16 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf
 		status = nfserr_bad_stateid;
 		if (nfsd4_is_deleg_cur(open))
 			goto out;
+		/*
+		 * Watch the superblock so unmount can trigger revocation
+		 * of NFSv4 state (opens, locks, delegations) held by
+		 * clients on this filesystem. nfsd_sb_watch() returns
+		 * immediately if a watch already exists for this sb.
+		 */
+		status = nfsd_sb_watch(SVC_NET(rqstp),
+				       current_fh->fh_export->ex_path.mnt);
+		if (status)
+			goto out;
 	}
 
 	if (!stp) {
@@ -9010,8 +9020,13 @@ static int nfs4_state_create_net(struct net *net)
 
 	shrinker_register(nn->nfsd_client_shrinker);
 
+	if (nfsd_sb_watch_setup(nn))
+		goto err_sb_entries;
+
 	return 0;
 
+err_sb_entries:
+	shrinker_free(nn->nfsd_client_shrinker);
 err_shrinker:
 	put_net(net);
 	kfree(nn->sessionid_hashtbl);
@@ -9111,6 +9126,8 @@ nfs4_state_shutdown_net(struct net *net)
 	struct list_head *pos, *next, reaplist;
 	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
 
+	nfsd_sb_watch_shutdown(nn);
+
 	shrinker_free(nn->nfsd_client_shrinker);
 	cancel_work_sync(&nn->nfsd_shrinker_work);
 	disable_delayed_work_sync(&nn->laundromat_work);
@@ -9465,6 +9482,18 @@ nfsd_get_dir_deleg(struct nfsd4_compound_state *cstate,
 	if (rfp != fp) {
 		put_nfs4_file(fp);
 		fp = rfp;
+	} else {
+		/*
+		 * Watch the superblock so unmount can trigger revocation
+		 * of directory delegations held by clients on this
+		 * filesystem. nfsd_sb_watch() returns immediately if a
+		 * watch already exists for this sb.
+		 */
+		if (nfsd_sb_watch(clp->net,
+				  cstate->current_fh.fh_export->ex_path.mnt)) {
+			put_nfs4_file(fp);
+			return ERR_PTR(-EAGAIN);
+		}
 	}
 
 	/* if this client already has one, return that it's unavailable */
fs/nfsd/nfsctl.c +8 -2
index e9acd2cd602c..5d8a95a48ff9 100644
@@ -2268,9 +2268,12 @@ static int __init init_nfsd(void)
 	retval = nfsd4_create_laundry_wq();
 	if (retval)
 		goto out_free_cld;
+	retval = nfsd_sb_watch_init();
+	if (retval)
+		goto out_free_laundry;
 	retval = register_filesystem(&nfsd_fs_type);
 	if (retval)
-		goto out_free_nfsd4;
+		goto out_free_sb;
 	retval = genl_register_family(&nfsd_nl_family);
 	if (retval)
 		goto out_free_filesystem;
@@ -2284,7 +2287,9 @@ static int __init init_nfsd(void)
 	genl_unregister_family(&nfsd_nl_family);
 out_free_filesystem:
 	unregister_filesystem(&nfsd_fs_type);
-out_free_nfsd4:
+out_free_sb:
+	nfsd_sb_watch_exit();
+out_free_laundry:
 	nfsd4_destroy_laundry_wq();
 out_free_cld:
 	unregister_cld_notifier();
@@ -2307,6 +2312,7 @@ static void __exit exit_nfsd(void)
 	remove_proc_entry("fs/nfs", NULL);
 	genl_unregister_family(&nfsd_nl_family);
 	unregister_filesystem(&nfsd_fs_type);
+	nfsd_sb_watch_exit();
 	nfsd4_destroy_laundry_wq();
 	unregister_cld_notifier();
 	unregister_pernet_subsys(&nfsd_net_ops);
fs/nfsd/sb_watch.c new file +273 -0

273 lines not on this page · raw message

fs/nfsd/state.h +7 -0
index 6fcbf1e427d4..4b57d04f868a 100644
@@ -853,6 +853,13 @@ static inline void nfsd4_cancel_copy_by_sb(struct net *net, struct super_block *
 }
 #endif
 
+/* superblock watch for unmount notification (sb_watch.c) */
+int nfsd_sb_watch_init(void);
+void nfsd_sb_watch_exit(void);
+__be32 nfsd_sb_watch(struct net *net, struct vfsmount *mnt);
+int nfsd_sb_watch_setup(struct nfsd_net *nn);
+void nfsd_sb_watch_shutdown(struct nfsd_net *nn);
+
 /* grace period management */
 bool nfsd4_force_end_grace(struct nfsd_net *nn);
 
Signature · 1 line
2.53.0
author maintainer
#28 lore raw

PATCH v3 3/3 nfsd: close cached files on filesystem unmount

In reply to Chuck Lever

From: Chuck Lever <[email protected]>

When a filesystem is unmounted while NFS is exporting it, the unmount can fail with EBUSY even after NFSv4 state has been revoked. This occurs because the nfsd_file cache holds open NFSv2/3 file handles that pin the filesystem.

Extend the mechanism that revokes NFSv4 state on unmount to also close cached file handles. nfsd_file_close_sb() walks the nfsd_file cache and disposes of entries belonging to the target superblock. It runs after NFSv4 state revocation, handling NFSv2/3 file handles that remain in the cache.

Entries under construction (nf_file not yet set) are skipped; these have no open file to close.

The hashtable walk releases the mutex periodically to avoid blocking other NFSD operations during large cache walks. Entries are disposed incrementally in batches, keeping memory usage bounded and spreading the I/O load.

A log message is emitted when cached file handles are closed during unmount, informing administrators that NFS clients may receive stale file handle errors.

A flush_workqueue() call is added to nfsd_sb_watch_shutdown() to ensure that any work items still executing complete before shutdown proceeds. Without this, if an unmount notification returns early due to signal interruption while the work function is still running, nfsd_file_cache_shutdown() could destroy the file cache slab while nfsd_file_close_sb() is still disposing entries.

---
 fs/nfsd/filecache.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
 fs/nfsd/filecache.h |  1 +
 fs/nfsd/sb_watch.c  | 10 ++++++++++
 3 files changed, 56 insertions(+)
fs/nfsd/filecache.c +45 -0
index 1e2b38ed1d35..d1a6f7cf40b2 100644
@@ -894,6 +894,51 @@ __nfsd_file_cache_purge(struct net *net)
 	nfsd_file_dispose_list(&dispose);
 }
 
+/**
+ * nfsd_file_close_sb - close GC-managed cached files for a superblock
+ * @sb: target superblock
+ *
+ * Walk the nfsd_file cache and close out GC-managed entries (those
+ * acquired via nfsd_file_acquire_gc) that belong to @sb. Called during
+ * filesystem unmount after NFSv4 state revocation to release remaining
+ * cached file handles that may be pinning the filesystem.
+ */
+void nfsd_file_close_sb(struct super_block *sb)
+{
+	struct rhashtable_iter iter;
+	struct nfsd_file *nf;
+	unsigned int closed = 0;
+	LIST_HEAD(dispose);
+
+	if (!test_bit(NFSD_FILE_CACHE_UP, &nfsd_file_flags))
+		return;
+
+	rhltable_walk_enter(&nfsd_file_rhltable, &iter);
+	do {
+		rhashtable_walk_start(&iter);
+
+		nf = rhashtable_walk_next(&iter);
+		while (!IS_ERR_OR_NULL(nf)) {
+			if (test_bit(NFSD_FILE_GC, &nf->nf_flags) &&
+			    nf->nf_file &&
+			    file_inode(nf->nf_file)->i_sb == sb) {
+				nfsd_file_cond_queue(nf, &dispose);
+				closed++;
+			}
+			nf = rhashtable_walk_next(&iter);
+		}
+
+		rhashtable_walk_stop(&iter);
+	} while (nf == ERR_PTR(-EAGAIN));
+	rhashtable_walk_exit(&iter);
+
+	nfsd_file_dispose_list(&dispose);
+
+	if (closed)
+		pr_info("nfsd: closed %u cached file handle%s on %s\n",
+			closed, closed == 1 ? "" : "s", sb->s_id);
+}
+
 static struct nfsd_fcache_disposal *
 nfsd_alloc_fcache_disposal(void)
 {
fs/nfsd/filecache.h +1 -0
index b383dbc5b921..66ca7fc6189b 100644
@@ -70,6 +70,7 @@ struct net *nfsd_file_put_local(struct nfsd_file __rcu **nf);
 struct nfsd_file *nfsd_file_get(struct nfsd_file *nf);
 struct file *nfsd_file_file(struct nfsd_file *nf);
 void nfsd_file_close_inode_sync(struct inode *inode);
+void nfsd_file_close_sb(struct super_block *sb);
 void nfsd_file_net_dispose(struct nfsd_net *nn);
 bool nfsd_file_is_cached(struct inode *inode);
 __be32 nfsd_file_acquire_gc(struct svc_rqst *rqstp, struct svc_fh *fhp,
fs/nfsd/sb_watch.c +10 -0
index 8f711956a12e..34e50afe566c 100644
@@ -65,6 +65,7 @@ static void nfsd_sb_revoke_work(struct work_struct *work)
 	/* Errors are logged by lockd; no recovery is possible. */
 	(void)nlmsvc_unlock_all_by_sb(watch->sb);
 	nfsd4_revoke_states(nn, watch->sb);
+	nfsd_file_close_sb(watch->sb);
 
 	pr_info("nfsd: state revocation for %s complete\n", watch->sb->s_id);
 
@@ -257,6 +258,15 @@ void nfsd_sb_watch_shutdown(struct nfsd_net *nn)
 {
 	umount_unregister_notifier(&nn->nfsd_umount_notifier);
 	nfsd_sb_watches_destroy(nn);
+	/*
+	 * Ensure any work items still running complete before shutdown
+	 * proceeds. This handles the case where an unmount notification
+	 * returned early due to signal interruption but the work function
+	 * is still executing nfsd_file_close_sb(). Without this flush,
+	 * nfsd_file_cache_shutdown() could destroy the slab while the
+	 * work function is still disposing file cache entries.
+	 */
+	flush_workqueue(nfsd_sb_watch_wq);
 }
 
 int nfsd_sb_watch_init(void)
Signature · 1 line
2.53.0
maintainer
#29 lore raw
In reply to Chuck Lever
Quoted 25 lines · Chuck Lever

From: Chuck Lever <[email protected]>

When an NFS server exports a filesystem and clients hold NFSv4 state (opens, locks, delegations), unmounting the underlying filesystem fails with EBUSY. The /proc/fs/nfsd/unlock_fs interface exists for administrators to manually revoke state before retrying the unmount, but this approach has significant operational drawbacks.

Manual intervention breaks automation workflows. Containerized NFS servers, orchestration systems, and unattended maintenance scripts cannot reliably unmount exported filesystems without implementing custom logic to detect the failure and invoke unlock_fs. System administrators managing many exports face tedious, error-prone procedures when decommissioning storage.

This series enables the NFS server to detect filesystem unmount events and automatically revoke associated state. The mechanism registers with a new SRCU notifier chain in VFS that fires during mount teardown, after processing stuck children but before fsnotify_vfsmount_delete(), while SB_ACTIVE is still set. When a filesystem is unmounted, all NFSv4 opens, locks, and delegations referencing it are revoked, async COPY operations are cancelled with NFS4ERR_ADMIN_REVOKED sent to clients, NLM locks are released, and cached file handles are closed.

Quoted 5 lines

With automatic revocation, unmount operations complete without administrator intervention once the brief state cleanup finishes. Clients receive immediate notification of state loss through standard NFSv4 error codes, allowing applications to handle the situation appropriately rather than encountering silent failures.

So anyone can force that just with unshare -U -m date? Creates a new namespace, populated by clones of all mounts you see, runs date(1) in that, then exits, with namespace dissolved. At that point all cloned mounts are released, each triggering your notifier chain...