Skip to contentSkip to conversation

FAILED: patch "[PATCH] ceph: fix hanging __ceph_get_caps() with stale mds_wanted" failed to apply to 6.6-stable tree

PATCH New [email protected] 2 participants 4 messages Jul 29 – Aug 7

Subsystems (2)
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

The patch below does not apply to the 6.6-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

To reproduce the conflict and resubmit, you may use the following commands:

git fetch https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ linux-6.6.y
git checkout FETCH_HEAD
git cherry-pick -x 50958bb928bad3bdba9e5d1b7ff4bbadcf6951e6
# <resolve conflicts, build, test, etc.>
git commit -s
git send-email --to '<[email protected]>' --in-reply-to '2026072909-oat-vivacious-f0ee@gregkh' --subject-prefix 'PATCH 6.6.y' 'HEAD^..'

Possible dependencies:

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

From 50958bb928bad3bdba9e5d1b7ff4bbadcf6951e6 Mon Sep 17 00:00:00 2001
From: Max Kellermann <[email protected]>
Date: Mon, 6 Jul 2026 17:06:59 +0200
Subject: [PATCH] ceph: fix hanging __ceph_get_caps() with stale mds_wanted

A reader can hang forever in __ceph_get_caps() when the client no longer holds FILE_RD, but local cap state still says that the capability is already wanted (via mds_wanted).

One way to trigger this is through MDS cap revocation. If another client performs a conflicting operation, the MDS can revoke FILE_RD from the reader; the next read then has to reacquire FILE_RD. If the cap update that should request FILE_RD never reaches the MDS after cap->mds_wanted was raised, the reader is left holding only non-file caps while local mds_wanted still includes the file read caps.

In that state, try_get_cap_refs() sees need <= mds_wanted and returns 0, so __ceph_get_caps() just waits on i_cap_wq. If the cap update that was supposed to request FILE_RD never reaches the MDS after cap->mds_wanted was` raised, no further request is sent and the waiter can sleep indefinitely until unrelated cap traffic happens to wake it up.

The ordering issue is that cap->mds_wanted is updated in __prep_cap() before the CEPH_MSG_CLIENT_CAPS message is actually queued for send. That makes one field serve two different meanings at
once: what this client wants, and what the client believes the MDS already knows it wants.

A proper fix would be to split those states and track whether a cap update is actually in flight or has been observed by the MDS. However, simply moving the cap->mds_wanted assignment later would not be sufficient: queueing the message in the messenger does not guarantee that the MDS processed that specific wanted set, and reconnect or message loss can still invalidate that assumption. Fixing that properly would require a larger rework of the cap state machine.

To allow simpler backports to stable kernels, this patch implements a simpler workaround:

- stop waiting forever in __ceph_get_caps(); after a bounded wait, fall back to the renew path

- make ceph_renew_caps() issue a synchronous OPEN request whenever the inode still does not actually hold the wanted caps, instead of only calling ceph_check_caps()

The extra issued-vs-wanted check in ceph_renew_caps() is necessary because the previous test only checked whether the inode still had any real caps at all. That is not enough after revocation: the client can still hold something like pLs and yet be missing FILE_RD completely. In that case, falling back to ceph_check_caps() is not sufficient, because it still trusts cap->mds_wanted and may resend nothing. By requiring (issued & wanted) == wanted before taking the asynchronous path, the code only uses ceph_check_caps() when the wanted caps are already actually issued. Otherwise, it sends the synchronous OPEN renew.

This preserves the existing asynchronous fast path when the wanted caps are already issued, avoids changing cap-state semantics, and fixes the hang by guaranteeing that a stalled waiter eventually retries through a path that does not rely on the stale mds_wanted state.

[ idryomov: move CEPH_GET_CAPS_WAIT_TIMEOUT from libceph.h to
mds_client.h, formatting ]

Fixes 0a454bdd501a ("ceph: reorganize __send_cap for less spinlock abuse")
fs/ceph/caps.c +15 -2
index 77b23fe51425..d7283fb54cec 100644
@@ -3094,7 +3094,19 @@ int __ceph_get_caps(struct inode *inode, struct ceph_file_info *fi, int need,
 					ret = -ERESTARTSYS;
 					break;
 				}
-				wait_woken(&wait, TASK_INTERRUPTIBLE, MAX_SCHEDULE_TIMEOUT);
+
+				/*
+				 * If a cap update is lost after
+				 * mds_wanted was raised, waiting
+				 * forever will never make progress.
+				 * Retry the renew path periodically
+				 * so we can resend synchronously.
+				 */
+				if (!wait_woken(&wait, TASK_INTERRUPTIBLE,
+						CEPH_GET_CAPS_WAIT_TIMEOUT)) {
+					ret = -EUCLEAN;
+					break;
+				}
 			}
 
 			remove_wait_queue(&ci->i_cap_wq, &wait);
@@ -3128,7 +3140,8 @@ int __ceph_get_caps(struct inode *inode, struct ceph_file_info *fi, int need,
 				continue;
 			}
 			if (ret == -EUCLEAN) {
-				/* session was killed, try renew caps */
+				/* session was killed or a waited cap
+				 * request needs a retry */
 				ret = ceph_renew_caps(inode, flags);
 				if (ret == 0)
 					continue;
fs/ceph/file.c +5 -4
index 71161f2b2151..a4a2a4b6a027 100644
@@ -314,7 +314,7 @@ static int ceph_init_file(struct inode *inode, struct file *file, int fmode)
 }
 
 /*
- * try renew caps after session gets killed.
+ * Retry cap acquisition after a stale session or a lost cap update.
  */
 int ceph_renew_caps(struct inode *inode, int fmode)
 {
@@ -322,14 +322,15 @@ int ceph_renew_caps(struct inode *inode, int fmode)
 	struct ceph_client *cl = mdsc->fsc->client;
 	struct ceph_inode_info *ci = ceph_inode(inode);
 	struct ceph_mds_request *req;
-	int err, flags, wanted;
+	int err, flags, wanted, issued;
 
 	spin_lock(&ci->i_ceph_lock);
 	__ceph_touch_fmode(ci, mdsc, fmode);
 	wanted = __ceph_caps_file_wanted(ci);
+	issued = __ceph_caps_issued(ci, NULL);
 	if (__ceph_is_any_real_caps(ci) &&
-	    (!(wanted & CEPH_CAP_ANY_WR) || ci->i_auth_cap)) {
-		int issued = __ceph_caps_issued(ci, NULL);
+	    (!(wanted & CEPH_CAP_ANY_WR) || ci->i_auth_cap) &&
+	    (issued & wanted) == wanted) {
 		spin_unlock(&ci->i_ceph_lock);
 		doutc(cl, "%p %llx.%llx want %s issued %s updating mds_wanted\n",
 		      inode, ceph_vinop(inode), ceph_cap_string(wanted),
fs/ceph/mds_client.h +1 -0
index 731d6ad04956..0ece4c9e3529 100644
@@ -77,6 +77,7 @@ struct ceph_fs_client;
 struct ceph_cap;
 
 #define MDS_AUTH_UID_ANY -1
+#define CEPH_GET_CAPS_WAIT_TIMEOUT (5 * HZ)
 #define CEPH_CAP_FLUSH_WAIT_TIMEOUT_SEC 60
 #define CEPH_CAP_FLUSH_MAX_DUMP_ENTRIES 5
 #define CEPH_CAP_FLUSH_MAX_DUMP_ITERS 5
maintainer
#1 lore raw

PATCH 1/3 libceph: add doutc and *_client debug macros support

From: Xiubo Li <[email protected]>

[ Upstream commit 5c5f0d2b5f92c47baf82b9b211e27edd7d195158 ]

This will help print the fsid and client's global_id in debug logs, and also print the function names.

[ idryomov: %lld -> %llu, leading space for doutc(), don't include
__func__ in pr_*() variants ]

Stable-dep-of: 50958bb928ba ("ceph: fix hanging __ceph_get_caps() with stale mds_wanted")

---
 include/linux/ceph/ceph_debug.h | 38 +++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
include/linux/ceph/ceph_debug.h +38 -0
index d5a5da838cafa..11a92a946016e 100644
@@ -19,12 +19,25 @@
 	pr_debug("%.*s %12.12s:%-4d : " fmt,				\
 		 8 - (int)sizeof(KBUILD_MODNAME), "    ",		\
 		 kbasename(__FILE__), __LINE__, ##__VA_ARGS__)
+#  define doutc(client, fmt, ...)					\
+	pr_debug("%.*s %12.12s:%-4d : [%pU %llu] " fmt,			\
+		 8 - (int)sizeof(KBUILD_MODNAME), "    ",		\
+		 kbasename(__FILE__), __LINE__,				\
+		 &client->fsid, client->monc.auth->global_id,		\
+		 ##__VA_ARGS__)
 # else
 /* faux printk call just to see any compiler warnings. */
 #  define dout(fmt, ...)	do {				\
 		if (0)						\
 			printk(KERN_DEBUG fmt, ##__VA_ARGS__);	\
 	} while (0)
+#  define doutc(client, fmt, ...)	do {			\
+		if (0)						\
+			printk(KERN_DEBUG "[%pU %llu] " fmt,	\
+			&client->fsid,				\
+			client->monc.auth->global_id,		\
+			##__VA_ARGS__);				\
+		} while (0)
 # endif
 
 #else
@@ -33,7 +46,32 @@
  * or, just wrap pr_debug
  */
 # define dout(fmt, ...)	pr_debug(" " fmt, ##__VA_ARGS__)
+# define doutc(client, fmt, ...)					\
+	pr_debug(" [%pU %llu] %s: " fmt, &client->fsid,			\
+		 client->monc.auth->global_id, __func__, ##__VA_ARGS__)
 
 #endif
 
+#define pr_notice_client(client, fmt, ...)				\
+	pr_notice("[%pU %llu]: " fmt, &client->fsid,			\
+		  client->monc.auth->global_id, ##__VA_ARGS__)
+#define pr_info_client(client, fmt, ...)				\
+	pr_info("[%pU %llu]: " fmt, &client->fsid,			\
+		client->monc.auth->global_id, ##__VA_ARGS__)
+#define pr_warn_client(client, fmt, ...)				\
+	pr_warn("[%pU %llu]: " fmt, &client->fsid,			\
+		client->monc.auth->global_id, ##__VA_ARGS__)
+#define pr_warn_once_client(client, fmt, ...)				\
+	pr_warn_once("[%pU %llu]: " fmt, &client->fsid,			\
+		     client->monc.auth->global_id, ##__VA_ARGS__)
+#define pr_err_client(client, fmt, ...)					\
+	pr_err("[%pU %llu]: " fmt, &client->fsid,			\
+	       client->monc.auth->global_id, ##__VA_ARGS__)
+#define pr_warn_ratelimited_client(client, fmt, ...)			\
+	pr_warn_ratelimited("[%pU %llu]: " fmt, &client->fsid,		\
+			    client->monc.auth->global_id, ##__VA_ARGS__)
+#define pr_err_ratelimited_client(client, fmt, ...)			\
+	pr_err_ratelimited("[%pU %llu]: " fmt, &client->fsid,		\
+			   client->monc.auth->global_id, ##__VA_ARGS__)
+
 #endif
Signature · 1 line
2.53.0
maintainer
#2 lore raw

PATCH 2/3 ceph: print cluster fsid and client global_id in all debug logs

From: Xiubo Li <[email protected]>

[ Upstream commit 38d46409c4639a1d659ebfa70e27a8bed6b8ee1d ]

Multiple CephFS mounts on a host is increasingly common so disambiguating messages like this is necessary and will make it easier to debug issues.

At the same this will improve the debug logs to make them easier to troubleshooting issues, such as print the ino# instead only printing the memory addresses of the corresponding inodes and print the dentry names instead of the corresponding memory addresses for the dentry,etc.

Stable-dep-of: 50958bb928ba ("ceph: fix hanging __ceph_get_caps() with stale mds_wanted")

---
 fs/ceph/acl.c        |   6 +-
 fs/ceph/addr.c       | 279 +++++++++--------
 fs/ceph/caps.c       | 710 +++++++++++++++++++++++++------------------
 fs/ceph/crypto.c     |  39 ++-
 fs/ceph/debugfs.c    |   6 +-
 fs/ceph/dir.c        | 218 +++++++------
 fs/ceph/export.c     |  39 +--
 fs/ceph/file.c       | 245 ++++++++-------
 fs/ceph/inode.c      | 485 ++++++++++++++++-------------
 fs/ceph/ioctl.c      |  13 +-
 fs/ceph/locks.c      |  57 ++--
 fs/ceph/mds_client.c | 554 ++++++++++++++++++---------------
 fs/ceph/mdsmap.c     |  24 +-
 fs/ceph/metric.c     |   5 +-
 fs/ceph/quota.c      |  29 +-
 fs/ceph/snap.c       | 174 ++++++-----
 fs/ceph/super.c      |  70 +++--
 fs/ceph/super.h      |   6 +
 fs/ceph/xattr.c      |  96 +++---
 19 files changed, 1745 insertions(+), 1310 deletions(-)
fs/ceph/acl.c +4 -2
index c53a1d2206225..e958419613972 100644
@@ -15,6 +15,7 @@
 #include <linux/slab.h>
 
 #include "super.h"
+#include "mds_client.h"
 
 static inline void ceph_set_cached_acl(struct inode *inode,
 					int type, struct posix_acl *acl)
@@ -31,6 +32,7 @@ static inline void ceph_set_cached_acl(struct inode *inode,
 
 struct posix_acl *ceph_get_acl(struct inode *inode, int type, bool rcu)
 {
+	struct ceph_client *cl = ceph_inode_to_client(inode);
 	int size;
 	unsigned int retry_cnt = 0;
 	const char *name;
@@ -72,8 +74,8 @@ struct posix_acl *ceph_get_acl(struct inode *inode, int type, bool rcu)
 	} else if (size == -ENODATA || size == 0) {
 		acl = NULL;
 	} else {
-		pr_err_ratelimited("get acl %llx.%llx failed, err=%d\n",
-				   ceph_vinop(inode), size);
+		pr_err_ratelimited_client(cl, "%llx.%llx failed, err=%d\n",
+					  ceph_vinop(inode), size);
 		acl = ERR_PTR(-EIO);
 	}
 
fs/ceph/addr.c +153 -126

757 lines not on this page · raw message

fs/ceph/caps.c +410 -300

1,717 lines not on this page · raw message

fs/ceph/crypto.c +26 -13
index 08c3856107316..d692ebfddedbc 100644
@@ -211,6 +211,7 @@ void ceph_fscrypt_as_ctx_to_req(struct ceph_mds_request *req,
 static struct inode *parse_longname(const struct inode *parent,
 				    const char *name, int *name_len)
 {
+	struct ceph_client *cl = ceph_inode_to_client(parent);
 	struct inode *dir = NULL;
 	struct ceph_vino vino = { .snap = CEPH_NOSNAP };
 	char *inode_number;
@@ -222,12 +223,12 @@ static struct inode *parse_longname(const struct inode *parent,
 	name++;
 	name_end = strrchr(name, '_');
 	if (!name_end) {
-		dout("Failed to parse long snapshot name: %s\n", name);
+		doutc(cl, "failed to parse long snapshot name: %s\n", name);
 		return ERR_PTR(-EIO);
 	}
 	*name_len = (name_end - name);
 	if (*name_len <= 0) {
-		pr_err("Failed to parse long snapshot name\n");
+		pr_err_client(cl, "failed to parse long snapshot name\n");
 		return ERR_PTR(-EIO);
 	}
 
@@ -239,7 +240,7 @@ static struct inode *parse_longname(const struct inode *parent,
 		return ERR_PTR(-ENOMEM);
 	ret = kstrtou64(inode_number, 10, &vino.ino);
 	if (ret) {
-		dout("Failed to parse inode number: %s\n", name);
+		doutc(cl, "failed to parse inode number: %s\n", name);
 		dir = ERR_PTR(ret);
 		goto out;
 	}
@@ -250,7 +251,7 @@ static struct inode *parse_longname(const struct inode *parent,
 		/* This can happen if we're not mounting cephfs on the root */
 		dir = ceph_get_inode(parent->i_sb, vino, NULL);
 		if (IS_ERR(dir))
-			dout("Can't find inode %s (%s)\n", inode_number, name);
+			doutc(cl, "can't find inode %s (%s)\n", inode_number, name);
 	}
 
 out:
@@ -261,6 +262,7 @@ static struct inode *parse_longname(const struct inode *parent,
 int ceph_encode_encrypted_dname(struct inode *parent, struct qstr *d_name,
 				char *buf)
 {
+	struct ceph_client *cl = ceph_inode_to_client(parent);
 	struct inode *dir = parent;
 	struct qstr iname;
 	u32 len;
@@ -329,7 +331,7 @@ int ceph_encode_encrypted_dname(struct inode *parent, struct qstr *d_name,
 
 	/* base64 encode the encrypted name */
 	elen = ceph_base64_encode(cryptbuf, len, buf);
-	dout("base64-encoded ciphertext name = %.*s\n", elen, buf);
+	doutc(cl, "base64-encoded ciphertext name = %.*s\n", elen, buf);
 
 	/* To understand the 240 limit, see CEPH_NOHASH_NAME_MAX comments */
 	WARN_ON(elen > 240);
@@ -504,7 +506,10 @@ int ceph_fscrypt_decrypt_block_inplace(const struct inode *inode,
 				  struct page *page, unsigned int len,
 				  unsigned int offs, u64 lblk_num)
 {
-	dout("%s: len %u offs %u blk %llu\n", __func__, len, offs, lblk_num);
+	struct ceph_client *cl = ceph_inode_to_client(inode);
+
+	doutc(cl, "%p %llx.%llx len %u offs %u blk %llu\n", inode,
+	      ceph_vinop(inode), len, offs, lblk_num);
 	return fscrypt_decrypt_block_inplace(inode, page, len, offs, lblk_num);
 }
 
@@ -513,7 +518,10 @@ int ceph_fscrypt_encrypt_block_inplace(const struct inode *inode,
 				  unsigned int offs, u64 lblk_num,
 				  gfp_t gfp_flags)
 {
-	dout("%s: len %u offs %u blk %llu\n", __func__, len, offs, lblk_num);
+	struct ceph_client *cl = ceph_inode_to_client(inode);
+
+	doutc(cl, "%p %llx.%llx len %u offs %u blk %llu\n", inode,
+	      ceph_vinop(inode), len, offs, lblk_num);
 	return fscrypt_encrypt_block_inplace(inode, page, len, offs, lblk_num,
 					     gfp_flags);
 }
@@ -582,6 +590,7 @@ int ceph_fscrypt_decrypt_extents(struct inode *inode, struct page **page,
 				 u64 off, struct ceph_sparse_extent *map,
 				 u32 ext_cnt)
 {
+	struct ceph_client *cl = ceph_inode_to_client(inode);
 	int i, ret = 0;
 	struct ceph_inode_info *ci = ceph_inode(inode);
 	u64 objno, objoff;
@@ -589,7 +598,8 @@ int ceph_fscrypt_decrypt_extents(struct inode *inode, struct page **page,
 
 	/* Nothing to do for empty array */
 	if (ext_cnt == 0) {
-		dout("%s: empty array, ret 0\n", __func__);
+		doutc(cl, "%p %llx.%llx empty array, ret 0\n", inode,
+		      ceph_vinop(inode));
 		return 0;
 	}
 
@@ -603,14 +613,17 @@ int ceph_fscrypt_decrypt_extents(struct inode *inode, struct page **page,
 		int fret;
 
 		if ((ext->off | ext->len) & ~CEPH_FSCRYPT_BLOCK_MASK) {
-			pr_warn("%s: bad encrypted sparse extent idx %d off %llx len %llx\n",
-				__func__, i, ext->off, ext->len);
+			pr_warn_client(cl,
+				"%p %llx.%llx bad encrypted sparse extent "
+				"idx %d off %llx len %llx\n",
+				inode, ceph_vinop(inode), i, ext->off,
+				ext->len);
 			return -EIO;
 		}
 		fret = ceph_fscrypt_decrypt_pages(inode, &page[pgidx],
 						 off + pgsoff, ext->len);
-		dout("%s: [%d] 0x%llx~0x%llx fret %d\n", __func__, i,
-				ext->off, ext->len, fret);
+		doutc(cl, "%p %llx.%llx [%d] 0x%llx~0x%llx fret %d\n", inode,
+		      ceph_vinop(inode), i, ext->off, ext->len, fret);
 		if (fret < 0) {
 			if (ret == 0)
 				ret = fret;
@@ -618,7 +631,7 @@ int ceph_fscrypt_decrypt_extents(struct inode *inode, struct page **page,
 		}
 		ret = pgsoff + fret;
 	}
-	dout("%s: ret %d\n", __func__, ret);
+	doutc(cl, "ret %d\n", ret);
 	return ret;
 }
 
fs/ceph/debugfs.c +4 -2
index 2f1e7498cd745..24c08078f5aa3 100644
@@ -398,7 +398,7 @@ DEFINE_SIMPLE_ATTRIBUTE(congestion_kb_fops, congestion_kb_get,
 
 void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc)
 {
-	dout("ceph_fs_debugfs_cleanup\n");
+	doutc(fsc->client, "begin\n");
 	debugfs_remove(fsc->debugfs_bdi);
 	debugfs_remove(fsc->debugfs_congestion_kb);
 	debugfs_remove(fsc->debugfs_mdsmap);
@@ -407,13 +407,14 @@ void ceph_fs_debugfs_cleanup(struct ceph_fs_client *fsc)
 	debugfs_remove(fsc->debugfs_status);
 	debugfs_remove(fsc->debugfs_mdsc);
 	debugfs_remove_recursive(fsc->debugfs_metrics_dir);
+	doutc(fsc->client, "done\n");
 }
 
 void ceph_fs_debugfs_init(struct ceph_fs_client *fsc)
 {
 	char name[100];
 
-	dout("ceph_fs_debugfs_init\n");
+	doutc(fsc->client, "begin\n");
 	fsc->debugfs_congestion_kb =
 		debugfs_create_file("writeback_congestion_kb",
 				    0600,
@@ -469,6 +470,7 @@ void ceph_fs_debugfs_init(struct ceph_fs_client *fsc)
 			    &metrics_size_fops);
 	debugfs_create_file("caps", 0400, fsc->debugfs_metrics_dir, fsc,
 			    &metrics_caps_fops);
+	doutc(fsc->client, "done\n");
 }
 
 
fs/ceph/dir.c +132 -86

353 lines not on this page · raw message

fs/ceph/export.c +22 -17
index 52c4daf2447d3..726af69d4d62c 100644

Lines of this file are not shown. View the raw message.

fs/ceph/file.c +141 -104

245 lines not on this page · raw message

fs/ceph/inode.c +267 -218

485 lines not on this page · raw message

fs/ceph/ioctl.c +9 -4
index 3f617146e4ad3..e861de3c79b9e 100644

Lines of this file are not shown. View the raw message.

fs/ceph/locks.c +33 -24

57 lines not on this page · raw message

fs/ceph/mds_client.c +313 -241

554 lines not on this page · raw message

fs/ceph/mdsmap.c +12 -12
index 66afb18df76b2..a476d5e6d39f4 100644

Lines of this file are not shown. View the raw message.

fs/ceph/metric.c +3 -2
index 6d3584f16f9a4..871c1090e5201 100644

Lines of this file are not shown. View the raw message.

fs/ceph/quota.c +18 -11
index ca4932e6f71bf..06ee397e0c3a6 100644

Lines of this file are not shown. View the raw message.

fs/ceph/snap.c +95 -79

174 lines not on this page · raw message

fs/ceph/super.c +40 -30

70 lines not on this page · raw message

fs/ceph/super.h +6 -0
index 5903e3fb6d750..e31ecbed609de 100644

Lines of this file are not shown. View the raw message.

fs/ceph/xattr.c +57 -39

96 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.53.0

Part of the diff is not shown here. View the raw message.

maintainer
#3 lore raw

PATCH 3/3 ceph: fix hanging __ceph_get_caps() with stale mds_wanted

In reply to Sasha Levin

From: Max Kellermann <[email protected]>

[ Upstream commit 50958bb928bad3bdba9e5d1b7ff4bbadcf6951e6 ]

A reader can hang forever in __ceph_get_caps() when the client no longer holds FILE_RD, but local cap state still says that the capability is already wanted (via mds_wanted).

One way to trigger this is through MDS cap revocation. If another client performs a conflicting operation, the MDS can revoke FILE_RD from the reader; the next read then has to reacquire FILE_RD. If the cap update that should request FILE_RD never reaches the MDS after cap->mds_wanted was raised, the reader is left holding only non-file caps while local mds_wanted still includes the file read caps.

In that state, try_get_cap_refs() sees need <= mds_wanted and returns 0, so __ceph_get_caps() just waits on i_cap_wq. If the cap update that was supposed to request FILE_RD never reaches the MDS after cap->mds_wanted was` raised, no further request is sent and the waiter can sleep indefinitely until unrelated cap traffic happens to wake it up.

The ordering issue is that cap->mds_wanted is updated in __prep_cap() before the CEPH_MSG_CLIENT_CAPS message is actually queued for send. That makes one field serve two different meanings at
once: what this client wants, and what the client believes the MDS already knows it wants.

A proper fix would be to split those states and track whether a cap update is actually in flight or has been observed by the MDS. However, simply moving the cap->mds_wanted assignment later would not be sufficient: queueing the message in the messenger does not guarantee that the MDS processed that specific wanted set, and reconnect or message loss can still invalidate that assumption. Fixing that properly would require a larger rework of the cap state machine.

To allow simpler backports to stable kernels, this patch implements a simpler workaround:

- stop waiting forever in __ceph_get_caps(); after a bounded wait, fall back to the renew path

- make ceph_renew_caps() issue a synchronous OPEN request whenever the inode still does not actually hold the wanted caps, instead of only calling ceph_check_caps()

The extra issued-vs-wanted check in ceph_renew_caps() is necessary because the previous test only checked whether the inode still had any real caps at all. That is not enough after revocation: the client can still hold something like pLs and yet be missing FILE_RD completely. In that case, falling back to ceph_check_caps() is not sufficient, because it still trusts cap->mds_wanted and may resend nothing. By requiring (issued & wanted) == wanted before taking the asynchronous path, the code only uses ceph_check_caps() when the wanted caps are already actually issued. Otherwise, it sends the synchronous OPEN renew.

This preserves the existing asynchronous fast path when the wanted caps are already issued, avoids changing cap-state semantics, and fixes the hang by guaranteeing that a stalled waiter eventually retries through a path that does not rely on the stale mds_wanted state.

[ idryomov: move CEPH_GET_CAPS_WAIT_TIMEOUT from libceph.h to
mds_client.h, formatting ]

Fixes 0a454bdd501a ("ceph: reorganize __send_cap for less spinlock abuse")
---
 fs/ceph/caps.c       | 17 +++++++++++++++--
 fs/ceph/file.c       |  9 +++++----
 fs/ceph/mds_client.h |  2 ++
 3 files changed, 22 insertions(+), 6 deletions(-)
fs/ceph/caps.c +15 -2
index 941799cb53319..dd667358e1fa7 100644
@@ -3091,7 +3091,19 @@ int __ceph_get_caps(struct inode *inode, struct ceph_file_info *fi, int need,
 					ret = -ERESTARTSYS;
 					break;
 				}
-				wait_woken(&wait, TASK_INTERRUPTIBLE, MAX_SCHEDULE_TIMEOUT);
+
+				/*
+				 * If a cap update is lost after
+				 * mds_wanted was raised, waiting
+				 * forever will never make progress.
+				 * Retry the renew path periodically
+				 * so we can resend synchronously.
+				 */
+				if (!wait_woken(&wait, TASK_INTERRUPTIBLE,
+						CEPH_GET_CAPS_WAIT_TIMEOUT)) {
+					ret = -EUCLEAN;
+					break;
+				}
 			}
 
 			remove_wait_queue(&ci->i_cap_wq, &wait);
@@ -3123,7 +3135,8 @@ int __ceph_get_caps(struct inode *inode, struct ceph_file_info *fi, int need,
 				continue;
 			}
 			if (ret == -EUCLEAN) {
-				/* session was killed, try renew caps */
+				/* session was killed or a waited cap
+				 * request needs a retry */
 				ret = ceph_renew_caps(inode, flags);
 				if (ret == 0)
 					continue;
fs/ceph/file.c +5 -4
index bd74e8ffe6af7..cecaabd7388c0 100644
@@ -294,7 +294,7 @@ static int ceph_init_file(struct inode *inode, struct file *file, int fmode)
 }
 
 /*
- * try renew caps after session gets killed.
+ * Retry cap acquisition after a stale session or a lost cap update.
  */
 int ceph_renew_caps(struct inode *inode, int fmode)
 {
@@ -302,14 +302,15 @@ int ceph_renew_caps(struct inode *inode, int fmode)
 	struct ceph_client *cl = mdsc->fsc->client;
 	struct ceph_inode_info *ci = ceph_inode(inode);
 	struct ceph_mds_request *req;
-	int err, flags, wanted;
+	int err, flags, wanted, issued;
 
 	spin_lock(&ci->i_ceph_lock);
 	__ceph_touch_fmode(ci, mdsc, fmode);
 	wanted = __ceph_caps_file_wanted(ci);
+	issued = __ceph_caps_issued(ci, NULL);
 	if (__ceph_is_any_real_caps(ci) &&
-	    (!(wanted & CEPH_CAP_ANY_WR) || ci->i_auth_cap)) {
-		int issued = __ceph_caps_issued(ci, NULL);
+	    (!(wanted & CEPH_CAP_ANY_WR) || ci->i_auth_cap) &&
+	    (issued & wanted) == wanted) {
 		spin_unlock(&ci->i_ceph_lock);
 		doutc(cl, "%p %llx.%llx want %s issued %s updating mds_wanted\n",
 		      inode, ceph_vinop(inode), ceph_cap_string(wanted),
fs/ceph/mds_client.h +2 -0
index d930eb79dc380..4237c6615a3f4 100644
@@ -68,6 +68,8 @@ enum ceph_feature_type {
 struct ceph_fs_client;
 struct ceph_cap;
 
+#define CEPH_GET_CAPS_WAIT_TIMEOUT (5 * HZ)
+
 /*
  * parsed info about a single inode.  pointers are into the encoded
  * on-wire structures within the mds reply message payload.
Signature · 1 line
2.53.0