Skip to contentSkip to conversation

Improve the scalability of NFSD's classic DRC

PATCH Superseded Chuck Lever 3 participants 18 messages Sep 10 – Sep 12

Subsystems (4)

v3 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Chuck Lever

author maintainer
#0 lore raw

v3 cover Improve the scalability of NFSD's classic DRC

A completed DRC entry currently stays in its bucket for 120 seconds whether or not the client already holds the reply. On a busy server those entries lengthen every bucket walk, and under pressure they crowd out entries a retransmit could still hit.

This series modifies the DRC to retire an entry once there is reason to believe its reply was delivered.

1. The transport reports delivery outright where it can. TCP now reports when snd_una passes the reply's sequence number, and RDMA reports on each Send completion.
2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).

UDP continues to use a traditional time-based eviction mechanism.

The first mechanism is the more reliable of the two, and the backup mechanism is more of an informed guess. A pipelined client sends its next request before the previous reply lands, and an entry evicted in that window is lost if the connection then drops and the client retransmits. In the current NFSD code, memory pressure and RC_EXPIRE already open this same window. For instance, on a server with fast networking and storage, pressure eviction already evicts entries far younger than 120 seconds.

Two DRC capacity guards that were sized for a cache full of stale replies can be removed, now that reply delivery reports keep the DRC small. The commit messages for those patches explain the rationale in detail.

v2 of this series (implied ACK only) was profiled with "perf record -e cycles -e cpu-clock -e LLC-load-misses -e branch-misses" during an NFSv3/RDMA 4KB random-write workload. v3 has not been re-profiled. nfsd_cache_lookup overhead dropped from 1.53% to 0.76% of CPU cycles during this test. The rb-tree operations (rb_erase, rb_insert_color) that dominated LLC cache misses fell from a combined 13.2% to 1.1% of all LLC-load-misses, because shorter-lived entries keep the per-bucket trees small.

---
Changes in v3:
- Add the reply-acknowledged callback and its TCP and RDMA reporters
ahead of implied ACK, which now defers to a pending report.
- Split the prune-loop restructure into its own patch.
- Move the eviction tracepoints ahead of the ack patches; the
implied-ACK event now lands with implied ACK.
- Keep err local to the pmap_register block in svc_setup_socket()
(per Jeff's review).
- Never move xpt_last_recv backwards when nfsd threads race.
- Fold the XPT_ORDERED patch into its consumer (per Jeff's review).
- State the re-execution exposure of implied-ACK eviction instead of
calling the DRC advisory (per Jeff's review).
- New patch: remove the DRC request checksum and payload_misses stat.
- New patch: remove the 256k-entry cap on the DRC size.
- Link to v2: https://patch.msgid.link/[email protected]

Changes in v2:
- Print the DRC eviction tracepoints' age field as unsigned.
- Link to v1: https://patch.msgid.link/[email protected]

---
Chuck Lever (12):
SUNRPC: Assign a unique identifier to each svc_xprt
NFSD: Track transport in DRC entries
NFSD: Prepare bucket pruning for out-of-order eviction
NFSD: Add tracepoints for DRC entry eviction
NFSD: Record DRC population in lookup tracepoints
NFSD: Add reply-acknowledged callback infrastructure
SUNRPC: Add TCP sequence-number ACK tracking for reply delivery
svcrdma: Fire reply-acknowledged callback on Send completion
SUNRPC: Record last-request timestamp on svc_xprt
NFSD: Evict unacknowledged DRC entries via implied ACK
NFSD: Remove DRC checksum and payload_misses stat
NFSD: Remove hard cap on duplicate reply cache size

 .../ABI/testing/procfs-nfsd-reply_cache_stats      |  11 +-
 fs/nfsd/cache.h                                    |  20 +-
 fs/nfsd/netns.h                                    |   2 -
 fs/nfsd/nfscache.c                                 | 310 +++++++++++++--------
 fs/nfsd/nfssvc.c                                   |  20 +-
 fs/nfsd/stats.h                                    |   5 -
 fs/nfsd/trace.h                                    |  81 +++++-
 include/linux/sunrpc/svc.h                         |  51 ++++
 include/linux/sunrpc/svc_rdma.h                    |   1 +
 include/linux/sunrpc/svc_xprt.h                    |   6 +-
 include/linux/sunrpc/svcsock.h                     |  10 +
 include/trace/events/sunrpc.h                      |  10 +-
 net/sunrpc/netns.h                                 |   4 +
 net/sunrpc/sunrpc_syms.c                           |   2 +
 net/sunrpc/svc.c                                   |   1 +
 net/sunrpc/svc_xprt.c                              |  53 +++-
 net/sunrpc/svcsock.c                               | 119 +++++++-
 net/sunrpc/xprtrdma/svc_rdma_sendto.c              |  14 +
 net/sunrpc/xprtrdma/svc_rdma_transport.c           |   7 +-
 19 files changed, 541 insertions(+), 186 deletions(-)

---
base-commit: abf2077ee32058e60d3f49ecab265d3c4bd953d9
change-id: 20260325-duplicate-reply-cache-f0fe7c7b740c

Best regards,

Signature · 1 line
Chuck Lever <[email protected]>
author maintainer
#1 lore raw

PATCH v3 1/12 SUNRPC: Assign a unique identifier to each svc_xprt

A consumer that associates state with a transport without holding a reference cannot detect ABA collisions: once a transport is freed, SLUB may hand out a new svc_xprt at the same address.

Allocate a per-netns identifier for each transport in svc_xprt_init() with xa_alloc_cyclic(), which delays reuse of an identifier after its transport is freed. svc_xprt_init() now returns a boolean, and its callers unwind when allocation fails. The NFSD duplicate reply cache is the first consumer.

---
 include/linux/sunrpc/svc_xprt.h          |  3 ++-
 include/trace/events/sunrpc.h            |  8 +++++--
 net/sunrpc/netns.h                       |  4 ++++
 net/sunrpc/sunrpc_syms.c                 |  2 ++
 net/sunrpc/svc_xprt.c                    | 39 ++++++++++++++++++++++++++++----
 net/sunrpc/svcsock.c                     | 33 +++++++++++++++++++--------
 net/sunrpc/xprtrdma/svc_rdma_transport.c |  5 +++-
 7 files changed, 75 insertions(+), 19 deletions(-)
include/linux/sunrpc/svc_xprt.h +2 -1
index 2af222f3ea2c..c62f789e2900 100644
@@ -56,6 +56,7 @@ struct svc_xprt {
 	struct svc_xprt_class	*xpt_class;
 	const struct svc_xprt_ops *xpt_ops;
 	struct kref		xpt_ref;
+	unsigned int		xpt_id;
 	ktime_t			xpt_qtime;
 	struct list_head	xpt_list;
 	struct lwq_node		xpt_ready;
@@ -162,7 +163,7 @@ static inline bool svc_xprt_is_dead(const struct svc_xprt *xprt)
 
 int	svc_reg_xprt_class(struct svc_xprt_class *);
 void	svc_unreg_xprt_class(struct svc_xprt_class *);
-void	svc_xprt_init(struct net *, struct svc_xprt_class *, struct svc_xprt *,
+bool	svc_xprt_init(struct net *, struct svc_xprt_class *, struct svc_xprt *,
 		      struct svc_serv *);
 int	svc_xprt_create_from_sa(struct svc_serv *serv, const char *xprt_name,
 				struct net *net, struct sockaddr *sap,
include/trace/events/sunrpc.h +6 -2
index ff855197880d..180346e520ff 100644
@@ -1986,7 +1986,8 @@ TRACE_EVENT(svc_xprt_create_err,
 		__sockaddr(server, (x)->xpt_locallen) \
 		__sockaddr(client, (x)->xpt_remotelen) \
 		__field(unsigned long, flags) \
-		__field(unsigned int, netns_ino)
+		__field(unsigned int, netns_ino) \
+		__field(unsigned int, xpt_id)
 
 #define SVC_XPRT_ENDPOINT_ASSIGNMENTS(x) \
 		do { \
@@ -1996,13 +1997,15 @@ TRACE_EVENT(svc_xprt_create_err,
 					  (x)->xpt_remotelen); \
 			__entry->flags = (x)->xpt_flags; \
 			__entry->netns_ino = (x)->xpt_net->ns.inum; \
+			__entry->xpt_id = (x)->xpt_id; \
 		} while (0)
 
 #define SVC_XPRT_ENDPOINT_FORMAT \
-		"server=%pISpc client=%pISpc flags=%s"
+		"server=%pISpc client=%pISpc xpt_id=%u flags=%s"
 
 #define SVC_XPRT_ENDPOINT_VARARGS \
 		__get_sockaddr(server), __get_sockaddr(client), \
+		__entry->xpt_id, \
 		show_svc_xprt_flags(__entry->flags)
 
 TRACE_EVENT(svc_xprt_enqueue,
@@ -2024,6 +2027,7 @@ TRACE_EVENT(svc_xprt_enqueue,
 				  xprt->xpt_remotelen);
 		__entry->flags = flags;
 		__entry->netns_ino = xprt->xpt_net->ns.inum;
+		__entry->xpt_id = xprt->xpt_id;
 	),
 
 	TP_printk(SVC_XPRT_ENDPOINT_FORMAT, SVC_XPRT_ENDPOINT_VARARGS)
net/sunrpc/netns.h +4 -0
index 4efb5f28d881..9523d860be27 100644
@@ -2,6 +2,7 @@
 #ifndef __SUNRPC_NETNS_H__
 #define __SUNRPC_NETNS_H__
 
+#include <linux/xarray.h>
 #include <net/net_namespace.h>
 #include <net/netns/generic.h>
 
@@ -34,6 +35,9 @@ struct sunrpc_net {
 	atomic_t pipe_users;
 	struct proc_dir_entry *use_gssp_proc;
 	struct proc_dir_entry *gss_krb5_enctypes;
+
+	struct xarray	svc_xprt_ids;
+	u32		svc_xprt_id_next;
 };
 
 extern unsigned int sunrpc_net_id;
net/sunrpc/sunrpc_syms.c +2 -0
index 1a3884a0376a..355404d541d4 100644
@@ -58,6 +58,7 @@ static __net_init int sunrpc_init_net(struct net *net)
 	spin_lock_init(&sn->rpc_client_lock);
 	spin_lock_init(&sn->rpcb_clnt_lock);
 	mutex_init(&sn->gssp_lock);
+	xa_init_flags(&sn->svc_xprt_ids, XA_FLAGS_ALLOC1);
 	return 0;
 
 err_pipefs:
@@ -74,6 +75,7 @@ static __net_exit void sunrpc_exit_net(struct net *net)
 {
 	struct sunrpc_net *sn = net_generic(net, sunrpc_net_id);
 
+	xa_destroy(&sn->svc_xprt_ids);
 	rpc_pipefs_exit_net(net);
 	unix_gid_cache_destroy(net);
 	ip_map_cache_destroy(net);
net/sunrpc/svc_xprt.c +34 -5
index d5634dd6d6cc..7e8ef4832421 100644
@@ -21,6 +21,8 @@
 #include <linux/netdevice.h>
 #include <trace/events/sunrpc.h>
 
+#include "netns.h"
+
 #define RPCDBG_FACILITY	RPCDBG_SVCXPRT
 
 static unsigned int svc_rpc_per_connection_limit __read_mostly;
@@ -169,7 +171,10 @@ static void svc_xprt_free(struct kref *kref)
 {
 	struct svc_xprt *xprt =
 		container_of(kref, struct svc_xprt, xpt_ref);
+	struct sunrpc_net *sn = net_generic(xprt->xpt_net, sunrpc_net_id);
 	struct module *owner = xprt->xpt_class->xcl_owner;
+
+	xa_erase(&sn->svc_xprt_ids, xprt->xpt_id);
 	if (test_bit(XPT_CACHE_AUTH, &xprt->xpt_flags))
 		svcauth_unix_info_release(xprt);
 	put_cred(xprt->xpt_cred);
@@ -190,13 +195,28 @@ void svc_xprt_put(struct svc_xprt *xprt)
 }
 EXPORT_SYMBOL_GPL(svc_xprt_put);
 
-/*
- * Called by transport drivers to initialize the transport independent
- * portion of the transport instance.
+/**
+ * svc_xprt_init - initialize transport-independent portion of a transport
+ * @net: network namespace in which the transport operates
+ * @xcl: transport class providing operations and metadata
+ * @xprt: svc_xprt to initialize
+ * @serv: RPC service that owns this transport
+ *
+ * Assigns @xprt->xpt_id, unique among the transports live in @net. The
+ * id value can be reused once @xprt is freed.
+ *
+ * Context: Process context. May sleep.
+ *
+ * Return:
+ *   %true: initialization succeeded
+ *   %false: initialization failed
  */
-void svc_xprt_init(struct net *net, struct svc_xprt_class *xcl,
+bool svc_xprt_init(struct net *net, struct svc_xprt_class *xcl,
 		   struct svc_xprt *xprt, struct svc_serv *serv)
 {
+	struct sunrpc_net *sn = net_generic(net, sunrpc_net_id);
+	u32 id;
+
 	memset(xprt, 0, sizeof(*xprt));
 	xprt->xpt_class = xcl;
 	xprt->xpt_ops = xcl->xcl_ops;
@@ -208,8 +228,17 @@ void svc_xprt_init(struct net *net, struct svc_xprt_class *xcl,
 	mutex_init(&xprt->xpt_mutex);
 	spin_lock_init(&xprt->xpt_lock);
 	set_bit(XPT_BUSY, &xprt->xpt_flags);
-	xprt->xpt_net = get_net_track(net, &xprt->ns_tracker, GFP_ATOMIC);
+	xprt->xpt_net = get_net_track(net, &xprt->ns_tracker, GFP_KERNEL);
 	strcpy(xprt->xpt_remotebuf, "uninitialized");
+
+	if (xa_alloc_cyclic(&sn->svc_xprt_ids, &id, xprt,
+			    XA_LIMIT(1, UINT_MAX), &sn->svc_xprt_id_next,
+			    GFP_KERNEL) < 0) {
+		put_net_track(xprt->xpt_net, &xprt->ns_tracker);
+		return false;
+	}
+	xprt->xpt_id = id;
+	return true;
 }
 EXPORT_SYMBOL_GPL(svc_xprt_init);
 
net/sunrpc/svcsock.c +23 -10
index e5459d504b6a..625aebbbc6b3 100644
@@ -803,10 +803,11 @@ static struct svc_xprt_class svc_udp_class = {
 	.xcl_flags = SVC_XPRT_FLAG_WSPACE_RESERVE,
 };
 
-static void svc_udp_init(struct svc_sock *svsk, struct svc_serv *serv)
+static bool svc_udp_init(struct svc_sock *svsk, struct svc_serv *serv)
 {
-	svc_xprt_init(sock_net(svsk->sk_sock->sk), &svc_udp_class,
-		      &svsk->sk_xprt, serv);
+	if (!svc_xprt_init(sock_net(svsk->sk_sock->sk), &svc_udp_class,
+			   &svsk->sk_xprt, serv))
+		return false;
 	clear_bit(XPT_CACHE_AUTH, &svsk->sk_xprt.xpt_flags);
 	svsk->sk_sk->sk_data_ready = svc_data_ready;
 	svsk->sk_sk->sk_write_space = svc_write_space;
@@ -833,6 +834,7 @@ static void svc_udp_init(struct svc_sock *svsk, struct svc_serv *serv)
 	default:
 		BUG();
 	}
+	return true;
 }
 
 /*
@@ -1476,12 +1478,13 @@ void svc_cleanup_xprt_sock(void)
 	svc_unreg_xprt_class(&svc_udp_class);
 }
 
-static void svc_tcp_init(struct svc_sock *svsk, struct svc_serv *serv)
+static bool svc_tcp_init(struct svc_sock *svsk, struct svc_serv *serv)
 {
 	struct sock	*sk = svsk->sk_sk;
 
-	svc_xprt_init(sock_net(svsk->sk_sock->sk), &svc_tcp_class,
-		      &svsk->sk_xprt, serv);
+	if (!svc_xprt_init(sock_net(svsk->sk_sock->sk), &svc_tcp_class,
+			   &svsk->sk_xprt, serv))
+		return false;
 	set_bit(XPT_CACHE_AUTH, &svsk->sk_xprt.xpt_flags);
 	set_bit(XPT_CONG_CTRL, &svsk->sk_xprt.xpt_flags);
 	if (sk->sk_state == TCP_LISTEN) {
@@ -1512,6 +1515,7 @@ static void svc_tcp_init(struct svc_sock *svsk, struct svc_serv *serv)
 			svc_xprt_deferred_close(&svsk->sk_xprt);
 		}
 	}
+	return true;
 }
 
 void svc_sock_update_bufs(struct svc_serv *serv)
@@ -1603,13 +1607,22 @@ static struct svc_sock *svc_setup_socket(struct svc_serv *serv,
 	inet->sk_user_data = svsk;
 
 	/* Initialize the socket */
-	if (sock->type == SOCK_DGRAM)
-		svc_udp_init(svsk, serv);
-	else
-		svc_tcp_init(svsk, serv);
+	if (sock->type == SOCK_DGRAM) {
+		if (!svc_udp_init(svsk, serv))
+			goto out_free;
+	} else {
+		if (!svc_tcp_init(svsk, serv))
+			goto out_free;
+	}
 
 	trace_svcsock_new(svsk, sock);
 	return svsk;
+
+out_free:
+	inet->sk_user_data = NULL;
+	kfree(svsk->sk_bvec);
+	kfree(svsk);
+	return ERR_PTR(-ENOMEM);
 }
 
 /**
net/sunrpc/xprtrdma/svc_rdma_transport.c +4 -1
index f949601b2144..610df78f9176 100644
@@ -189,7 +189,10 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,
 	if (!cma_xprt)
 		return NULL;
 
-	svc_xprt_init(net, &svc_rdma_class, &cma_xprt->sc_xprt, serv);
+	if (!svc_xprt_init(net, &svc_rdma_class, &cma_xprt->sc_xprt, serv)) {
+		kfree(cma_xprt);
+		return NULL;
+	}
 	INIT_LIST_HEAD(&cma_xprt->sc_accept_q);
 	INIT_LIST_HEAD(&cma_xprt->sc_rq_dto_q);
 	INIT_LIST_HEAD(&cma_xprt->sc_read_complete_q);
Signature · 1 line
2.55.0
author maintainer
#2 lore raw

PATCH v3 2/12 NFSD: Track transport in DRC entries

In reply to Chuck Lever

A cached reply carries no record of the transport its request arrived on. Transport-aware eviction needs that association.

Record the xprt id in each entry. No behavior change.

---
 fs/nfsd/cache.h    | 1 +
 fs/nfsd/nfscache.c | 1 +
 2 files changed, 2 insertions(+)
fs/nfsd/cache.h +1 -0
index 3bc4856e34b8..5fbf1bc37c03 100644
@@ -37,6 +37,7 @@ struct nfsd_cacherep {
 	unsigned char		c_state,	/* unused, inprog, done */
 				c_type,		/* status, buffer */
 				c_secure : 1;	/* req came from port < 1024 */
+	unsigned int		c_xprt;		/* svc_xprt that carried req */
 	unsigned long		c_timestamp;
 	union {
 		struct kvec	u_vec;
fs/nfsd/nfscache.c +1 -0
index 80364b91331a..b25b4f9e92f7 100644
@@ -109,6 +109,7 @@ nfsd_cacherep_alloc(struct svc_rqst *rqstp, __wsum csum,
 		rp->c_key.k_vers = rqstp->rq_vers;
 		rp->c_key.k_len = rqstp->rq_arg.len;
 		rp->c_key.k_csum = csum;
+		rp->c_xprt = rqstp->rq_xprt->xpt_id;
 	}
 	return rp;
 }
Signature · 1 line
2.55.0
author maintainer
#3 lore raw

PATCH v3 3/12 NFSD: Prepare bucket pruning for out-of-order eviction

In reply to Chuck Lever

nfsd_prune_bucket_locked() evicts for memory pressure and RC_EXPIRE. Both reasons hold for every entry older than the first one they hold for, so the walk stops at the first entry it cannot evict. An eviction reason that applies to one entry but not to an older one, such as confirmation that a particular reply reached the client, cannot be added to that loop.

Restructure the loop so each reason is a separate test that jumps to a shared eviction label, and continue past an entry no reason applies to. The restructured scan no longer stops at the first entry it cannot evict, so bound the work per call to four times the eviction limit. Pass the shrinker's remaining nr_to_scan budget as that limit, bounding its walk of each bucket the same way.

---
 fs/nfsd/nfscache.c | 36 ++++++++++++++++++++++++------------
 1 file changed, 24 insertions(+), 12 deletions(-)
fs/nfsd/nfscache.c +24 -12
index b25b4f9e92f7..e6504e192dfa 100644
@@ -257,8 +257,8 @@ nfsd_cache_bucket_find(__be32 xid, struct nfsd_net *nn)
 }
 
 /*
- * Remove and return no more than @max expired entries in bucket @b.
- * If @max is zero, do not limit the number of removed entries.
+ * Remove and return no more than @max evictable entries in bucket @b,
+ * visiting at most 4 * @max entries. @max must not be zero.
  */
 static void
 nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
@@ -266,20 +266,31 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
 {
 	unsigned long expiry = jiffies - RC_EXPIRE;
 	struct nfsd_cacherep *rp, *tmp;
-	unsigned int freed = 0;
+	unsigned int freed = 0, visited = 0;
 
 	lockdep_assert_held(&b->cache_lock);
 
 	/* The bucket LRU is ordered oldest-first. */
 	list_for_each_entry_safe(rp, tmp, &b->lru_head, c_lru) {
-		if (atomic_read(&nn->num_drc_entries) <= nn->max_drc_entries &&
-		    time_before(expiry, rp->c_timestamp))
-			break;
+		if (atomic_read(&nn->num_drc_entries) > nn->max_drc_entries)
+			goto evict;
+		if (time_before_eq(rp->c_timestamp, expiry))
+			goto evict;
+		goto next;
 
+evict:
 		nfsd_cacherep_unlink_locked(nn, b, rp);
 		list_add(&rp->c_lru, dispose);
+		freed++;
 
-		if (max && ++freed >= max)
+next:
+		/*
+		 * A client controls its XIDs, so it can pack one bucket with
+		 * entries that are not yet evictable and turn each miss into
+		 * a full-bucket walk under cache_lock. A skipped entry is
+		 * left for a later prune.
+		 */
+		if (freed >= max || ++visited >= max * 4)
 			break;
 	}
 }
@@ -307,9 +318,9 @@ nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc)
  * @shrink: our registered shrinker context
  * @sc: garbage collection parameters
  *
- * Free expired entries on each bucket's LRU list until we've released
- * nr_to_scan freed objects. Nothing will be released if the cache
- * has not exceeded it's max_drc_entries limit.
+ * Free entries on each bucket's LRU list until nr_to_scan objects have been
+ * released. Entries are evicted when they have expired or the cache exceeds
+ * its max_drc_entries limit.
  *
  * Returns the number of entries released by this call.
  */
@@ -328,11 +339,12 @@ nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
 			continue;
 
 		spin_lock(&b->cache_lock);
-		nfsd_prune_bucket_locked(nn, b, 0, &dispose);
+		nfsd_prune_bucket_locked(nn, b, sc->nr_to_scan - freed,
+					 &dispose);
 		spin_unlock(&b->cache_lock);
 
 		freed += nfsd_cacherep_dispose(&dispose);
-		if (freed > sc->nr_to_scan)
+		if (freed >= sc->nr_to_scan)
 			break;
 	}
 	return freed;
Signature · 1 line
2.55.0
author maintainer
#4 lore raw

PATCH v3 4/12 NFSD: Add tracepoints for DRC entry eviction

In reply to Chuck Lever

The DRC pruning path evicts entries for memory pressure and expiry, and neither emits a trace event, so there is no way to see which reason is retiring entries or how old they are when it happens. Eviction reasons that follow will need the same visibility.

Add an event class that records the cache population, the XID, the transport, and the entry's age, and define an event for each eviction reason.

---
 fs/nfsd/nfscache.c |  8 ++++++--
 fs/nfsd/trace.h    | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 2 deletions(-)
fs/nfsd/nfscache.c +6 -2
index e6504e192dfa..ec8c40781cb2 100644
@@ -272,10 +272,14 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
 
 	/* The bucket LRU is ordered oldest-first. */
 	list_for_each_entry_safe(rp, tmp, &b->lru_head, c_lru) {
-		if (atomic_read(&nn->num_drc_entries) > nn->max_drc_entries)
+		if (atomic_read(&nn->num_drc_entries) > nn->max_drc_entries) {
+			trace_nfsd_drc_evict_pressure(nn, rp);
 			goto evict;
-		if (time_before_eq(rp->c_timestamp, expiry))
+		}
+		if (time_before_eq(rp->c_timestamp, expiry)) {
+			trace_nfsd_drc_evict_expired(nn, rp);
 			goto evict;
+		}
 		goto next;
 
 evict:
fs/nfsd/trace.h +37 -0
index 2ae7f150a72c..255877e65d7c 100644
@@ -1557,6 +1557,43 @@ TRACE_EVENT(nfsd_drc_mismatch,
 		__entry->ingress)
 );
 
+DECLARE_EVENT_CLASS(nfsd_drc_entry_class,
+	TP_PROTO(
+		const struct nfsd_net *nn,
+		const struct nfsd_cacherep *rp
+	),
+	TP_ARGS(nn, rp),
+	TP_STRUCT__entry(
+		__field(unsigned long long, boot_time)
+		__field(unsigned int, num_drc_entries)
+		__field(u32, xid)
+		__field(unsigned int, xprt)
+		__field(unsigned long, age)
+	),
+	TP_fast_assign(
+		__entry->boot_time = nn->boot_time;
+		__entry->num_drc_entries = atomic_read(&nn->num_drc_entries);
+		__entry->xid = be32_to_cpu(rp->c_key.k_xid);
+		__entry->xprt = rp->c_xprt;
+		__entry->age = time_is_after_jiffies(rp->c_timestamp) ?
+				1 : jiffies - rp->c_timestamp;
+	),
+	TP_printk("boot_time=%16llx entries=%u xid=0x%08x xprt=%u age=%lu",
+		__entry->boot_time, __entry->num_drc_entries,
+		__entry->xid, __entry->xprt, __entry->age)
+);
+
+#define DEFINE_NFSD_DRC_ENTRY_EVENT(name)			\
+DEFINE_EVENT(nfsd_drc_entry_class, nfsd_drc_##name,		\
+	TP_PROTO(						\
+		const struct nfsd_net *nn,			\
+		const struct nfsd_cacherep *rp			\
+	),							\
+	TP_ARGS(nn, rp))
+
+DEFINE_NFSD_DRC_ENTRY_EVENT(evict_pressure);
+DEFINE_NFSD_DRC_ENTRY_EVENT(evict_expired);
+
 TRACE_EVENT(nfsd_cb_args,
 	TP_PROTO(
 		const struct nfs4_client *clp,
Signature · 1 line
2.55.0
author maintainer
#5 lore raw

PATCH v3 5/12 NFSD: Record DRC population in lookup tracepoints

In reply to Chuck Lever

nfsd_drc_found reports the outcome of a lookup but not how full the cache was at the time, so a trace cannot show whether retransmits are being caught while the cache runs near its cap.

---
 fs/nfsd/nfscache.c |  3 ++-
 fs/nfsd/trace.h    | 11 +++++++----
 2 files changed, 9 insertions(+), 5 deletions(-)
fs/nfsd/nfscache.c +2 -1
index ec8c40781cb2..10fdc60f9bb9 100644
@@ -562,7 +562,8 @@ int nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start,
 	}
 
 out_trace:
-	trace_nfsd_drc_found(nn, rqstp, rtn);
+	trace_nfsd_drc_found(nn, atomic_read(&nn->num_drc_entries),
+			     rqstp, rtn);
 out_unlock:
 	spin_unlock(&b->cache_lock);
 out:
fs/nfsd/trace.h +7 -4
index 255877e65d7c..a07d3dc76f27 100644
@@ -1513,23 +1513,26 @@ TRACE_DEFINE_ENUM(RC_DOIT);
 TRACE_EVENT(nfsd_drc_found,
 	TP_PROTO(
 		const struct nfsd_net *nn,
+		unsigned int num_drc_entries,
 		const struct svc_rqst *rqstp,
 		int result
 	),
-	TP_ARGS(nn, rqstp, result),
+	TP_ARGS(nn, num_drc_entries, rqstp, result),
 	TP_STRUCT__entry(
 		__field(unsigned long long, boot_time)
+		__field(unsigned int, num_drc_entries)
 		__field(unsigned long, result)
 		__field(u32, xid)
 	),
 	TP_fast_assign(
 		__entry->boot_time = nn->boot_time;
+		__entry->num_drc_entries = num_drc_entries;
 		__entry->result = result;
 		__entry->xid = be32_to_cpu(rqstp->rq_xid);
 	),
-	TP_printk("boot_time=%16llx xid=0x%08x result=%s",
-		__entry->boot_time, __entry->xid,
-		show_drc_retval(__entry->result))
+	TP_printk("boot_time=%16llx entries=%u xid=0x%08x result=%s",
+		__entry->boot_time, __entry->num_drc_entries,
+		__entry->xid, show_drc_retval(__entry->result))
 
 );
 
Signature · 1 line
2.55.0
author maintainer
#6 lore raw

PATCH v3 6/12 NFSD: Add reply-acknowledged callback infrastructure

In reply to Chuck Lever

The DRC retains an entry for RC_EXPIRE whether or not the client has received the reply. TCP and RDMA can confirm delivery, which would allow the entry to be evicted much earlier.

Early eviction is safe because a client retransmits a request only when its reply never arrived. Once the transport confirms the reply reached the client, no retransmit of that XID can follow, so the entry can never again satisfy a lookup. A reply still in flight when a connection drops is never confirmed, so the entries a reconnecting client retransmits against keep their full RC_EXPIRE retention.

Add a callback and opaque private pointer on struct svc_serv so a transport can report reply delivery without depending on NFS types. The report carries an opaque cookie naming the entry by XID, transport identifier, and generation. The generation guards XID
reuse: without it a stale report could mark a successor entry delivered, evicting a reply the client never received. The handler caps its newest-first bucket walk because a client chooses which bucket its XIDs hash to and could otherwise pin cache_lock with an unbounded walk.

A transport that reports on replies sets XPT_REPLY_ACK, and its entries are marked c_ack_pending until the report arrives, so an eviction reason that only guesses at delivery can defer to it.

---
 fs/nfsd/cache.h                 | 10 +++--
 fs/nfsd/nfscache.c              | 86 ++++++++++++++++++++++++++++++++++++++++-
 fs/nfsd/nfssvc.c                | 10 ++---
 fs/nfsd/trace.h                 | 34 ++++++++++++++++
 include/linux/sunrpc/svc.h      | 48 +++++++++++++++++++++++
 include/linux/sunrpc/svc_xprt.h |  1 +
 include/trace/events/sunrpc.h   |  1 +
 7 files changed, 180 insertions(+), 10 deletions(-)
fs/nfsd/cache.h +7 -3
index 5fbf1bc37c03..8ad23a1fcb57 100644
@@ -36,8 +36,11 @@ struct nfsd_cacherep {
 	struct list_head	c_lru;
 	unsigned char		c_state,	/* unused, inprog, done */
 				c_type,		/* status, buffer */
-				c_secure : 1;	/* req came from port < 1024 */
+				c_secure : 1,	/* req came from port < 1024 */
+				c_acked : 1,	/* reply delivery confirmed */
+				c_ack_pending : 1; /* transport will report */
 	unsigned int		c_xprt;		/* svc_xprt that carried req */
+	u32			c_ack_gen;	/* uniquifies the ack cookie */
 	unsigned long		c_timestamp;
 	union {
 		struct kvec	u_vec;
@@ -80,10 +83,11 @@ enum {
 /* Checksum this amount of the request */
 #define RC_CSUMLEN		(256U)
 
+svc_ack_cookie_t nfsd_cache_ack_cookie(const struct nfsd_cacherep *rp);
 int	nfsd_drc_slab_create(void);
 void	nfsd_drc_slab_free(void);
-int	nfsd_reply_cache_init(struct nfsd_net *);
-void	nfsd_reply_cache_shutdown(struct nfsd_net *);
+int	nfsd_reply_cache_init(struct nfsd_net *, struct svc_serv *);
+void	nfsd_reply_cache_shutdown(struct nfsd_net *, struct svc_serv *);
 int	nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start,
 			  unsigned int len, struct nfsd_cacherep **cacherep);
 void	nfsd_cache_update(struct svc_rqst *rqstp, struct nfsd_cacherep *rp,
fs/nfsd/nfscache.c +84 -2

152 lines not on this page · raw message

fs/nfsd/nfssvc.c +5 -5
index c04ef9d180ce..d6687ffbec45 100644
@@ -389,7 +389,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
 	if (ret)
 		goto out_lockd;
 
-	ret = nfsd_reply_cache_init(nn);
+	ret = nfsd_reply_cache_init(nn, nn->nfsd_serv);
 	if (ret)
 		goto out_filecache;
 
@@ -404,7 +404,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
 	return 0;
 
 out_reply_cache:
-	nfsd_reply_cache_shutdown(nn);
+	nfsd_reply_cache_shutdown(nn, nn->nfsd_serv);
 out_filecache:
 	nfsd_file_cache_shutdown_net(net);
 out_lockd:
@@ -417,7 +417,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)
 	return ret;
 }
 
-static void nfsd_shutdown_net(struct net *net)
+static void nfsd_shutdown_net(struct net *net, struct svc_serv *serv)
 {
 	struct nfsd_net *nn = net_generic(net, nfsd_net_id);
 
@@ -427,7 +427,7 @@ static void nfsd_shutdown_net(struct net *net)
 
 		nfsd_export_flush(net);
 		nfs4_state_shutdown_net(net);
-		nfsd_reply_cache_shutdown(nn);
+		nfsd_reply_cache_shutdown(nn, serv);
 		nfsd_file_cache_shutdown_net(net);
 		if (test_bit(NFSD_NET_LOCKD_UP, &nn->flags)) {
 			lockd_down(net);
@@ -539,7 +539,7 @@ void nfsd_destroy_serv(struct net *net)
 	 * other initialization has been done except the rpcb information.
 	 */
 	svc_xprt_destroy_all(serv, net, true);
-	nfsd_shutdown_net(net);
+	nfsd_shutdown_net(net, serv);
 	svc_destroy(&serv);
 }
 
fs/nfsd/trace.h +34 -0
index a07d3dc76f27..0af3a8231fe4 100644
@@ -1594,9 +1594,43 @@ DEFINE_EVENT(nfsd_drc_entry_class, nfsd_drc_##name,		\
 	),							\
 	TP_ARGS(nn, rp))
 
+DEFINE_NFSD_DRC_ENTRY_EVENT(evict_acked);
 DEFINE_NFSD_DRC_ENTRY_EVENT(evict_pressure);
 DEFINE_NFSD_DRC_ENTRY_EVENT(evict_expired);
 
+TRACE_EVENT(nfsd_drc_reply_acked,
+	TP_PROTO(
+		const struct nfsd_net *nn,
+		unsigned int num_drc_entries,
+		u32 xid,
+		unsigned int xprt,
+		bool delivered,
+		bool found
+	),
+	TP_ARGS(nn, num_drc_entries, xid, xprt, delivered, found),
+	TP_STRUCT__entry(
+		__field(unsigned long long, boot_time)
+		__field(unsigned int, num_drc_entries)
+		__field(u32, xid)
+		__field(unsigned int, xprt)
+		__field(bool, delivered)
+		__field(bool, found)
+	),
+	TP_fast_assign(
+		__entry->boot_time = nn->boot_time;
+		__entry->num_drc_entries = num_drc_entries;
+		__entry->xid = xid;
+		__entry->xprt = xprt;
+		__entry->delivered = delivered;
+		__entry->found = found;
+	),
+	TP_printk("boot_time=%16llx entries=%u xid=0x%08x xprt=%u %s %s",
+		__entry->boot_time, __entry->num_drc_entries,
+		__entry->xid, __entry->xprt,
+		__entry->delivered ? "delivered" : "untracked",
+		__entry->found ? "found" : "stale")
+);
+
 TRACE_EVENT(nfsd_cb_args,
 	TP_PROTO(
 		const struct nfs4_client *clp,
include/linux/sunrpc/svc.h +48 -0

66 lines not on this page · raw message

include/linux/sunrpc/svc_xprt.h +1 -0
index c62f789e2900..ec1dcb51c2a2 100644
@@ -101,6 +101,7 @@ enum {
 	XPT_LOCAL,		/* connection from loopback interface */
 	XPT_KILL_TEMP,		/* call xpo_kill_temp_xprt before closing */
 	XPT_CONG_CTRL,		/* has congestion control */
+	XPT_REPLY_ACK,		/* reports reply delivery via svc_reply_acked */
 	XPT_HANDSHAKE,		/* xprt requests a handshake */
 	XPT_TLS_SESSION,	/* transport-layer security established */
 	XPT_PEER_AUTH,		/* peer has been authenticated */
include/trace/events/sunrpc.h +1 -0
index 180346e520ff..a9d42625c99b 100644
@@ -1931,6 +1931,7 @@ TRACE_EVENT(svc_stats_latency,
 	svc_xprt_flag(LOCAL)						\
 	svc_xprt_flag(KILL_TEMP)					\
 	svc_xprt_flag(CONG_CTRL)					\
+	svc_xprt_flag(REPLY_ACK)					\
 	svc_xprt_flag(HANDSHAKE)					\
 	svc_xprt_flag(TLS_SESSION)					\
 	svc_xprt_flag(PEER_AUTH)					\
Signature · 1 line
2.55.0
author maintainer
#7 lore raw

PATCH v3 7/12 SUNRPC: Add TCP sequence-number ACK tracking for reply delivery

In reply to Chuck Lever

A DRC entry for a reply sent over TCP lingers until RC_EXPIRE even though the peer's TCP acknowledgment has already confirmed that the reply arrived.

The acknowledgment confirms only that the reply reached the peer's TCP stack. If the connection drops before the NFS client reads it, the client retransmits the call and the evicted entry cannot answer it. Memory pressure and RC_EXPIRE already open the same window.

When the upper layer caches a reply it stores a cookie on the svc_rqst. After svc_tcp_sendto() transmits the reply, record the cookie with the current write_seq in a fixed-size per-socket ring. Before appending, drain entries whose sequence number is at or before snd_una and report each as delivered. The send path is single-threaded, so the ring needs no locking. sk_wmem_queued cannot be used for this because it counts per-skb truesize, not payload bytes.

On a kTLS session the snapshot needs one more check. A short push inside the TLS layer leaves the rest of the record as partially_sent_record, and tls_sw_sendmsg() still returns the full plaintext count, so write_seq can fall short of the reply. Record the reply only when no partially_sent_record remains after the send.

Some replies get no ring entry: the ring is full, the send fails, the transport is already dead, or a kTLS record is still partly unsent. Report each of these as untracked at once, so the upper layer knows no delivery report will follow. When the socket is freed, report entries still in the ring as undelivered. Set XPT_REPLY_ACK on TCP transports so the upper layer expects these reports.

---
 fs/nfsd/nfscache.c             | 15 +++++---
 include/linux/sunrpc/svc.h     |  3 ++
 include/linux/sunrpc/svcsock.h | 10 +++++
 net/sunrpc/svc.c               |  1 +
 net/sunrpc/svcsock.c           | 85 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 108 insertions(+), 6 deletions(-)
fs/nfsd/nfscache.c +9 -6
index b0231c659237..473bf825c37a 100644
@@ -289,12 +289,14 @@ nfsd_cache_bucket_find(__be32 xid, struct nfsd_net *nn)
 }
 
 /*
- * The generation match keeps a stale cookie from acknowledging a
- * later entry that reuses the same XID and transport. The walk
- * starts at the MRU end of the bucket LRU, where the entry for a
- * just-sent reply sits. The visit cap bounds the time spent under
- * cache_lock when a client packs the bucket; an entry missed under
- * the cap is left to the other eviction reasons.
+ * The generation match keeps a stale cookie from acknowledging a later
+ * entry that reuses the same XID and transport.
+ *
+ * Average bucket occupancy is small (TARGET_BUCKET_SIZE), so a linear walk
+ * of the bucket LRU beats an rb-tree lookup. The walk starts at the MRU
+ * end, where the entry for a just-sent reply sits. The visit cap bounds
+ * the time spent under cache_lock when a client packs the bucket; an entry
+ * missed under the cap is left to the other eviction reasons.
  */
 static void nfsd_reply_ack(void *data, const svc_ack_cookie_t *cookie,
 			   bool delivered)
@@ -723,6 +725,7 @@ void nfsd_cache_update(struct svc_rqst *rqstp, struct nfsd_cacherep *rp,
 	rp->c_ack_gen = nfsd_cache_next_ack_gen();
 	rp->c_type = cachetype;
 	rp->c_state = RC_DONE;
+	rqstp->rq_ack_cookie = nfsd_cache_ack_cookie(rp);
 	spin_unlock(&b->cache_lock);
 	return;
 }
include/linux/sunrpc/svc.h +3 -0
index 8c9e27752698..da3822c54f3e 100644
@@ -315,6 +315,9 @@ struct svc_rqst {
 	unsigned int		bc_to_retries;
 	unsigned int		rq_status_counter; /* RPC processing counter */
 	void			*rq_private;	/* For use by the service thread */
+
+	svc_ack_cookie_t	rq_ack_cookie;	/* zero when no reply-ack
+						 * tracking is requested */
 };
 
 /* bits for rq_flags */
include/linux/sunrpc/svcsock.h +10 -0
index 372a00882ca6..595be631b68b 100644
@@ -41,6 +41,16 @@ struct svc_sock {
 
 	struct page_frag_cache  sk_frag_cache;
 
+	/* TCP reply-ACK tracking (single-threaded send path) */
+#define SVC_ACK_RING_BITS	6
+#define SVC_ACK_RING_SIZE	(1 << SVC_ACK_RING_BITS)
+	struct {
+		u32		ae_pos;		/* write_seq after send */
+		svc_ack_cookie_t ae_cookie;	/* opaque DRC cookie */
+	}			sk_ack_ring[SVC_ACK_RING_SIZE];
+	unsigned int		sk_ack_head;
+	unsigned int		sk_ack_tail;
+
 	struct completion	sk_handshake_done;
 
 	/* received data */
net/sunrpc/svc.c +1 -0
index f73412e123a1..33dd9cbaff42 100644
@@ -1488,6 +1488,7 @@ svc_process_common(struct svc_rqst *rqstp)
 
 	/* Reset the accept_stat for the RPC */
 	rqstp->rq_accept_statp = NULL;
+	rqstp->rq_ack_cookie = (svc_ack_cookie_t){};
 
 	/* Will be turned off only when NFSv4 Sessions are used */
 	set_bit(RQ_USEDEFERRAL, &rqstp->rq_flags);
net/sunrpc/svcsock.c +85 -0

149 lines not on this page · raw message

Signature · 1 line
2.55.0
author maintainer
#8 lore raw

PATCH v3 8/12 svcrdma: Fire reply-acknowledged callback on Send completion

In reply to Chuck Lever

svcrdma posts RDMA Write work requests ahead of the Send, so a successful Send completion confirms that the transport header and the reply payload have both reached the client.

Store rq_ack_cookie in the send context before posting and report the reply as delivered from the Send completion handler, so the DRC can evict the entry without waiting for RC_EXPIRE. Set XPT_REPLY_ACK on RDMA transports so the upper layer expects these reports. A flushed Send reports the reply as undelivered: the connection is closing, and the entry stays cached for a retransmit on the next connection.

---
 include/linux/sunrpc/svc_rdma.h          |  1 +
 net/sunrpc/xprtrdma/svc_rdma_sendto.c    | 14 ++++++++++++++
 net/sunrpc/xprtrdma/svc_rdma_transport.c |  1 +
 3 files changed, 16 insertions(+)
include/linux/sunrpc/svc_rdma.h +1 -0
index 76aa5ec4ab40..7261dd3a892e 100644
@@ -248,6 +248,7 @@ struct svc_rdma_send_ctxt {
 	struct list_head	sc_write_info_list;
 	struct svc_rdma_write_info sc_reply_info;
 
+	svc_ack_cookie_t	sc_ack_cookie;
 	void			*sc_xprt_buf;
 	int			sc_page_count;
 	int			sc_cur_sge_no;
net/sunrpc/xprtrdma/svc_rdma_sendto.c +14 -0
index c09659b17351..884e2d76cb6f 100644
@@ -220,6 +220,7 @@ struct svc_rdma_send_ctxt *svc_rdma_send_ctxt_get(struct svcxprt_rdma *rdma)
 	ctxt->sc_send_wr.num_sge = 0;
 	ctxt->sc_cur_sge_no = 0;
 	ctxt->sc_page_count = 0;
+	ctxt->sc_ack_cookie = (svc_ack_cookie_t){};
 	ctxt->sc_wr_chain = &ctxt->sc_send_wr;
 	ctxt->sc_sqecount = 1;
 
@@ -470,11 +471,17 @@ static void svc_rdma_wc_send(struct ib_cq *cq, struct ib_wc *wc)
 	if (unlikely(wc->status != IB_WC_SUCCESS))
 		goto flushed;
 
+	if (svc_ack_cookie_present(&ctxt->sc_ack_cookie))
+		svc_reply_acked(rdma->sc_xprt.xpt_server,
+				&ctxt->sc_ack_cookie, true);
 	trace_svcrdma_wc_send(&ctxt->sc_cid);
 	svc_rdma_send_ctxt_put(rdma, ctxt);
 	return;
 
 flushed:
+	if (svc_ack_cookie_present(&ctxt->sc_ack_cookie))
+		svc_reply_acked(rdma->sc_xprt.xpt_server,
+				&ctxt->sc_ack_cookie, false);
 	if (wc->status != IB_WC_WR_FLUSH_ERR)
 		trace_svcrdma_wc_send_err(wc, &ctxt->sc_cid);
 	else
@@ -1198,6 +1205,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)
 	if (ret < 0)
 		goto put_ctxt;
 
+	sctxt->sc_ack_cookie = rqstp->rq_ack_cookie;
 	ret = svc_rdma_send_reply_msg(rdma, sctxt, rctxt, rqstp);
 	if (ret < 0)
 		goto send_err;
@@ -1207,6 +1215,12 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)
 	if (ret != -E2BIG && ret != -EINVAL)
 		goto put_ctxt;
 
+	/* The sctxt is reused for the RDMA_ERROR message. Clear the
+	 * ack cookie so that message's Send completion does not
+	 * report the unsent reply as delivered.
+	 */
+	sctxt->sc_ack_cookie = (svc_ack_cookie_t){};
+
 	/* Send completion releases payload pages that were part
 	 * of previously posted RDMA Writes.
 	 */
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -0
index 610df78f9176..64e964ac941c 100644
@@ -218,6 +218,7 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,
 	 * transports are suitable here.
 	 */
 	set_bit(XPT_CONG_CTRL, &cma_xprt->sc_xprt.xpt_flags);
+	set_bit(XPT_REPLY_ACK, &cma_xprt->sc_xprt.xpt_flags);
 
 	return cma_xprt;
 }
Signature · 1 line
2.55.0
author maintainer
#9 lore raw

PATCH v3 9/12 SUNRPC: Record last-request timestamp on svc_xprt

In reply to Chuck Lever

On a connection-oriented transport, the arrival of a later request suggests the client received the prior reply. It is not proof: a client with several requests outstanding sends the next one without waiting for a reply.

Add xpt_last_recv to struct svc_xprt. svc_handle_xprt() samples jiffies before calling xpo_recvfrom() and publishes the sample after a successful receive. The DRC reads this value as a best-effort signal that a completed reply has been delivered.

---
 include/linux/sunrpc/svc_xprt.h |  1 +
 net/sunrpc/svc_xprt.c           | 14 ++++++++++++++
 2 files changed, 15 insertions(+)
include/linux/sunrpc/svc_xprt.h +1 -0
index ec1dcb51c2a2..da234a078c5d 100644
@@ -81,6 +81,7 @@ struct svc_xprt {
 	struct net		*xpt_net;
 	netns_tracker		ns_tracker;
 	const struct cred	*xpt_cred;
+	unsigned long		xpt_last_recv;	/* jiffies of last request */
 	struct rpc_xprt		*xpt_bc_xprt;	/* NFSv4.1 backchannel */
 	struct rpc_xprt_switch	*xpt_bc_xps;	/* NFSv4.1 backchannel */
 };
net/sunrpc/svc_xprt.c +14 -0
index 7e8ef4832421..f57dac54c8dc 100644
@@ -230,6 +230,7 @@ bool svc_xprt_init(struct net *net, struct svc_xprt_class *xcl,
 	set_bit(XPT_BUSY, &xprt->xpt_flags);
 	xprt->xpt_net = get_net_track(net, &xprt->ns_tracker, GFP_KERNEL);
 	strcpy(xprt->xpt_remotebuf, "uninitialized");
+	xprt->xpt_last_recv = jiffies;
 
 	if (xa_alloc_cyclic(&sn->svc_xprt_ids, &id, xprt,
 			    XA_LIMIT(1, UINT_MAX), &sn->svc_xprt_id_next,
@@ -901,6 +902,8 @@ static void svc_handle_xprt(struct svc_rqst *rqstp, struct svc_xprt *xprt)
 		svc_xprt_received(xprt);
 	} else if (svc_xprt_reserve_slot(rqstp, xprt)) {
 		/* XPT_DATA|XPT_DEFERRED case: */
+		unsigned long recv_time = jiffies;
+
 		rqstp->rq_deferred = svc_deferred_dequeue(xprt);
 		if (rqstp->rq_deferred)
 			len = svc_deferred_recv(rqstp);
@@ -916,6 +919,17 @@ static void svc_handle_xprt(struct svc_rqst *rqstp, struct svc_xprt *xprt)
 
 		clear_bit(XPT_OLD, &xprt->xpt_flags);
 
+		/*
+		 * A deferred request arrived before its deferral, so its
+		 * replay does not advance the timestamp. recv_time was
+		 * sampled before xpo_recvfrom() released XPT_BUSY.
+		 * time_after() keeps a slow thread from pushing the
+		 * timestamp past requests received since then.
+		 */
+		if (!rqstp->rq_deferred &&
+		    time_after(recv_time, READ_ONCE(xprt->xpt_last_recv)))
+			WRITE_ONCE(xprt->xpt_last_recv, recv_time);
+
 		rqstp->rq_chandle.defer = svc_defer;
 
 		if (serv->sv_stats)
Signature · 1 line
2.55.0
author maintainer
#10 lore raw

PATCH v3 10/12 NFSD: Evict unacknowledged DRC entries via implied ACK

In reply to Chuck Lever

A transport reports reply delivery only while it is tracking the reply. The TCP ring has a fixed size, so a burst leaves some replies untracked, and a kTLS reply is untracked when part of its record is still unsent. Entries for those replies stay in their bucket until RC_EXPIRE elapses or the cache exceeds max_drc_entries, lengthening every lookup that hashes there.

RFC 1813 Section 4.5 observes that on a connection-oriented transport a duplicate request arises from reconnection, not from within a live connection. A fresh request on a live TCP or RDMA connection therefore means the client is not retransmitting an earlier one. UDP offers no such signal.

The signal is not conclusive. An entry can be evicted while its reply is still in flight, and a reconnect then executes the request again. Memory pressure and RC_EXPIRE already open the same window.

Add an XPT_ORDERED flag marking transports that offer the signal. During bucket pruning, evict an RC_DONE entry when the current request's transport is XPT_ORDERED, no delivery report is pending, and its c_timestamp predates xpt_last_recv. Only that transport is consulted, because it is the one the pruning thread holds a reference to.

On svcrdma, handle_connect_req() zeroes the remote port in the DRC key so a cached reply survives a reconnect, and implied-ACK eviction can retire such an entry while the original connection is still up. TCP gives up the same thing when a client re-binds its reserved port.

---
 fs/nfsd/nfscache.c                       | 45 +++++++++++++++++++++++++++++---
 fs/nfsd/trace.h                          |  1 +
 include/linux/sunrpc/svc_xprt.h          |  1 +
 include/trace/events/sunrpc.h            |  1 +
 net/sunrpc/svcsock.c                     |  1 +
 net/sunrpc/xprtrdma/svc_rdma_transport.c |  1 +
 6 files changed, 47 insertions(+), 3 deletions(-)
fs/nfsd/nfscache.c +42 -3
index 473bf825c37a..9f9baf7910ff 100644
@@ -90,6 +90,36 @@ nfsd_hashsize(unsigned int limit)
 	return roundup_pow_of_two(limit / TARGET_BUCKET_SIZE);
 }
 
+/*
+ * A later request on @xprt is taken as evidence that the client
+ * received @rp's reply, because a client does not retransmit within
+ * a live connection. Only XPT_ORDERED transports qualify: a UDP
+ * client retransmits on timeout, and a datagram from any of the
+ * peers sharing the UDP svc_xprt would evict another client's reply.
+ *
+ * A pipelined client sends its next request before @rp's reply
+ * arrives, so while the transport is still going to report on that
+ * reply, wait for the report instead.
+ *
+ * c_timestamp is set after xpt_last_recv was recorded for @rp's own
+ * request, so a newer xpt_last_recv means a later request arrived.
+ */
+static bool nfsd_cacherep_implied_ack(struct svc_xprt *xprt,
+				      struct nfsd_cacherep *rp)
+{
+	unsigned long last_req;
+
+	if (!xprt || rp->c_xprt != xprt->xpt_id)
+		return false;
+	if (!test_bit(XPT_ORDERED, &xprt->xpt_flags))
+		return false;
+	if (rp->c_ack_pending)
+		return false;
+
+	last_req = READ_ONCE(xprt->xpt_last_recv);
+	return time_after(last_req, rp->c_timestamp);
+}
+
 static struct nfsd_cacherep *
 nfsd_cacherep_alloc(struct svc_rqst *rqstp, __wsum csum,
 		    struct nfsd_net *nn)
@@ -337,10 +367,14 @@ static void nfsd_reply_ack(void *data, const svc_ack_cookie_t *cookie,
 /*
  * Remove and return no more than @max evictable entries in bucket @b,
  * visiting at most 4 * @max entries. @max must not be zero.
+ *
+ * @xprt is the transport the current request arrived on, or NULL when the
+ * caller has none.
  */
 static void
 nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
-			 unsigned int max, struct list_head *dispose)
+			 unsigned int max, struct list_head *dispose,
+			 struct svc_xprt *xprt)
 {
 	unsigned long expiry = jiffies - RC_EXPIRE;
 	struct nfsd_cacherep *rp, *tmp;
@@ -362,6 +396,11 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
 			trace_nfsd_drc_evict_expired(nn, rp);
 			goto evict;
 		}
+		if (rp->c_state == RC_DONE &&
+		    nfsd_cacherep_implied_ack(xprt, rp)) {
+			trace_nfsd_drc_evict_implied_ack(nn, rp);
+			goto evict;
+		}
 		goto next;
 
 evict:
@@ -426,7 +465,7 @@ nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc)
 
 		spin_lock(&b->cache_lock);
 		nfsd_prune_bucket_locked(nn, b, sc->nr_to_scan - freed,
-					 &dispose);
+					 &dispose, NULL);
 		spin_unlock(&b->cache_lock);
 
 		freed += nfsd_cacherep_dispose(&dispose);
@@ -599,7 +638,7 @@ int nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start,
 		goto found_entry;
 	*cacherep = rp;
 	rp->c_state = RC_INPROG;
-	nfsd_prune_bucket_locked(nn, b, 3, &dispose);
+	nfsd_prune_bucket_locked(nn, b, 3, &dispose, rqstp->rq_xprt);
 	spin_unlock(&b->cache_lock);
 
 	nfsd_cacherep_dispose(&dispose);
fs/nfsd/trace.h +1 -0
index 0af3a8231fe4..c99b2a369d0d 100644
@@ -1597,6 +1597,7 @@ DEFINE_EVENT(nfsd_drc_entry_class, nfsd_drc_##name,		\
 DEFINE_NFSD_DRC_ENTRY_EVENT(evict_acked);
 DEFINE_NFSD_DRC_ENTRY_EVENT(evict_pressure);
 DEFINE_NFSD_DRC_ENTRY_EVENT(evict_expired);
+DEFINE_NFSD_DRC_ENTRY_EVENT(evict_implied_ack);
 
 TRACE_EVENT(nfsd_drc_reply_acked,
 	TP_PROTO(
include/linux/sunrpc/svc_xprt.h +1 -0
index da234a078c5d..cc29a8db269b 100644
@@ -103,6 +103,7 @@ enum {
 	XPT_KILL_TEMP,		/* call xpo_kill_temp_xprt before closing */
 	XPT_CONG_CTRL,		/* has congestion control */
 	XPT_REPLY_ACK,		/* reports reply delivery via svc_reply_acked */
+	XPT_ORDERED,		/* connection-oriented, in-order delivery */
 	XPT_HANDSHAKE,		/* xprt requests a handshake */
 	XPT_TLS_SESSION,	/* transport-layer security established */
 	XPT_PEER_AUTH,		/* peer has been authenticated */
include/trace/events/sunrpc.h +1 -0
index a9d42625c99b..e793e8ea322b 100644
@@ -1932,6 +1932,7 @@ TRACE_EVENT(svc_stats_latency,
 	svc_xprt_flag(KILL_TEMP)					\
 	svc_xprt_flag(CONG_CTRL)					\
 	svc_xprt_flag(REPLY_ACK)					\
+	svc_xprt_flag(ORDERED)						\
 	svc_xprt_flag(HANDSHAKE)					\
 	svc_xprt_flag(TLS_SESSION)					\
 	svc_xprt_flag(PEER_AUTH)					\
net/sunrpc/svcsock.c +1 -0
index a28905181589..94361e5a743e 100644
@@ -1567,6 +1567,7 @@ static bool svc_tcp_init(struct svc_sock *svsk, struct svc_serv *serv)
 	set_bit(XPT_CACHE_AUTH, &svsk->sk_xprt.xpt_flags);
 	set_bit(XPT_CONG_CTRL, &svsk->sk_xprt.xpt_flags);
 	set_bit(XPT_REPLY_ACK, &svsk->sk_xprt.xpt_flags);
+	set_bit(XPT_ORDERED, &svsk->sk_xprt.xpt_flags);
 	if (sk->sk_state == TCP_LISTEN) {
 		strcpy(svsk->sk_xprt.xpt_remotebuf, "listener");
 		set_bit(XPT_LISTENER, &svsk->sk_xprt.xpt_flags);
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -0
index 64e964ac941c..acf437516ae1 100644
@@ -219,6 +219,7 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,
 	 */
 	set_bit(XPT_CONG_CTRL, &cma_xprt->sc_xprt.xpt_flags);
 	set_bit(XPT_REPLY_ACK, &cma_xprt->sc_xprt.xpt_flags);
+	set_bit(XPT_ORDERED, &cma_xprt->sc_xprt.xpt_flags);
 
 	return cma_xprt;
 }
Signature · 1 line
2.55.0
author maintainer
#11 lore raw

PATCH v3 11/12 NFSD: Remove DRC checksum and payload_misses stat

In reply to Chuck Lever

nfsd_cache_csum() costs CPU on every non-idempotent request. It also prevents the use of zero-copy RDMA receives for WRITE and SYMLINK because the payload must be in the server's memory before the DRC lookup can run.

Commit 01a7decf7593 ("nfsd: keep a checksum of the first 256 bytes of request") added the checksum when a growing cache made XID collisions easier to hit. It is the only guard against a reused
XID: two calls with the same procedure and equal-length arguments match on every other key field.

ACK-driven eviction retires a TCP or RDMA entry once the transport confirms delivery, so a fresh XID rarely finds a resident entry to collide with (~100/2^32 per request). A UDP entry can wait out RC_EXPIRE, but the Linux NFS client seeds its XIDs from get_random_u32(), so even a rebooted client does not replay its previous sequence. A client that repeats a live XID cannot reliably match replies to its own calls anyway.

An acknowledged entry stays in its bucket until the next prune visits it, and a lookup matches it in the meantime. The client already holds that reply, so treat a call carrying its XID as a
miss: evict the entry and insert the new one in its place.

Remove nfsd_cache_csum(), RC_CSUMLEN, and k_csum, along with the nfsd_drc_mismatch tracepoint and the payload_misses stat that counted checksum-detected collisions. Drop the start and len parameters from nfsd_cache_lookup(), so nfsd_dispatch() no longer snapshots the argument stream before decoding.

The "payload misses" line disappears from /proc/fs/nfsd/reply_cache_stats. No known userspace tool parses it.

---
 .../ABI/testing/procfs-nfsd-reply_cache_stats      |  11 +--
 fs/nfsd/cache.h                                    |   9 +-
 fs/nfsd/netns.h                                    |   2 -
 fs/nfsd/nfscache.c                                 | 103 +++++----------------
 fs/nfsd/nfssvc.c                                   |  10 +-
 fs/nfsd/stats.h                                    |   5 -
 fs/nfsd/trace.h                                    |  24 -----
 7 files changed, 28 insertions(+), 136 deletions(-)
Documentation/ABI/testing/procfs-nfsd-reply_cache_stats +4 -7
index 57ed5f8e6597..7a22ac7c1ccd 100644
@@ -19,19 +19,16 @@ Description:
 		cache misses             s64     Requests not found in cache
 		not cached               s64     Idempotent requests that
 		                                 bypass the cache
-		payload misses           s64     XID matched but request
-		                                 checksum did not
 		longest chain len        u32     Longest hash chain observed
 		cachesize at longest     u32     Cache size when longest
 		                                 chain was recorded
 		=======================  ======  ==========================
 
 		Counter fields (cache hits, cache misses, not cached,
-		payload misses, mem usage) are maintained with per-cpu
-		counters and may briefly show stale values under
-		concurrent load. There is no way to reset these
-		counters; consumers should compute rates by sampling
-		over time.
+		mem usage) are maintained with per-cpu counters and
+		may briefly show stale values under concurrent load.
+		There is no way to reset these counters; consumers
+		should compute rates by sampling over time.
 
 		New fields may be appended in future kernels. Parsers
 		should match on field name, not line position.
fs/nfsd/cache.h +2 -7
index 8ad23a1fcb57..6bb57d20da84 100644
@@ -22,9 +22,7 @@ struct nfsd_net;
  */
 struct nfsd_cacherep {
 	struct {
-		/* Keep often-read xid, csum in the same cache line: */
 		__be32			k_xid;
-		__wsum			k_csum;
 		u32			k_proc;
 		u32			k_prot;
 		u32			k_vers;
@@ -80,16 +78,13 @@ enum {
 /* Cache entries expire after this time period */
 #define RC_EXPIRE		(120 * HZ)
 
-/* Checksum this amount of the request */
-#define RC_CSUMLEN		(256U)
-
 svc_ack_cookie_t nfsd_cache_ack_cookie(const struct nfsd_cacherep *rp);
 int	nfsd_drc_slab_create(void);
 void	nfsd_drc_slab_free(void);
 int	nfsd_reply_cache_init(struct nfsd_net *, struct svc_serv *);
 void	nfsd_reply_cache_shutdown(struct nfsd_net *, struct svc_serv *);
-int	nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start,
-			  unsigned int len, struct nfsd_cacherep **cacherep);
+int	nfsd_cache_lookup(struct svc_rqst *rqstp,
+			  struct nfsd_cacherep **cacherep);
 void	nfsd_cache_update(struct svc_rqst *rqstp, struct nfsd_cacherep *rp,
 			  int cachetype, __be32 *statp);
 int	nfsd_reply_cache_stats_show(struct seq_file *m, void *v);
fs/nfsd/netns.h +0 -2
index 0ce7da20aba3..30231b9027dd 100644
@@ -39,8 +39,6 @@ enum nfsd_net_flag {
 };
 
 enum {
-	/* cache misses due only to checksum comparison failures */
-	NFSD_STATS_PAYLOAD_MISSES,
 	/* amount of memory (in bytes) currently consumed by the DRC */
 	NFSD_STATS_DRC_MEM_USAGE,
 	NFSD_STATS_RC_HITS,		/* repcache hits */
fs/nfsd/nfscache.c +21 -82

178 lines not on this page · raw message

fs/nfsd/nfssvc.c +1 -9
index d6687ffbec45..bc3e0e046bc2 100644
@@ -1004,7 +1004,6 @@ int nfsd_dispatch(struct svc_rqst *rqstp)
 	const struct svc_procedure *proc = rqstp->rq_procinfo;
 	__be32 *statp = rqstp->rq_accept_statp;
 	struct nfsd_cacherep *rp;
-	unsigned int start, len;
 	__be32 *nfs_reply;
 
 	/*
@@ -1013,13 +1012,6 @@ int nfsd_dispatch(struct svc_rqst *rqstp)
 	 */
 	ntli->ntli_cachetype = proc->pc_cachetype;
 
-	/*
-	 * ->pc_decode advances the argument stream past the NFS
-	 * Call header, so grab the header's starting location and
-	 * size now for the call to nfsd_cache_lookup().
-	 */
-	start = xdr_stream_pos(&rqstp->rq_arg_stream);
-	len = xdr_stream_remaining(&rqstp->rq_arg_stream);
 	if (!proc->pc_decode(rqstp, &rqstp->rq_arg_stream))
 		goto out_decode_err;
 
@@ -1033,7 +1025,7 @@ int nfsd_dispatch(struct svc_rqst *rqstp)
 	smp_store_release(&rqstp->rq_status_counter, rqstp->rq_status_counter | 1);
 
 	rp = NULL;
-	switch (nfsd_cache_lookup(rqstp, start, len, &rp)) {
+	switch (nfsd_cache_lookup(rqstp, &rp)) {
 	case RC_DOIT:
 		break;
 	case RC_REPLY:
fs/nfsd/stats.h +0 -5
index aabfbb1a9c71..4a556dfbf64a 100644
@@ -97,11 +97,6 @@ static inline void nfsd_stats_io_write_add(struct nfsd_net *nn,
 					 amount);
 }
 
-static inline void nfsd_stats_payload_misses_inc(struct nfsd_net *nn)
-{
-	percpu_counter_inc(&nn->counter[NFSD_STATS_PAYLOAD_MISSES]);
-}
-
 /**
  * nfsd_stats_drc_mem_usage_add - Add memory used by a cache item
  * @nn: target network namespace
fs/nfsd/trace.h +0 -24
index c99b2a369d0d..b9d89a1e1c2e 100644
@@ -1536,30 +1536,6 @@ TRACE_EVENT(nfsd_drc_found,
 
 );
 
-TRACE_EVENT(nfsd_drc_mismatch,
-	TP_PROTO(
-		const struct nfsd_net *nn,
-		const struct nfsd_cacherep *key,
-		const struct nfsd_cacherep *rp
-	),
-	TP_ARGS(nn, key, rp),
-	TP_STRUCT__entry(
-		__field(unsigned long long, boot_time)
-		__field(u32, xid)
-		__field(u32, cached)
-		__field(u32, ingress)
-	),
-	TP_fast_assign(
-		__entry->boot_time = nn->boot_time;
-		__entry->xid = be32_to_cpu(key->c_key.k_xid);
-		__entry->cached = (__force u32)key->c_key.k_csum;
-		__entry->ingress = (__force u32)rp->c_key.k_csum;
-	),
-	TP_printk("boot_time=%16llx xid=0x%08x cached-csum=0x%08x ingress-csum=0x%08x",
-		__entry->boot_time, __entry->xid, __entry->cached,
-		__entry->ingress)
-);
-
 DECLARE_EVENT_CLASS(nfsd_drc_entry_class,
 	TP_PROTO(
 		const struct nfsd_net *nn,
Signature · 1 line
2.55.0
author maintainer
#12 lore raw

PATCH v3 12/12 NFSD: Remove hard cap on duplicate reply cache size

In reply to Chuck Lever

The DRC matters most during a network partition, when clients cannot receive replies and the server must hold them for retransmission after reconnect. Explicit and implied ACK are inactive then, leaving only RC_EXPIRE, and a 256k entry ceiling can be too small for a large client cohort through a long partition. During normal operation the ACK paths keep the cache small regardless of the maximum, so the cap constrains only the failure case, where headroom matters most.

Remove the cap and let the square-root formula govern sizing. It scales sub-linearly with memory, and the hash table already uses kvzalloc, so larger sizes need no physical contiguity. The limit is unchanged at 16 GB and below. At 1 TB the formula yields 1048576 entries, four times the old cap. Even at 1 KB per entry, the worst case the old comment assumed, a full cache is 1 GB, 0.1% of that host's memory.

---
 fs/nfsd/nfscache.c | 31 ++++++++++++++++---------------
 1 file changed, 16 insertions(+), 15 deletions(-)
fs/nfsd/nfscache.c +16 -15
index 4ab6595a0bcd..5adf3c225a4f 100644
@@ -48,34 +48,35 @@ static void	nfsd_reply_ack(void *data, const svc_ack_cookie_t *cookie,
 			       bool delivered);
 
 /*
- * Put a cap on the size of the DRC based on the amount of available
- * low memory in the machine.
+ * Set the size of the DRC based on the amount of available low
+ * memory in the machine.  The sizing formula scales with the
+ * square root of available pages, so growth is sub-linear:
  *
  *  64MB:    8192
- * 128MB:   11585
+ * 128MB:   11584
  * 256MB:   16384
- * 512MB:   23170
+ * 512MB:   23168
  *   1GB:   32768
- *   2GB:   46340
+ *   2GB:   46336
  *   4GB:   65536
- *   8GB:   92681
+ *   8GB:   92672
  *  16GB:  131072
+ *  32GB:  185344
+ *  64GB:  262144
+ * 128GB:  370688
+ * 256GB:  524288
+ * 512GB:  741440
+ *   1TB: 1048576
  *
- * ...with a hard cap of 256k entries. In the worst case, each entry will be
- * ~1k, so the above numbers should give a rough max of the amount of memory
- * used in k.
- *
- * XXX: these limits are per-container, so memory used will increase
- * linearly with number of containers.  Maybe that's OK.
+ * These limits are per-net-namespace, but the per-namespace
+ * shrinker reclaims entries under memory pressure.
  */
 static unsigned int
 nfsd_cache_size_limit(void)
 {
-	unsigned int limit;
 	unsigned long low_pages = totalram_pages() - totalhigh_pages();
 
-	limit = (16 * int_sqrt(low_pages)) << (PAGE_SHIFT-10);
-	return min_t(unsigned int, limit, 256*1024);
+	return (16 * int_sqrt(low_pages)) << (PAGE_SHIFT - 10);
 }
 
 /*
Signature · 1 line
2.55.0
maintainer Reviewed-by on the v3 series
#13 lore raw
In reply to Chuck Lever
Quoted 96 lines · Chuck Lever

A completed DRC entry currently stays in its bucket for 120 seconds whether or not the client already holds the reply. On a busy server those entries lengthen every bucket walk, and under pressure they crowd out entries a retransmit could still hit.

This series modifies the DRC to retire an entry once there is reason to believe its reply was delivered.

1. The transport reports delivery outright where it can. TCP now reports when snd_una passes the reply's sequence number, and RDMA reports on each Send completion.
2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).

UDP continues to use a traditional time-based eviction mechanism.

The first mechanism is the more reliable of the two, and the backup mechanism is more of an informed guess. A pipelined client sends its next request before the previous reply lands, and an entry evicted in that window is lost if the connection then drops and the client retransmits. In the current NFSD code, memory pressure and RC_EXPIRE already open this same window. For instance, on a server with fast networking and storage, pressure eviction already evicts entries far younger than 120 seconds.

Two DRC capacity guards that were sized for a cache full of stale replies can be removed, now that reply delivery reports keep the DRC small. The commit messages for those patches explain the rationale in detail.

v2 of this series (implied ACK only) was profiled with "perf record -e cycles -e cpu-clock -e LLC-load-misses -e branch-misses" during an NFSv3/RDMA 4KB random-write workload. v3 has not been re-profiled. nfsd_cache_lookup overhead dropped from 1.53% to 0.76% of CPU cycles during this test. The rb-tree operations (rb_erase, rb_insert_color) that dominated LLC cache misses fell from a combined 13.2% to 1.1% of all LLC-load-misses, because shorter-lived entries keep the per-bucket trees small.

---
Changes in v3:
- Add the reply-acknowledged callback and its TCP and RDMA reporters
ahead of implied ACK, which now defers to a pending report.
- Split the prune-loop restructure into its own patch.
- Move the eviction tracepoints ahead of the ack patches; the
implied-ACK event now lands with implied ACK.
- Keep err local to the pmap_register block in svc_setup_socket()
(per Jeff's review).
- Never move xpt_last_recv backwards when nfsd threads race.
- Fold the XPT_ORDERED patch into its consumer (per Jeff's review).
- State the re-execution exposure of implied-ACK eviction instead of
calling the DRC advisory (per Jeff's review).
- New patch: remove the DRC request checksum and payload_misses stat.
- New patch: remove the 256k-entry cap on the DRC size.
- Link to v2: https://patch.msgid.link/[email protected]

Changes in v2:
- Print the DRC eviction tracepoints' age field as unsigned.
- Link to v1: https://patch.msgid.link/[email protected]

---
Chuck Lever (12):
SUNRPC: Assign a unique identifier to each svc_xprt
NFSD: Track transport in DRC entries
NFSD: Prepare bucket pruning for out-of-order eviction
NFSD: Add tracepoints for DRC entry eviction
NFSD: Record DRC population in lookup tracepoints
NFSD: Add reply-acknowledged callback infrastructure
SUNRPC: Add TCP sequence-number ACK tracking for reply delivery
svcrdma: Fire reply-acknowledged callback on Send completion
SUNRPC: Record last-request timestamp on svc_xprt
NFSD: Evict unacknowledged DRC entries via implied ACK
NFSD: Remove DRC checksum and payload_misses stat
NFSD: Remove hard cap on duplicate reply cache size

 .../ABI/testing/procfs-nfsd-reply_cache_stats      |  11 +-
 fs/nfsd/cache.h                                    |  20 +-
 fs/nfsd/netns.h                                    |   2 -
 fs/nfsd/nfscache.c                                 | 310 +++++++++++++--------
 fs/nfsd/nfssvc.c                                   |  20 +-
 fs/nfsd/stats.h                                    |   5 -
 fs/nfsd/trace.h                                    |  81 +++++-
 include/linux/sunrpc/svc.h                         |  51 ++++
 include/linux/sunrpc/svc_rdma.h                    |   1 +
 include/linux/sunrpc/svc_xprt.h                    |   6 +-
 include/linux/sunrpc/svcsock.h                     |  10 +
 include/trace/events/sunrpc.h                      |  10 +-
 net/sunrpc/netns.h                                 |   4 +
 net/sunrpc/sunrpc_syms.c                           |   2 +
 net/sunrpc/svc.c                                   |   1 +
 net/sunrpc/svc_xprt.c                              |  53 +++-
 net/sunrpc/svcsock.c                               | 119 +++++++-
 net/sunrpc/xprtrdma/svc_rdma_sendto.c              |  14 +
 net/sunrpc/xprtrdma/svc_rdma_transport.c           |   7 +-
 19 files changed, 541 insertions(+), 186 deletions(-)
Quoted 7 lines

---
base-commit: abf2077ee32058e60d3f49ecab265d3c4bd953d9
change-id: 20260325-duplicate-reply-cache-f0fe7c7b740c

Best regards,

Signature · 1 line
Chuck Lever <[email protected]>

I like the idea overall. I'm still a bit nervous that might see a non- idempotent operations fail occasionally, since this is aggressively expiring them, but let's see how it does.

#14 lore raw
In reply to Chuck Lever
Quoted 11 lines · Chuck Lever

A completed DRC entry currently stays in its bucket for 120 seconds whether or not the client already holds the reply. On a busy server those entries lengthen every bucket walk, and under pressure they crowd out entries a retransmit could still hit.

This series modifies the DRC to retire an entry once there is reason to believe its reply was delivered.

1. The transport reports delivery outright where it can. TCP now reports when snd_una passes the reply's sequence number, and RDMA reports on each Send completion.

As you note in the relevant patch, the TCP ack can arrive before the NFS client has seen the reply. But I don't think you drop the cached reply immediately, so that probably don't matter.
The risk is that the second non-idempotent request might fail (mkdir, but dir already exists), not that it will produce a different positive result (technically possible, but vanishingly so). Once we see the TCP ACK we can be confident the client *will* see that reply (network fault won't cause it not to) so it will mostly likely act on the first reply and ignore the second.

Quoted 3 lines

2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).

I don't follow this. Why would no report come? TCP will always send and ACK.
And I don't read RFC 1813 4.5 as saying a client does NOT retransmit on a live connection, only that it DOES on a re-opened connection. I believe that when an NFS client gets a timeout (6 minutes on TCP?) it will resend without re-opening the TCP connection.

And as you say it is possible (I would say "likely") that a client will have multiple requests outstanding. Even multiple non-idempotent requests. So I cannot see that a later request tells us anything about receiving an earlier reply.

A non-idempotent request often returns a new filehandle, and using the presentation of that filehandle as evidence that the earlier reply was seen would be sound, but I cannot see much other opportunity to draw a connection from later requests to earlier replies.

So I'm perplexed as to why point 2 is needed, or how it is justified.

Thanks,
NeilBrown

Quoted 91 lines

UDP continues to use a traditional time-based eviction mechanism.

The first mechanism is the more reliable of the two, and the backup mechanism is more of an informed guess. A pipelined client sends its next request before the previous reply lands, and an entry evicted in that window is lost if the connection then drops and the client retransmits. In the current NFSD code, memory pressure and RC_EXPIRE already open this same window. For instance, on a server with fast networking and storage, pressure eviction already evicts entries far younger than 120 seconds.

Two DRC capacity guards that were sized for a cache full of stale replies can be removed, now that reply delivery reports keep the DRC small. The commit messages for those patches explain the rationale in detail.

v2 of this series (implied ACK only) was profiled with "perf record -e cycles -e cpu-clock -e LLC-load-misses -e branch-misses" during an NFSv3/RDMA 4KB random-write workload. v3 has not been re-profiled. nfsd_cache_lookup overhead dropped from 1.53% to 0.76% of CPU cycles during this test. The rb-tree operations (rb_erase, rb_insert_color) that dominated LLC cache misses fell from a combined 13.2% to 1.1% of all LLC-load-misses, because shorter-lived entries keep the per-bucket trees small.

---
Changes in v3:
- Add the reply-acknowledged callback and its TCP and RDMA reporters
ahead of implied ACK, which now defers to a pending report.
- Split the prune-loop restructure into its own patch.
- Move the eviction tracepoints ahead of the ack patches; the
implied-ACK event now lands with implied ACK.
- Keep err local to the pmap_register block in svc_setup_socket()
(per Jeff's review).
- Never move xpt_last_recv backwards when nfsd threads race.
- Fold the XPT_ORDERED patch into its consumer (per Jeff's review).
- State the re-execution exposure of implied-ACK eviction instead of
calling the DRC advisory (per Jeff's review).
- New patch: remove the DRC request checksum and payload_misses stat.
- New patch: remove the 256k-entry cap on the DRC size.
- Link to v2: https://patch.msgid.link/[email protected]

Changes in v2:
- Print the DRC eviction tracepoints' age field as unsigned.
- Link to v1: https://patch.msgid.link/[email protected]

---
Chuck Lever (12):
SUNRPC: Assign a unique identifier to each svc_xprt
NFSD: Track transport in DRC entries
NFSD: Prepare bucket pruning for out-of-order eviction
NFSD: Add tracepoints for DRC entry eviction
NFSD: Record DRC population in lookup tracepoints
NFSD: Add reply-acknowledged callback infrastructure
SUNRPC: Add TCP sequence-number ACK tracking for reply delivery
svcrdma: Fire reply-acknowledged callback on Send completion
SUNRPC: Record last-request timestamp on svc_xprt
NFSD: Evict unacknowledged DRC entries via implied ACK
NFSD: Remove DRC checksum and payload_misses stat
NFSD: Remove hard cap on duplicate reply cache size

 .../ABI/testing/procfs-nfsd-reply_cache_stats      |  11 +-
 fs/nfsd/cache.h                                    |  20 +-
 fs/nfsd/netns.h                                    |   2 -
 fs/nfsd/nfscache.c                                 | 310 +++++++++++++--------
 fs/nfsd/nfssvc.c                                   |  20 +-
 fs/nfsd/stats.h                                    |   5 -
 fs/nfsd/trace.h                                    |  81 +++++-
 include/linux/sunrpc/svc.h                         |  51 ++++
 include/linux/sunrpc/svc_rdma.h                    |   1 +
 include/linux/sunrpc/svc_xprt.h                    |   6 +-
 include/linux/sunrpc/svcsock.h                     |  10 +
 include/trace/events/sunrpc.h                      |  10 +-
 net/sunrpc/netns.h                                 |   4 +
 net/sunrpc/sunrpc_syms.c                           |   2 +
 net/sunrpc/svc.c                                   |   1 +
 net/sunrpc/svc_xprt.c                              |  53 +++-
 net/sunrpc/svcsock.c                               | 119 +++++++-
 net/sunrpc/xprtrdma/svc_rdma_sendto.c              |  14 +
 net/sunrpc/xprtrdma/svc_rdma_transport.c           |   7 +-
 19 files changed, 541 insertions(+), 186 deletions(-)

---
base-commit: abf2077ee32058e60d3f49ecab265d3c4bd953d9
change-id: 20260325-duplicate-reply-cache-f0fe7c7b740c

Best regards,

Signature · 1 line
Chuck Lever <[email protected]>
author maintainer
#15 lore raw
Quoted 8 lines · NeilBrown
Quoted 3 lines · Chuck Lever

1. The transport reports delivery outright where it can. TCP now reports when snd_una passes the reply's sequence number, and RDMA reports on each Send completion.

As you note in the relevant patch, the TCP ack can arrive before the NFS client has seen the reply. But I don't think you drop the cached reply immediately, so that probably don't matter.

Right. The ACK marks the entry, and the next prune of that bucket evicts it.

Quoted 6 lines
Quoted 3 lines

2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).

I don't follow this. Why would no report come? TCP will always send and ACK.

The cover letter was unclear. TCP always ACKs, but svc_tcp_sendto() does not always track the reply. The per-socket ring is fixed-size, so a burst overflows it. A kTLS send that leaves part of the record unsent is not recorded, and neither is a failed send. Each of those is reported as untracked right away, so the DRC entry has no report to wait for.

Implied ACK was meant to retire those entries early. Without it, an untracked entry is handled the way every entry is handled today. The untracked entries just sit in their buckets for the full two minutes, lengthening each lookup that hashes there, which is the cost this series set out to remove.

Quoted 4 lines

And I don't read RFC 1813 4.5 as saying a client does NOT retransmit on a live connection, only that it DOES on a re-opened connection. I believe that when an NFS client gets a timeout (6 minutes on TCP?) it will resend without re-opening the TCP connection.

Section 4.5 says that a connection break with automatic reestablishment requires duplicate request processing. I misread that as naming the only way a duplicate reaches a server over a connection-oriented transport, and then reasoned that a request arriving on a live connection could not be one. Re-reading, I see that text supports no such exclusion. It only gives one case that a DRC must handle, and says nothing about what a client does while the connection remains healthy.

Quoted 4 lines

And as you say it is possible (I would say "likely") that a client will have multiple requests outstanding. Even multiple non-idempotent requests. So I cannot see that a later request tells us anything about receiving an earlier reply.

Agreed.

Quoted 1 line

So I'm perplexed as to why point 2 is needed, or how it is justified.

With the explicit reports in place I expect implied ACK to fire
rarely: svcrdma reports every reply, and on TCP only a ring overflow leaves a reply untracked. Shall I drop implied ACK from v4 of the series, measure the untracked rate under load, and size the ring from that instead?

Signature · 1 line
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)
#16 lore raw
Quoted 21 lines · Chuck Lever
Quoted 8 lines · NeilBrown
Quoted 3 lines · Chuck Lever

1. The transport reports delivery outright where it can. TCP now reports when snd_una passes the reply's sequence number, and RDMA reports on each Send completion.

As you note in the relevant patch, the TCP ack can arrive before the NFS client has seen the reply. But I don't think you drop the cached reply immediately, so that probably don't matter.

Right. The ACK marks the entry, and the next prune of that bucket evicts it.

Quoted 6 lines
Quoted 3 lines

2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).

I don't follow this. Why would no report come? TCP will always send and ACK.

Quoted 6 lines

The cover letter was unclear. TCP always ACKs, but svc_tcp_sendto() does not always track the reply. The per-socket ring is fixed-size, so a burst overflows it. A kTLS send that leaves part of the record unsent is not recorded, and neither is a failed send. Each of those is reported as untracked right away, so the DRC entry has no report to wait for.

I don't see the need for the per-socket ring.
When a reply is sent - record the TCP sequence number in the DRC. When pruning the DRC, get the TCP ack number first and compare it against sequence numbers.
Do this often enough that that entries are pruned before the sequence wraps past them.

kTLS is clearly more subtle but there must be some way record an approximate seq number for an incomplete send .. maybe current seq + header-size + queued size ??

Unsent replies will never get an ack, but presumably the client will resend, be answered using the cached reply, and we can then get an ack on the new connection. I think it would be wrong to drop an unsent reply before the timeout, unless it gets re-sent and transport-acked before then.

I think the opaque cookie aspect of the design is a mistake. Everything is fully sequenced over a network connection so a sequence number (clearly paired with xpt_id) makes more sense...
Or does RDMA not sequence replies? If so then it can only ack a single reply at a time, not a range of replies. But that needn't be a barrier.

When a message is transmitted the transport hands a seq-number (and xpt_id) to the cache. When it gets an ack it hands a range of seq-numbers to the cache. For a transport that had out-of-order acks, this range would always be of length one (that transport might need a private atomic_t to generate the sequence numbers). For a transport that batches acks (like TCP) the range could be much longer.

Also, I wonder if the xpt_id should be 64bit assigned sequentially with no xa keeping track of used one. 64bits never wraps.

I think that with this design the implied ACK would add no value.

Thanks,
NeilBrown

author maintainer
#17 lore raw
Quoted 59 lines · NeilBrown
Quoted 21 lines · Chuck Lever
Quoted 8 lines · NeilBrown
Quoted 3 lines · Chuck Lever

1. The transport reports delivery outright where it can. TCP now reports when snd_una passes the reply's sequence number, and RDMA reports on each Send completion.

As you note in the relevant patch, the TCP ack can arrive before the NFS client has seen the reply. But I don't think you drop the cached reply immediately, so that probably don't matter.

Right. The ACK marks the entry, and the next prune of that bucket evicts it.

Quoted 6 lines
Quoted 3 lines

2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).

I don't follow this. Why would no report come? TCP will always send and ACK.

Quoted 6 lines

The cover letter was unclear. TCP always ACKs, but svc_tcp_sendto() does not always track the reply. The per-socket ring is fixed-size, so a burst overflows it. A kTLS send that leaves part of the record unsent is not recorded, and neither is a failed send. Each of those is reported as untracked right away, so the DRC entry has no report to wait for.

I don't see the need for the per-socket ring.
When a reply is sent - record the TCP sequence number in the DRC. When pruning the DRC, get the TCP ack number first and compare it against sequence numbers.
Do this often enough that that entries are pruned before the sequence wraps past them.

kTLS is clearly more subtle but there must be some way record an approximate seq number for an incomplete send .. maybe current seq + header-size + queued size ??

Unsent replies will never get an ack, but presumably the client will resend, be answered using the cached reply, and we can then get an ack on the new connection. I think it would be wrong to drop an unsent reply before the timeout, unless it gets re-sent and transport-acked before then.

I think the opaque cookie aspect of the design is a mistake. Everything is fully sequenced over a network connection so a sequence number (clearly paired with xpt_id) makes more sense...
Or does RDMA not sequence replies? If so then it can only ack a single reply at a time, not a range of replies. But that needn't be a barrier.

When a message is transmitted the transport hands a seq-number (and xpt_id) to the cache. When it gets an ack it hands a range of seq-numbers to the cache. For a transport that had out-of-order acks, this range would always be of length one (that transport might need a private atomic_t to generate the sequence numbers). For a transport that batches acks (like TCP) the range could be much longer.

Quoted 4 lines

Also, I wonder if the xpt_id should be 64bit assigned sequentially with no xa keeping track of used one. 64bits never wraps.

I think that with this design the implied ACK would add no value.

Thanks, all makes sense to me and reduces code complexity.

Signature · 1 line
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)