Skip to contentSkip to conversation

Improve the scalability of NFSD's classic DRC

PATCH Superseded Chuck Lever 1 participant 14 messages Oct 29

Subsystems (4)

v7 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Chuck Lever

author maintainer
#0 lore raw

v7 cover Improve the scalability of NFSD's classic DRC

The NFSD Duplicate Reply Cache evicts cache entries older than 120 seconds (RC_EXPIRE). On an NFS server with more than 64 GB of physical memory, during a steady-state workload the 256k-entry limit on DRC size, and not RC_EXPIRE, governs retention of DRC entries. Cache entries are thus evicted before a retransmit on a connected transport can hit them, since the the default retransmit timeout for such transports is 3 x 60 seconds.

To remedy this situation, this series modifies the DRC eviction algorithm to make an entry eligible for eviction once the RPC transport has seen the client peer acknowledge that entry's reply. This approach is used in several other common NFS server implementations today.

Each network connection publishes two positions: the reply it has just sent and the point up to which the peer has acknowledged. The DRC then compares only those two values to decide on eviction. UDP entries keep the current time-based eviction.

TCP and RDMA transports provide acknowledgement semantics that are precise to the byte. On a TCP-with-TLS connection, byte-level acknowledgement is not available to consumers. Therefore the ACK position advances only while no TLS record is open or parked on a short send, and is withheld while decrypted data waits in the receive path.

---
Changes in v7:
- Count requests in flight per transport, from first read to lookup.
- Withhold the acked position while another request is in flight.
- TCP: hold the position until copied_seq passes the rcv_nxt mark.
- TCP: read snd_una and rcv_nxt under the socket lock.
- Add the svcsock_tcp_acked tracepoint.
- Drop a call whose transport closed before its DRC lookup.
- Explain why svcrdma publishes its position without a receive gate.
- Narrow the reclaimability claim in the cap removal.
- Correct the collision-window claim in the checksum removal.
- New patch: advance the acked position on TLS connections.
- Link to v6: https://patch.msgid.link/[email protected]

Changes in v6:
- Make the DRC size limit independent of page size (Neil)
- Drop same-connection retransmits (Neil)
- Link to v5: https://patch.msgid.link/[email protected]

Changes in v5:
- Actually make xpt_id 64 bits wide (per Neil's review).
- Move the DRC cap removal first (per Neil's review).
- Rewrite the cap removal's description to stand on its own.
- Drop the 4x visit bound from the prune walk (per Neil's review).
- Evict acknowledged entries only where the prune walk reaches them.
- Restore the shrinker's unlimited per-bucket scan.
- Link to v4: https://patch.msgid.link/[email protected]

Changes in v4:
- Track reply positions instead of ack cookies (per Neil's review).
- Make xpt_id a 64-bit counter and drop the xarray (per Neil's review).
- Drop implied ACK and the last-request timestamp (per Neil's review).
- Keep an unsent reply until RC_EXPIRE (per Neil's review).
- Pin a DRC entry against eviction until its reply has been sent.
- svcrdma: Serialize Send posting so positions match completion order.
- Drop the nfsd_drc_reply_acked tracepoint.
- Link to v3: https://patch.msgid.link/[email protected]

Changes in v3:
- Add the reply-acknowledged callback and its TCP and RDMA reporters
ahead of implied ACK, which now defers to a pending report.
- Split the prune-loop restructure into its own patch.
- Move the eviction tracepoints ahead of the ack patches; the
implied-ACK event now lands with implied ACK.
- Keep err local to the pmap_register block in svc_setup_socket()
(per Jeff's review).
- Never move xpt_last_recv backwards when nfsd threads race.
- Fold the XPT_ORDERED patch into its consumer (per Jeff's review).
- State the re-execution exposure of implied-ACK eviction instead of
calling the DRC advisory (per Jeff's review).
- New patch: remove the DRC request checksum and payload_misses stat.
- New patch: remove the 256k-entry cap on the DRC size.
- Link to v2: https://patch.msgid.link/[email protected]

Changes in v2:
- Print the DRC eviction tracepoints' age field as unsigned.
- Link to v1: https://patch.msgid.link/[email protected]

---
Chuck Lever (13):
NFSD: Make the DRC size limit independent of page size
NFSD: Remove hard cap on duplicate reply cache size
SUNRPC: Assign a unique identifier to each svc_xprt
NFSD: Track transport in DRC entries
NFSD: Prepare bucket pruning for additional eviction reasons
NFSD: Add tracepoints for DRC entry eviction
NFSD: Record DRC population in lookup tracepoints
SUNRPC: Publish reply positions for upper-layer consumers
SUNRPC: Publish TCP reply positions
svcrdma: Publish RDMA reply positions
NFSD: Evict acknowledged DRC entries
NFSD: Remove DRC checksum and payload_misses stat
SUNRPC: Advance the acked position on TLS connections

 .../ABI/testing/procfs-nfsd-reply_cache_stats      |  11 +-
 fs/nfsd/cache.h                                    |  16 +-
 fs/nfsd/netns.h                                    |   2 -
 fs/nfsd/nfscache.c                                 | 234 ++++++++++++---------
 fs/nfsd/nfsd.h                                     |   3 +
 fs/nfsd/nfssvc.c                                   |  11 +-
 fs/nfsd/stats.h                                    |   5 -
 fs/nfsd/trace.h                                    |  47 +++--
 include/linux/sunrpc/svc.h                         |  18 ++
 include/linux/sunrpc/svc_rdma.h                    |   5 +-
 include/linux/sunrpc/svc_xprt.h                    |  26 +++
 include/linux/sunrpc/svcsock.h                     |  16 ++
 include/trace/events/sunrpc.h                      |  62 +++++-
 net/sunrpc/svc.c                                   |   3 +
 net/sunrpc/svc_xprt.c                              |  69 +++++-
 net/sunrpc/svcsock.c                               | 175 ++++++++++++++-
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c         |   2 +-
 net/sunrpc/xprtrdma/svc_rdma_sendto.c              |  26 ++-
 net/sunrpc/xprtrdma/svc_rdma_transport.c           |  12 ++
 19 files changed, 587 insertions(+), 156 deletions(-)

---
base-commit: f8e6c07aa05170165c4fefdf044310d4d48cc1bf
change-id: 20260325-duplicate-reply-cache-f0fe7c7b740c

Best regards,

Signature · 1 line
Chuck Lever <[email protected]>
author maintainer
#1 lore raw

PATCH v7 1/13 NFSD: Make the DRC size limit independent of page size

nfsd_cache_size_limit() takes the square root of the low-memory page count and then shifts the result by the page size in KB. The shift belongs under the root. A host with 64 KB pages has a page count 16 times smaller than a 4 KB host with the same memory, so its root is 4 times smaller and its shift 16 times larger, and the limit comes out 4 times larger. With 8 GB of low memory the limit is 92672 entries on 4 KB pages and 370688 on 64 KB pages.

Take the root of the byte count instead. The result matches the table above the function on every page size. Low memory on a 32-bit host is under 1 GB, so the byte count fits in an unsigned long.

Fixes 0338dd157282 ("nfsd: dynamically allocate DRC entries")
---
 fs/nfsd/nfscache.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
fs/nfsd/nfscache.c +1 -1
index 80364b91331a..b43b276a92b3 100644
@@ -73,7 +73,7 @@ nfsd_cache_size_limit(void)
 	unsigned int limit;
 	unsigned long low_pages = totalram_pages() - totalhigh_pages();
 
-	limit = (16 * int_sqrt(low_pages)) << (PAGE_SHIFT-10);
+	limit = int_sqrt(low_pages << PAGE_SHIFT);
 	return min_t(unsigned int, limit, 256*1024);
 }
 
Signature · 1 line
2.56.0
author maintainer
#2 lore raw

PATCH v7 2/13 NFSD: Remove hard cap on duplicate reply cache size

In reply to Chuck Lever

nfsd_cache_size_limit() scales the DRC entry limit with the square root of low memory, then clamps the result at 256k entries. The clamp binds on any host with more than 64 GB of low memory. Once the cache reaches max_drc_entries, nfsd_prune_bucket_locked() evicts oldest-first regardless of age, so on such a host the cap governs retention rather than RC_EXPIRE. A server that completes more than about 2200 calls per second fills 256k entries inside the 120 second RC_EXPIRE window, and from then on entries are evicted before they expire. Those are the entries a retransmit could still hit.

Commit 0338dd157282 ("nfsd: dynamically allocate DRC entries") added the clamp in 2013, when the formula's worst case of 1 KB per entry made 256 MB a reasonable ceiling for the hosts of the day. The formula was already sized to memory; the clamp froze it at that generation of hardware.

Remove the cap and let the square-root formula govern sizing. It scales sub-linearly with memory, and the hash table already uses kvzalloc, so larger sizes need no physical contiguity. The limit is unchanged at 64 GB and below. At 1 TB the formula yields 1048576 entries, four times the old cap. Even at 1 KB per entry, the worst case the old comment assumed, a full cache is 1 GB, 0.1% of that host's memory. Below the limit, the shrinker frees only entries older than RC_EXPIRE, so the unexpired entries are not reclaimable, and each net namespace sizes its own cache.

---
 fs/nfsd/nfscache.c | 24 +++++++++++++-----------
 1 file changed, 13 insertions(+), 11 deletions(-)
fs/nfsd/nfscache.c +13 -11
index b43b276a92b3..391cbca74836 100644
@@ -47,8 +47,8 @@ static unsigned long nfsd_reply_cache_scan(struct shrinker *shrink,
 					   struct shrink_control *sc);
 
 /*
- * Put a cap on the size of the DRC based on the amount of available
- * low memory in the machine.
+ * Size the DRC by the amount of low memory in the machine. The
+ * limit scales with the square root of available memory:
  *
  *  64MB:    8192
  * 128MB:   11585
@@ -59,22 +59,24 @@ static unsigned long nfsd_reply_cache_scan(struct shrinker *shrink,
  *   4GB:   65536
  *   8GB:   92681
  *  16GB:  131072
+ *  32GB:  185363
+ *  64GB:  262144
+ * 128GB:  370727
+ * 256GB:  524288
+ * 512GB:  741455
+ *   1TB: 1048576
  *
- * ...with a hard cap of 256k entries. In the worst case, each entry will be
- * ~1k, so the above numbers should give a rough max of the amount of memory
- * used in k.
- *
- * XXX: these limits are per-container, so memory used will increase
- * linearly with number of containers.  Maybe that's OK.
+ * These limits are per-net-namespace, so memory used increases
+ * linearly with the number of namespaces. Below the limit, the
+ * shrinker frees only entries older than RC_EXPIRE, so unexpired
+ * entries are not reclaimable under memory pressure.
  */
 static unsigned int
 nfsd_cache_size_limit(void)
 {
-	unsigned int limit;
 	unsigned long low_pages = totalram_pages() - totalhigh_pages();
 
-	limit = int_sqrt(low_pages << PAGE_SHIFT);
-	return min_t(unsigned int, limit, 256*1024);
+	return int_sqrt(low_pages << PAGE_SHIFT);
 }
 
 /*
Signature · 1 line
2.56.0
author maintainer
#3 lore raw

PATCH v7 3/13 SUNRPC: Assign a unique identifier to each svc_xprt

In reply to Chuck Lever

A consumer that associates state with a transport without holding a reference cannot detect ABA collisions: once a transport is freed, SLUB may hand out a new svc_xprt at the same address.

Assign each transport a 64-bit identifier from a global sequential counter in svc_xprt_init(). The counter cannot wrap within the lifetime of a transport, so two live transports never share an identifier and a consumer needs no release when the transport is freed. The NFSD duplicate reply cache is the first consumer.

---
 include/linux/sunrpc/svc_xprt.h |  1 +
 include/trace/events/sunrpc.h   |  8 ++++++--
 net/sunrpc/svc_xprt.c           | 17 ++++++++++++++---
 3 files changed, 21 insertions(+), 5 deletions(-)
include/linux/sunrpc/svc_xprt.h +1 -0
index 2af222f3ea2c..52082d8ea6dd 100644
@@ -56,6 +56,7 @@ struct svc_xprt {
 	struct svc_xprt_class	*xpt_class;
 	const struct svc_xprt_ops *xpt_ops;
 	struct kref		xpt_ref;
+	u64			xpt_id;
 	ktime_t			xpt_qtime;
 	struct list_head	xpt_list;
 	struct lwq_node		xpt_ready;
include/trace/events/sunrpc.h +6 -2
index ff855197880d..77efe63d2c14 100644
@@ -1986,7 +1986,8 @@ TRACE_EVENT(svc_xprt_create_err,
 		__sockaddr(server, (x)->xpt_locallen) \
 		__sockaddr(client, (x)->xpt_remotelen) \
 		__field(unsigned long, flags) \
-		__field(unsigned int, netns_ino)
+		__field(unsigned int, netns_ino) \
+		__field(u64, xpt_id)
 
 #define SVC_XPRT_ENDPOINT_ASSIGNMENTS(x) \
 		do { \
@@ -1996,13 +1997,15 @@ TRACE_EVENT(svc_xprt_create_err,
 					  (x)->xpt_remotelen); \
 			__entry->flags = (x)->xpt_flags; \
 			__entry->netns_ino = (x)->xpt_net->ns.inum; \
+			__entry->xpt_id = (x)->xpt_id; \
 		} while (0)
 
 #define SVC_XPRT_ENDPOINT_FORMAT \
-		"server=%pISpc client=%pISpc flags=%s"
+		"server=%pISpc client=%pISpc xpt_id=%llu flags=%s"
 
 #define SVC_XPRT_ENDPOINT_VARARGS \
 		__get_sockaddr(server), __get_sockaddr(client), \
+		__entry->xpt_id, \
 		show_svc_xprt_flags(__entry->flags)
 
 TRACE_EVENT(svc_xprt_enqueue,
@@ -2024,6 +2027,7 @@ TRACE_EVENT(svc_xprt_enqueue,
 				  xprt->xpt_remotelen);
 		__entry->flags = flags;
 		__entry->netns_ino = xprt->xpt_net->ns.inum;
+		__entry->xpt_id = xprt->xpt_id;
 	),
 
 	TP_printk(SVC_XPRT_ENDPOINT_FORMAT, SVC_XPRT_ENDPOINT_VARARGS)
net/sunrpc/svc_xprt.c +14 -3
index 076c624fe675..3eb5dc6cc330 100644
@@ -26,6 +26,8 @@
 static unsigned int svc_rpc_per_connection_limit __read_mostly;
 module_param(svc_rpc_per_connection_limit, uint, 0644);
 
+static atomic64_t svc_xprt_id_seq;
+
 
 static struct svc_deferred_req *svc_deferred_dequeue(struct svc_xprt *xprt);
 static int svc_deferred_recv(struct svc_rqst *rqstp);
@@ -191,9 +193,17 @@ void svc_xprt_put(struct svc_xprt *xprt)
 }
 EXPORT_SYMBOL_GPL(svc_xprt_put);
 
-/*
- * Called by transport drivers to initialize the transport independent
- * portion of the transport instance.
+/**
+ * svc_xprt_init - initialize transport-independent portion of a transport
+ * @net: network namespace in which the transport operates
+ * @xcl: transport class providing operations and metadata
+ * @xprt: svc_xprt to initialize
+ * @serv: RPC service that owns this transport
+ *
+ * @xprt->xpt_id is never reused: a 64-bit counter does not wrap
+ * within the lifetime of a transport.
+ *
+ * Context: Process context. May sleep.
  */
 void svc_xprt_init(struct net *net, struct svc_xprt_class *xcl,
 		   struct svc_xprt *xprt, struct svc_serv *serv)
@@ -211,6 +221,7 @@ void svc_xprt_init(struct net *net, struct svc_xprt_class *xcl,
 	set_bit(XPT_BUSY, &xprt->xpt_flags);
 	xprt->xpt_net = get_net_track(net, &xprt->ns_tracker, GFP_ATOMIC);
 	strcpy(xprt->xpt_remotebuf, "uninitialized");
+	xprt->xpt_id = atomic64_inc_return(&svc_xprt_id_seq);
 }
 EXPORT_SYMBOL_GPL(svc_xprt_init);
 
Signature · 1 line
2.56.0
author maintainer
#4 lore raw

PATCH v7 4/13 NFSD: Track transport in DRC entries

In reply to Chuck Lever

A cached reply carries no record of the transport its request arrived on. Transport-aware eviction needs that association.

Record the xprt id in each entry. struct nfsd_cacherep grows from 128 to 136 bytes. No behavior change.

---
 fs/nfsd/cache.h    | 1 +
 fs/nfsd/nfscache.c | 1 +
 2 files changed, 2 insertions(+)
fs/nfsd/cache.h +1 -0
index 3bc4856e34b8..589ffe4762d5 100644
@@ -37,6 +37,7 @@ struct nfsd_cacherep {
 	unsigned char		c_state,	/* unused, inprog, done */
 				c_type,		/* status, buffer */
 				c_secure : 1;	/* req came from port < 1024 */
+	u64			c_xprt;		/* svc_xprt that carried req */
 	unsigned long		c_timestamp;
 	union {
 		struct kvec	u_vec;
fs/nfsd/nfscache.c +1 -0
index 391cbca74836..01ad6a9ebe1f 100644
@@ -111,6 +111,7 @@ nfsd_cacherep_alloc(struct svc_rqst *rqstp, __wsum csum,
 		rp->c_key.k_vers = rqstp->rq_vers;
 		rp->c_key.k_len = rqstp->rq_arg.len;
 		rp->c_key.k_csum = csum;
+		rp->c_xprt = rqstp->rq_xprt->xpt_id;
 	}
 	return rp;
 }
Signature · 1 line
2.56.0
author maintainer
#5 lore raw

PATCH v7 5/13 NFSD: Prepare bucket pruning for additional eviction reasons

In reply to Chuck Lever

nfsd_prune_bucket_locked() combines its two eviction reasons, memory pressure and RC_EXPIRE, in a single test and stops the walk at the first entry the test does not evict. A reason that needs its own handling, a tracepoint per reason or a test that must run ahead of the stop, has nowhere to go.

Test each reason on its own and jump to a shared eviction label. The walk still stops at the first entry no reason applies to.

---
 fs/nfsd/nfscache.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
fs/nfsd/nfscache.c +6 -3
index 01ad6a9ebe1f..988b955590cc 100644
@@ -274,10 +274,13 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
 
 	/* The bucket LRU is ordered oldest-first. */
 	list_for_each_entry_safe(rp, tmp, &b->lru_head, c_lru) {
-		if (atomic_read(&nn->num_drc_entries) <= nn->max_drc_entries &&
-		    time_before(expiry, rp->c_timestamp))
-			break;
+		if (atomic_read(&nn->num_drc_entries) > nn->max_drc_entries)
+			goto evict;
+		if (time_before_eq(rp->c_timestamp, expiry))
+			goto evict;
+		break;
 
+evict:
 		nfsd_cacherep_unlink_locked(nn, b, rp);
 		list_add(&rp->c_lru, dispose);
 
Signature · 1 line
2.56.0
author maintainer
#6 lore raw

PATCH v7 6/13 NFSD: Add tracepoints for DRC entry eviction

In reply to Chuck Lever

The DRC pruning path evicts entries for memory pressure and expiry, and neither emits a trace event, so there is no way to see which reason is retiring entries or how old they are when it happens. A further eviction reason needs the same visibility.

Add an event class that records the cache population, the XID, the transport, and the entry's age, and define an event for each eviction reason.

---
 fs/nfsd/nfscache.c |  8 ++++++--
 fs/nfsd/trace.h    | 37 +++++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 2 deletions(-)
fs/nfsd/nfscache.c +6 -2
index 988b955590cc..680025334f1f 100644
@@ -274,10 +274,14 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
 
 	/* The bucket LRU is ordered oldest-first. */
 	list_for_each_entry_safe(rp, tmp, &b->lru_head, c_lru) {
-		if (atomic_read(&nn->num_drc_entries) > nn->max_drc_entries)
+		if (atomic_read(&nn->num_drc_entries) > nn->max_drc_entries) {
+			trace_nfsd_drc_evict_pressure(nn, rp);
 			goto evict;
-		if (time_before_eq(rp->c_timestamp, expiry))
+		}
+		if (time_before_eq(rp->c_timestamp, expiry)) {
+			trace_nfsd_drc_evict_expired(nn, rp);
 			goto evict;
+		}
 		break;
 
 evict:
fs/nfsd/trace.h +37 -0
index e7b186307c21..e19e7075d879 100644
@@ -1708,6 +1708,43 @@ TRACE_EVENT(nfsd_drc_mismatch,
 		__entry->ingress)
 );
 
+DECLARE_EVENT_CLASS(nfsd_drc_entry_class,
+	TP_PROTO(
+		const struct nfsd_net *nn,
+		const struct nfsd_cacherep *rp
+	),
+	TP_ARGS(nn, rp),
+	TP_STRUCT__entry(
+		__field(unsigned long long, boot_time)
+		__field(unsigned int, num_drc_entries)
+		__field(u32, xid)
+		__field(u64, xprt)
+		__field(unsigned long, age)
+	),
+	TP_fast_assign(
+		__entry->boot_time = nn->boot_time;
+		__entry->num_drc_entries = atomic_read(&nn->num_drc_entries);
+		__entry->xid = be32_to_cpu(rp->c_key.k_xid);
+		__entry->xprt = rp->c_xprt;
+		__entry->age = time_is_after_jiffies(rp->c_timestamp) ?
+				1 : jiffies - rp->c_timestamp;
+	),
+	TP_printk("boot_time=%16llx entries=%u xid=0x%08x xprt=%llu age=%lu",
+		__entry->boot_time, __entry->num_drc_entries,
+		__entry->xid, __entry->xprt, __entry->age)
+);
+
+#define DEFINE_NFSD_DRC_ENTRY_EVENT(name)			\
+DEFINE_EVENT(nfsd_drc_entry_class, nfsd_drc_##name,		\
+	TP_PROTO(						\
+		const struct nfsd_net *nn,			\
+		const struct nfsd_cacherep *rp			\
+	),							\
+	TP_ARGS(nn, rp))
+
+DEFINE_NFSD_DRC_ENTRY_EVENT(evict_pressure);
+DEFINE_NFSD_DRC_ENTRY_EVENT(evict_expired);
+
 TRACE_EVENT(nfsd_cb_args,
 	TP_PROTO(
 		const struct nfs4_client *clp,
Signature · 1 line
2.56.0
author maintainer
#7 lore raw

PATCH v7 7/13 NFSD: Record DRC population in lookup tracepoints

In reply to Chuck Lever

nfsd_drc_found reports the outcome of a lookup but not how full the cache was at the time, so a trace cannot show whether retransmits are being caught while the cache runs near its cap.

---
 fs/nfsd/nfscache.c |  4 +++-
 fs/nfsd/trace.h    | 11 +++++++----
 2 files changed, 10 insertions(+), 5 deletions(-)
fs/nfsd/nfscache.c +3 -1
index 680025334f1f..453ff718ff6c 100644
@@ -486,6 +486,7 @@ int nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start,
 	struct nfsd_drc_bucket	*b;
 	int type = ntli->ntli_cachetype;
 	LIST_HEAD(dispose);
+	unsigned int entries;
 	int rtn = RC_DOIT;
 
 	if (type == RC_NOCACHE) {
@@ -555,7 +556,8 @@ int nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start,
 	}
 
 out_trace:
-	trace_nfsd_drc_found(nn, rqstp, rtn);
+	entries = atomic_read(&nn->num_drc_entries);
+	trace_nfsd_drc_found(nn, entries, rqstp, rtn);
 out_unlock:
 	spin_unlock(&b->cache_lock);
 out:
fs/nfsd/trace.h +7 -4
index e19e7075d879..b1ee0f468206 100644
@@ -1664,23 +1664,26 @@ TRACE_DEFINE_ENUM(RC_DOIT);
 TRACE_EVENT(nfsd_drc_found,
 	TP_PROTO(
 		const struct nfsd_net *nn,
+		unsigned int num_drc_entries,
 		const struct svc_rqst *rqstp,
 		int result
 	),
-	TP_ARGS(nn, rqstp, result),
+	TP_ARGS(nn, num_drc_entries, rqstp, result),
 	TP_STRUCT__entry(
 		__field(unsigned long long, boot_time)
+		__field(unsigned int, num_drc_entries)
 		__field(unsigned long, result)
 		__field(u32, xid)
 	),
 	TP_fast_assign(
 		__entry->boot_time = nn->boot_time;
+		__entry->num_drc_entries = num_drc_entries;
 		__entry->result = result;
 		__entry->xid = be32_to_cpu(rqstp->rq_xid);
 	),
-	TP_printk("boot_time=%16llx xid=0x%08x result=%s",
-		__entry->boot_time, __entry->xid,
-		show_drc_retval(__entry->result))
+	TP_printk("boot_time=%16llx entries=%u xid=0x%08x result=%s",
+		__entry->boot_time, __entry->num_drc_entries,
+		__entry->xid, show_drc_retval(__entry->result))
 
 );
 
Signature · 1 line
2.56.0
author maintainer
#8 lore raw

PATCH v7 8/13 SUNRPC: Publish reply positions for upper-layer consumers

In reply to Chuck Lever

An RPC server's duplicate reply cache has to hold a cached reply until the client has received it. SUNRPC gives an upper layer no way to learn that, so a cache entry lives for the full retention interval even when the reply was delivered at once.

Add two transport-neutral values that an upper layer can compare:
the position of each reply on its transport, recorded in the svc_rqst by xpo_sendto(), and the position up to which the peer has acknowledged, recorded on the svc_xprt. The units are private to the transport (bytes for a stream, Send sequence for RDMA); a consumer learns only whether one is less than or equal to the other. A reply that xpo_sendto() does not hand to the wire keeps position zero and is never reported as acknowledged.

The acknowledged position is an atomic64_t. A transport publishes it outside any lock the consumer holds, and a torn 64-bit read on a 32-bit host could run ahead of the peer. A transport raises the position through svc_xprt_advance_acked(), a plain read and store that computes a maximum only while one context per transport writes. A cmpxchg loop would make the helper safe from any context at the cost of a locked instruction on every acknowledgment.

An acknowledgment proves the peer holds the reply, so no retransmit of that call follows. A retransmit the peer sent before the reply reached it precedes the acknowledgment in the byte stream and can still be ahead of its lookup when the transport refreshes the
position: unread in the socket, or read and parked in svc_authenticate(), which can sleep for seconds in the ip_map upcall or defer the request. A consumer that evicts on the refreshed position sends the retransmit through the procedure again.

Count a transport's requests from first read until the consumer reports the lookup with svc_xprt_recv_done(), and carry the count across a deferral. Publish the position through svc_xprt_acked_pos(), so the transport can withhold the position while any request but the caller's is counted. A request that never reaches a lookup is uncounted when it is released.

Add a per-service hook that runs once per request when its reply phase ends, whether svc_send() sent the reply or svc_process() dropped it, so an upper layer can record the reply's position against its own state either way. On the drop path the position is zero.

---
 include/linux/sunrpc/svc.h      | 18 ++++++++++++++
 include/linux/sunrpc/svc_xprt.h | 25 ++++++++++++++++++++
 net/sunrpc/svc.c                |  3 +++
 net/sunrpc/svc_xprt.c           | 52 +++++++++++++++++++++++++++++++++++++++++
 4 files changed, 98 insertions(+)
include/linux/sunrpc/svc.h +18 -0
index b1dff5c8aa89..fc83b24934b0 100644
@@ -59,6 +59,8 @@ enum {
 };
 
 
+struct svc_rqst;
+
 /*
  * RPC service.
  *
@@ -101,6 +103,13 @@ struct svc_serv {
 						 * connection */
 	bool			sv_bc_enabled;	/* service uses backchannel */
 #endif /* CONFIG_SUNRPC_BACKCHANNEL */
+
+	/*
+	 * Called once per request after its reply phase, whether
+	 * xpo_sendto() succeeded, failed, or was never reached because
+	 * svc_process() dropped the reply. See rq_reply_pos.
+	 */
+	void			(*sv_reply_sent)(struct svc_rqst *rqstp);
 };
 
 /* This is used by pool_stats to find and lock an svc */
@@ -272,6 +281,13 @@ struct svc_rqst {
 	unsigned int		bc_to_retries;
 	unsigned int		rq_status_counter; /* RPC processing counter */
 	void			*rq_private;	/* For use by the service thread */
+
+	/*
+	 * Transport position of this request's reply, set by
+	 * xpo_sendto(); zero when the reply did not reach the wire.
+	 * Comparable only with the xpt_acked_pos of the same svc_xprt.
+	 */
+	u64			rq_reply_pos;
 };
 
 /* bits for rq_flags */
@@ -282,6 +298,7 @@ enum {
 	RQ_DROPME,		/* drop current reply */
 	RQ_VICTIM,		/* Have agreed to shut down */
 	RQ_DATA,		/* request has data */
+	RQ_RECV_PENDING,	/* counted in xpt_recv_pending */
 };
 
 #define SVC_NET(rqst) (rqst->rq_xprt ? rqst->rq_xprt->xpt_net : rqst->rq_bc_net)
@@ -364,6 +381,7 @@ static inline void svc_thread_init_status(struct svc_rqst *rqstp, int err)
 struct svc_deferred_req {
 	u32			prot;	/* protocol (UDP or TCP) */
 	bool			secure;	/* RQ_SECURE of the original request */
+	bool			recv_pending; /* RQ_RECV_PENDING, carried over */
 	struct svc_xprt		*xprt;
 	struct sockaddr_storage	addr;	/* where reply must go */
 	size_t			addrlen;
include/linux/sunrpc/svc_xprt.h +25 -0
index 52082d8ea6dd..aea98ddbfe52 100644
@@ -28,6 +28,7 @@ struct svc_xprt_ops {
 	void		(*xpo_free)(struct svc_xprt *);
 	void		(*xpo_kill_temp_xprt)(struct svc_xprt *);
 	void		(*xpo_handshake)(struct svc_xprt *xprt);
+	u64		(*xpo_acked_pos)(struct svc_rqst *rqstp);
 };
 
 struct svc_xprt_class {
@@ -66,6 +67,10 @@ struct svc_xprt {
 	atomic_t		xpt_reserved;	/* outq space rsvd, UDP only */
 	atomic_t		xpt_nr_rqsts;	/* Number of requests */
 	struct mutex		xpt_mutex;	/* to serialize sending data */
+	atomic64_t		xpt_acked_pos;	/* replies up to this position
+						 * are acknowledged; 0 = none */
+	/* requests read but not yet past svc_xprt_recv_done() */
+	atomic_t		xpt_recv_pending;
 	spinlock_t		xpt_lock;	/* protects sk_deferred
 						 * and xpt_auth_cache */
 	void			*xpt_auth_cache;/* auth cache */
@@ -161,6 +166,24 @@ static inline bool svc_xprt_is_dead(const struct svc_xprt *xprt)
 		(test_bit(XPT_CLOSE, &xprt->xpt_flags) != 0);
 }
 
+/**
+ * svc_xprt_advance_acked - Raise a transport's acknowledged position
+ * @xprt: transport that sent the replies
+ * @pos: position the peer has acknowledged, in @xprt's own units
+ *
+ * Context: Any context. Callers must serialize against other writers
+ *	of @xprt's xpt_acked_pos.
+ * Return: %true if the position was raised, %false if @pos does not
+ *	exceed the recorded position.
+ */
+static inline bool svc_xprt_advance_acked(struct svc_xprt *xprt, u64 pos)
+{
+	if (pos <= atomic64_read(&xprt->xpt_acked_pos))
+		return false;
+	atomic64_set(&xprt->xpt_acked_pos, pos);
+	return true;
+}
+
 int	svc_reg_xprt_class(struct svc_xprt_class *);
 void	svc_unreg_xprt_class(struct svc_xprt_class *);
 void	svc_xprt_init(struct net *, struct svc_xprt_class *, struct svc_xprt *,
@@ -175,6 +198,8 @@ int	svc_xprt_create(struct svc_serv *serv, const char *xprt_name,
 void	svc_xprt_destroy_all(struct svc_serv *serv, struct net *net,
 			     bool unregister);
 void	svc_xprt_received(struct svc_xprt *xprt);
+void	svc_xprt_recv_done(struct svc_rqst *rqstp);
+u64	svc_xprt_acked_pos(struct svc_rqst *rqstp);
 void	svc_xprt_enqueue(struct svc_xprt *xprt);
 void	svc_xprt_put(struct svc_xprt *xprt);
 void	svc_xprt_copy_addrs(struct svc_rqst *rqstp, struct svc_xprt *xprt);
net/sunrpc/svc.c +3 -0
index af8d87c1de83..d4b8aa685cfe 100644
@@ -1497,6 +1497,7 @@ svc_process_common(struct svc_rqst *rqstp)
 
 	/* Reset the accept_stat for the RPC */
 	rqstp->rq_accept_statp = NULL;
+	rqstp->rq_reply_pos = 0;
 
 	/* Will be turned off only when NFSv4 Sessions are used */
 	set_bit(RQ_USEDEFERRAL, &rqstp->rq_flags);
@@ -1742,6 +1743,8 @@ void svc_process(struct svc_rqst *rqstp)
 		goto out_baddir;
 
 	if (!svc_process_common(rqstp)) {
+		if (rqstp->rq_server->sv_reply_sent)
+			rqstp->rq_server->sv_reply_sent(rqstp);
 		svc_release_rqst(rqstp);
 		goto out_drop;
 	}
net/sunrpc/svc_xprt.c +52 -0

84 lines not on this page · raw message

Signature · 1 line
2.56.0
author maintainer
#9 lore raw

PATCH v7 9/13 SUNRPC: Publish TCP reply positions

In reply to Chuck Lever

Add the svcsock side of reply-position reporting. A reply's position is the running count of bytes handed to the socket, kept per socket under xpt_mutex. A TCP sequence number cannot serve as the position:
the 32-bit space wraps in a few hundred milliseconds at 40 GbE, and a duplicate reply cache entry lives for two minutes.

The acknowledged position is derived from the socket's unacknowledged byte count, write_seq minus snd_una, after each successful send. Refreshing the position only in the send path means a consumer sees the acknowledgment state as of the previous reply on that connection, stale by one round trip but never ahead of the peer. A failed or refused send leaves rq_reply_pos at zero, so the reply is never reported as acknowledged.

A call the peer sent before it acknowledged a reply precedes the acknowledgment in the stream, so the socket has received that call by the time the acknowledgment raises the position. Record rcv_nxt as a mark whenever the position rises, and withhold the position from svc_tcp_acked_pos() until copied_seq has reached the mark and no request but the caller's is counted in xpt_recv_pending. A record in progress is counted from its first octet, so a partially read retransmit withholds the position too. The socket lock keeps snd_una and rcv_nxt from moving between the two reads and orders the pair against the reader. Each cacheable call takes the lock once, after the first position is published, and on a busy connection the position lags a refresh by the time the requests ahead of the mark take to reach their lookups.

The mark covers a retransmit TCP has delivered in order. When a segment of the retransmit is lost on the wire, the acknowledgment the peer sent after that segment arrives first. TCP processes the acknowledgment, so snd_una covers the reply while rcv_nxt stops short of the retransmit, and the socket does not record the gap. That retransmit stays exposed until TCP delivers the lost segment.

Under kTLS the socket counts ciphertext while svcsock counts plaintext, and tls_sw_sendmsg() can return the full plaintext count with part of a record still waiting for socket write space. The unacknowledged count then understates what the peer has yet to receive. Whether a record is pending is state private to net/tls, so a TLS session publishes no acknowledged position and its replies are retained for the full interval.

The svcsock_tcp_send tracepoint fires before the position is updated and logs only the byte count, so a trace cannot show whether xpt_acked_pos ever claimed more than the client had acknowledged. Add svcsock_tcp_acked, recording write_seq, snd_una, and sk_send_pos, the candidate position they yield, and the position published after the monotonic clamp. The clamp can hold the published position above the candidate, so the record carries both. xpt_id separates samples from successive connections that share a remote address.

---
 include/linux/sunrpc/svcsock.h |   7 +++
 include/trace/events/sunrpc.h  |  39 +++++++++++++++
 net/sunrpc/svcsock.c           | 109 ++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 154 insertions(+), 1 deletion(-)
include/linux/sunrpc/svcsock.h +7 -0
index 372a00882ca6..0fa62f9c6465 100644
@@ -38,9 +38,16 @@ struct svc_sock {
 	/* Total length of the data (not including fragment headers)
 	 * received so far in the fragments making up this rpc: */
 	u32			sk_datalen;
+	/* The record in progress is counted in xpt_recv_pending */
+	bool			sk_recv_counted;
 
 	struct page_frag_cache  sk_frag_cache;
 
+	/* reply bytes handed to the socket; protected by xpt_mutex */
+	u64			sk_send_pos;
+	/* rcv_nxt when xpt_acked_pos was last raised; socket lock */
+	u32			sk_acked_rcv;
+
 	struct completion	sk_handshake_done;
 
 	/* received data */
include/trace/events/sunrpc.h +39 -0
index 77efe63d2c14..a5d914376627 100644
@@ -2349,6 +2349,45 @@ TRACE_EVENT(svcsock_tcp_recv_short,
 	)
 );
 
+TRACE_EVENT(svcsock_tcp_acked,
+	TP_PROTO(
+		const struct svc_xprt *xprt,
+		u64 send_pos,
+		u32 write_seq,
+		u32 snd_una,
+		u64 candidate,
+		u64 acked_pos
+	),
+
+	TP_ARGS(xprt, send_pos, write_seq, snd_una, candidate, acked_pos),
+
+	TP_STRUCT__entry(
+		__field(u64, xpt_id)
+		__field(u64, send_pos)
+		__field(u64, candidate)
+		__field(u64, acked_pos)
+		__field(u32, write_seq)
+		__field(u32, snd_una)
+		__string(addr, xprt->xpt_remotebuf)
+	),
+
+	TP_fast_assign(
+		__entry->xpt_id = xprt->xpt_id;
+		__entry->send_pos = send_pos;
+		__entry->candidate = candidate;
+		__entry->acked_pos = acked_pos;
+		__entry->write_seq = write_seq;
+		__entry->snd_una = snd_una;
+		__assign_str(addr);
+	),
+
+	TP_printk("addr=%s xpt_id=%llu send_pos=%llu write_seq=%u snd_una=%u candidate=%llu acked_pos=%llu",
+		__get_str(addr), __entry->xpt_id, __entry->send_pos,
+		__entry->write_seq, __entry->snd_una, __entry->candidate,
+		__entry->acked_pos
+	)
+);
+
 TRACE_EVENT(svcsock_tcp_state,
 	TP_PROTO(
 		const struct svc_xprt *xprt,
net/sunrpc/svcsock.c +108 -1

149 lines not on this page · raw message

Signature · 1 line
2.56.0
author maintainer
#10 lore raw

PATCH v7 10/13 svcrdma: Publish RDMA reply positions

In reply to Chuck Lever

Add the svcrdma side of reply-position reporting. A reply's position is the sequence number of its Send on the transport, and the Send completion for that reply publishes the same number as the transport's acknowledged position. A successful Send completion means the peer's HCA has acknowledged the message, so a reply whose Send has completed is in the client's receive buffer.

Completions on one queue pair arrive in posting order. Several nfsd threads post on one queue pair with no serialization above the provider, so the order in which Sends are queued is decided inside the provider. Assign the sequence number and post the Send under one lock so positions match queuing order. ib_post_send() does not sleep; the Send Queue wait stays outside the lock.

Only a posted RPC Reply reports a position. The RDMA_ERROR message and backchannel calls post with no position pointer, and a Send that fails to post leaves rq_reply_pos at zero.

svcrdma publishes the position as the Send completion raised it, with neither the request count nor the receive mark that svcsock keeps. xprtrdma does not retransmit on a live connection:
xprt_rdma_timer() disconnects on an RPC timeout, and the retransmit arrives on a new transport, whose lookups do not evict the old transport's entries. RFC 8166 does not forbid a retransmit on the same connection. A client that sends one needs the mark as well as the count, and svcrdma has no rcv_nxt to record. Send and Receive completions arrive on separate queues with no common lock, so the mark needs an ordering between the two completion handlers that svcrdma does not provide. The gate waits for a client that retransmits on a live connection.

---
 include/linux/sunrpc/svc_rdma.h            |  5 ++++-
 net/sunrpc/xprtrdma/svc_rdma_backchannel.c |  2 +-
 net/sunrpc/xprtrdma/svc_rdma_sendto.c      | 26 +++++++++++++++++++++++---
 net/sunrpc/xprtrdma/svc_rdma_transport.c   | 12 ++++++++++++
 4 files changed, 40 insertions(+), 5 deletions(-)
include/linux/sunrpc/svc_rdma.h +4 -1
index 60966f92e8c8..71bd5bcc5ce2 100644
@@ -96,6 +96,8 @@ struct svcxprt_rdma {
 
 	spinlock_t	     sc_send_lock;
 	struct llist_head    sc_send_ctxts;
+	spinlock_t	     sc_post_lock;	/* orders Send posting */
+	u64		     sc_post_seq;	/* Sends posted so far */
 	spinlock_t	     sc_rw_ctxt_lock;
 	struct llist_head    sc_rw_ctxts;
 
@@ -244,6 +246,7 @@ struct svc_rdma_send_ctxt {
 	struct ib_send_wr	*sc_wr_chain;
 	int			sc_sqecount;
 	struct ib_cqe		sc_cqe;
+	u64			sc_pos;
 	struct xdr_buf		sc_hdrbuf;
 	struct xdr_stream	sc_stream;
 
@@ -309,7 +312,7 @@ extern struct svc_rdma_send_ctxt *
 extern void svc_rdma_send_ctxt_put(struct svcxprt_rdma *rdma,
 				   struct svc_rdma_send_ctxt *ctxt);
 extern int svc_rdma_post_send(struct svcxprt_rdma *rdma,
-			      struct svc_rdma_send_ctxt *ctxt);
+			      struct svc_rdma_send_ctxt *ctxt, u64 *pos);
 extern int svc_rdma_map_reply_msg(struct svcxprt_rdma *rdma,
 				  struct svc_rdma_send_ctxt *sctxt,
 				  const struct svc_rdma_pcl *write_pcl,
net/sunrpc/xprtrdma/svc_rdma_backchannel.c +1 -1
index e0768d1ee556..20b7dfc20d2b 100644
@@ -94,7 +94,7 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma,
 	sctxt->sc_page_count = 1;
 
 	sctxt->sc_send_wr.opcode = IB_WR_SEND;
-	return svc_rdma_post_send(rdma, sctxt);
+	return svc_rdma_post_send(rdma, sctxt, NULL);
 }
 
 /* Server-side transport endpoint wants a whole page for its send
net/sunrpc/xprtrdma/svc_rdma_sendto.c +23 -3
index b9eee2150ea1..07df06480294 100644
@@ -222,6 +222,7 @@ struct svc_rdma_send_ctxt *svc_rdma_send_ctxt_get(struct svcxprt_rdma *rdma)
 	ctxt->sc_page_count = 0;
 	ctxt->sc_wr_chain = &ctxt->sc_send_wr;
 	ctxt->sc_sqecount = 1;
+	ctxt->sc_pos = 0;
 
 	return ctxt;
 
@@ -491,6 +492,7 @@ static void svc_rdma_wc_send(struct ib_cq *cq, struct ib_wc *wc)
 		goto flushed;
 
 	trace_svcrdma_wc_send(&ctxt->sc_cid);
+	svc_xprt_advance_acked(&rdma->sc_xprt, ctxt->sc_pos);
 	svc_rdma_send_ctxt_put(rdma, ctxt);
 	return;
 
@@ -507,24 +509,30 @@ static void svc_rdma_wc_send(struct ib_cq *cq, struct ib_wc *wc)
  * svc_rdma_post_send - Post a WR chain to the Send Queue
  * @rdma: transport context
  * @ctxt: WR chain to post
+ * @pos: OUT: position of this Send on @rdma, or NULL
  *
  * Copy fields in @ctxt to stack variables in order to guarantee
  * that these values remain available after the ib_post_send() call.
  * In some error flow cases, svc_rdma_wc_send() releases @ctxt.
  *
+ * @pos is written only when the chain was posted. Positions on one
+ * transport increase in posting order, and the Send completion of
+ * @ctxt publishes its position in xpt_acked_pos.
+ *
  * Return values:
  *   %0: The transport owns @ctxt; a Send completion or the
  *       transport destructor releases it
  *   %-ENOTCONN: The connection was lost
  */
 int svc_rdma_post_send(struct svcxprt_rdma *rdma,
-		       struct svc_rdma_send_ctxt *ctxt)
+		       struct svc_rdma_send_ctxt *ctxt, u64 *pos)
 {
 	struct ib_send_wr *first_wr = ctxt->sc_wr_chain;
 	struct ib_send_wr *send_wr = &ctxt->sc_send_wr;
 	const struct ib_send_wr *bad_wr = first_wr;
 	struct rpc_rdma_cid cid = ctxt->sc_cid;
 	int ret, sqecount = ctxt->sc_sqecount;
+	u64 seq;
 
 	might_sleep();
 
@@ -539,7 +547,17 @@ int svc_rdma_post_send(struct svcxprt_rdma *rdma,
 		return ret;
 
 	trace_svcrdma_post_send(ctxt);
+
+	/*
+	 * Assign the position and post under one lock so positions
+	 * match the order the provider queues the Sends, and thus
+	 * completion order.
+	 */
+	spin_lock(&rdma->sc_post_lock);
+	seq = ++rdma->sc_post_seq;
+	ctxt->sc_pos = seq;
 	ret = ib_post_send(rdma->sc_qp, first_wr, &bad_wr);
+	spin_unlock(&rdma->sc_post_lock);
 	if (ret) {
 		ret = svc_rdma_post_send_err(rdma, &cid, bad_wr,
 					     first_wr, sqecount, ret);
@@ -552,6 +570,8 @@ int svc_rdma_post_send(struct svcxprt_rdma *rdma,
 		llist_add(&ctxt->sc_node, &rdma->sc_send_stranded_ctxts);
 		return 0;
 	}
+	if (pos)
+		*pos = seq;
 	return 0;
 }
 
@@ -1082,7 +1102,7 @@ static int svc_rdma_send_reply_msg(struct svcxprt_rdma *rdma,
 		send_wr->opcode = IB_WR_SEND;
 	}
 
-	return svc_rdma_post_send(rdma, sctxt);
+	return svc_rdma_post_send(rdma, sctxt, &rqstp->rq_reply_pos);
 }
 
 /**
@@ -1152,7 +1172,7 @@ void svc_rdma_send_error_msg(struct svcxprt_rdma *rdma,
 	 */
 	sctxt->sc_wr_chain = &sctxt->sc_send_wr;
 	sctxt->sc_sqecount = 1;
-	if (svc_rdma_post_send(rdma, sctxt))
+	if (svc_rdma_post_send(rdma, sctxt, NULL))
 		goto put_ctxt;
 	return;
 
net/sunrpc/xprtrdma/svc_rdma_transport.c +12 -0
index df6c3501f97c..7f0c66c8bfd3 100644
@@ -77,6 +77,7 @@ static void svc_rdma_detach(struct svc_xprt *xprt);
 static void svc_rdma_free(struct svc_xprt *xprt);
 static int svc_rdma_has_wspace(struct svc_xprt *xprt);
 static void svc_rdma_kill_temp_xprt(struct svc_xprt *);
+static u64 svc_rdma_acked_pos(struct svc_rqst *rqstp);
 
 static const struct svc_xprt_ops svc_rdma_ops = {
 	.xpo_create = svc_rdma_create,
@@ -89,6 +90,7 @@ static const struct svc_xprt_ops svc_rdma_ops = {
 	.xpo_has_wspace = svc_rdma_has_wspace,
 	.xpo_accept = svc_rdma_accept,
 	.xpo_kill_temp_xprt = svc_rdma_kill_temp_xprt,
+	.xpo_acked_pos = svc_rdma_acked_pos,
 };
 
 struct svc_xprt_class svc_rdma_class = {
@@ -209,6 +211,7 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,
 	lockdep_set_class(&cma_xprt->sc_send_lock, &svcrdma_sctx_lock);
 	spin_lock_init(&cma_xprt->sc_rw_ctxt_lock);
 	lockdep_set_class(&cma_xprt->sc_rw_ctxt_lock, &svcrdma_rwctx_lock);
+	spin_lock_init(&cma_xprt->sc_post_lock);
 
 	/*
 	 * Note that this implies that the underlying transport support
@@ -724,6 +727,15 @@ static int svc_rdma_has_wspace(struct svc_xprt *xprt)
 	return 1;
 }
 
+/*
+ * svcrdma does not count requests in xpt_recv_pending, so the
+ * position is published as the Send completion raised it.
+ */
+static u64 svc_rdma_acked_pos(struct svc_rqst *rqstp)
+{
+	return atomic64_read(&rqstp->rq_xprt->xpt_acked_pos);
+}
+
 static void svc_rdma_kill_temp_xprt(struct svc_xprt *xprt)
 {
 }
Signature · 1 line
2.56.0
author maintainer
#11 lore raw

PATCH v7 11/13 NFSD: Evict acknowledged DRC entries

In reply to Chuck Lever

A duplicate reply cache entry exists to answer a retransmit. Once the client has received the reply, no retransmit of that XID can follow, yet the entry stays in the cache until RC_EXPIRE elapses or memory pressure forces it out. On a busy server the cache is mostly replies the client already has.

Record each reply's transport position in its cache entry from the sv_reply_sent hook, and compare it with the transport's acknowledged position when a later request prunes the same bucket. An entry whose position the peer has acknowledged is evicted before the age and pressure checks run. Only entries that arrived on the current request's transport are compared: the prune holds a reference on that transport alone, so its acknowledged position is the only one safe to read. The shrinker passes no transport and never evicts on this basis. The walk still stops at the first entry that has neither expired nor been acknowledged, so an acknowledged entry behind one from another transport, or behind a UDP or TLS entry, stays until that entry expires. Only the entries the existing walk reaches are evicted here; a per-transport list of unacknowledged replies would reach the rest.

Read the acknowledged position through svc_xprt_acked_pos() before taking cache_lock, since the transport may sleep, and report the lookup with svc_xprt_recv_done() under the lock, after the insert. A prune by another thread on the same bucket runs under the same lock, so the prune either sees this request still counted or runs after the request has found its entry. A position the transport vouched for before the lock stays safe under the lock: a retransmit arriving later follows the acknowledgment.

A transport acknowledgment means the peer's transport holds the reply, not that the RPC layer above it has read the reply. Two retransmits can therefore find the entry already gone. A client that discards unread socket data when it resets a connection retransmits the call over the new connection. A client whose RPC timer fires after the reply was acknowledged but before its RPC layer read it retransmits the call on the same connection. In the second case the call is executed again, but TCP delivers the first reply ahead of the second, so the client completes the call with the first reply and discards the second as an unknown XID. The re-execution is visible only to a third party that reused the name in between. A call still in progress when its retransmit arrives has sent no reply, so its entry is not acknowledged and the retransmit is dropped as before. Retention therefore ends at delivery rather than at consumption; only NFSv4.1 sessions close that gap.

A call read before its connection closed can still be stalled ahead of its lookup while the retransmit on the new connection runs and the retransmit's entry is acknowledged and evicted. The original call then misses and runs the procedure again. No reply to the original can reach the client, so drop a cacheable call at lookup when its transport is dead. A call on a transport whose close NFSD has not yet seen still runs. The per-transport accounting does not cover that call.

Between nfsd_cache_update() and the hook the entry is pinned against eviction, and the nfsd thread keeps its pointer in thread-local state. Every nfsd_dispatch() path that leaves an entry in RC_DONE returns 1, so svc_process() runs the hook whether it sends the reply or drops it after svc_authorise() fails, and the pin is always cleared. The drop and encode-error paths update with RC_NOCACHE, which frees the entry without pinning it.

A reply that never reached the wire, or that went out over UDP, keeps position zero and is retained for the full RC_EXPIRE interval.

---
 fs/nfsd/cache.h    |  6 +++-
 fs/nfsd/nfscache.c | 96 +++++++++++++++++++++++++++++++++++++++++++++++++++---
 fs/nfsd/nfsd.h     |  3 ++
 fs/nfsd/nfssvc.c   |  1 +
 fs/nfsd/trace.h    |  1 +
 5 files changed, 102 insertions(+), 5 deletions(-)
fs/nfsd/cache.h +5 -1
index 589ffe4762d5..894e0b61cbfc 100644
@@ -36,8 +36,11 @@ struct nfsd_cacherep {
 	struct list_head	c_lru;
 	unsigned char		c_state,	/* unused, inprog, done */
 				c_type,		/* status, buffer */
-				c_secure : 1;	/* req came from port < 1024 */
+				c_secure : 1,	/* req came from port < 1024 */
+				c_inflight : 1;	/* reply cached but not yet sent */
 	u64			c_xprt;		/* svc_xprt that carried req */
+	u64			c_pos;		/* reply's position on c_xprt;
+						 * 0 = never acknowledged */
 	unsigned long		c_timestamp;
 	union {
 		struct kvec	u_vec;
@@ -88,6 +91,7 @@ int	nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start,
 			  unsigned int len, struct nfsd_cacherep **cacherep);
 void	nfsd_cache_update(struct svc_rqst *rqstp, struct nfsd_cacherep *rp,
 			  int cachetype, __be32 *statp);
+void	nfsd_cache_reply_sent(struct svc_rqst *rqstp);
 int	nfsd_reply_cache_stats_show(struct seq_file *m, void *v);
 
 #endif /* NFSCACHE_H */
fs/nfsd/nfscache.c +92 -4

165 lines not on this page · raw message

fs/nfsd/nfsd.h +3 -0
index b520b51692b0..0edbd4b9abae 100644
@@ -51,9 +51,12 @@ extern atomic_t			nfsd_th_cnt;		/* number of available threads */
 
 extern const struct seq_operations nfs_exports_op;
 
+struct nfsd_cacherep;
+
 struct nfsd_thread_local_info {
 	struct nfs4_client	**ntli_lease_breaker;
 	int			ntli_cachetype;
+	struct nfsd_cacherep	*ntli_cacherep;
 };
 
 /*
fs/nfsd/nfssvc.c +1 -0
index 9151dfc8ddd9..562bb06bb1ef 100644
@@ -681,6 +681,7 @@ int nfsd_create_serv(struct net *net, bool no_rpcbind)
 		return -ENOMEM;
 	}
 	serv->sv_tls = true;
+	serv->sv_reply_sent = nfsd_cache_reply_sent;
 
 	/* svc_bind() reads this, so set it first. */
 	serv->sv_no_rpcbind = no_rpcbind;
fs/nfsd/trace.h +1 -0
index b1ee0f468206..ad31d36c7c8d 100644
@@ -1747,6 +1747,7 @@ DEFINE_EVENT(nfsd_drc_entry_class, nfsd_drc_##name,		\
 
 DEFINE_NFSD_DRC_ENTRY_EVENT(evict_pressure);
 DEFINE_NFSD_DRC_ENTRY_EVENT(evict_expired);
+DEFINE_NFSD_DRC_ENTRY_EVENT(evict_acked);
 
 TRACE_EVENT(nfsd_cb_args,
 	TP_PROTO(
Signature · 1 line
2.56.0
author maintainer
#12 lore raw

PATCH v7 12/13 NFSD: Remove DRC checksum and payload_misses stat

In reply to Chuck Lever

nfsd_cache_csum() costs CPU on every non-idempotent request. It also prevents the use of zero-copy RDMA receives for WRITE and SYMLINK because the payload must be in the server's memory before the DRC lookup can run.

Commit 01a7decf7593 ("nfsd: keep a checksum of the first 256 bytes of request") added the checksum when a growing cache made XID collisions easier to hit. It is the only guard against a reused
XID: two calls with the same procedure and equal-length arguments match on every other key field.

Acknowledgment-based eviction retires a TCP or RDMA entry once the transport confirms delivery and a later request from the same connection visits its bucket. Three kinds of entry outlast that: the entries a connection has issued since it last visited each bucket, entries from a connection the client has since closed, which no later request can evict because their transport is gone, and UDP entries. Those wait out RC_EXPIRE, or longer in a bucket that no miss visits, since only a miss prunes expired entries. A collision needs the same XID from the same address and port while the entry lasts. The Linux NFS client's XIDs are sequential from a seed drawn with get_random_u32(), so an XID recurs only after 2^32 calls, and a rebooted client does not replay its previous sequence. A client that restarts its sequence from a fixed value and rebinds its previous source port while the entry lasts is unguarded, but a client that repeats a live XID cannot reliably match replies to its own calls anyway.

A call whose XID matches an entry sent on the same connection is therefore a retransmit. TCP and RDMA deliver the queued reply or drop the connection, so drop the call as though the entry were still in progress rather than replaying it. A match from another connection, or from UDP, is replayed as before.

Remove the checksum, the nfsd_drc_mismatch tracepoint, and the payload_misses stat that counted checksum-detected collisions. nfsd_dispatch() no longer snapshots the argument stream before decoding, and with k_csum gone, struct nfsd_cacherep shrinks to 136 bytes.

The "payload misses" line disappears from /proc/fs/nfsd/reply_cache_stats. No known userspace tool parses it.

---
 .../ABI/testing/procfs-nfsd-reply_cache_stats      | 11 +--
 fs/nfsd/cache.h                                    |  9 +-
 fs/nfsd/netns.h                                    |  2 -
 fs/nfsd/nfscache.c                                 | 96 ++++------------------
 fs/nfsd/nfssvc.c                                   | 10 +--
 fs/nfsd/stats.h                                    |  5 --
 fs/nfsd/trace.h                                    | 24 ------
 7 files changed, 23 insertions(+), 134 deletions(-)
Documentation/ABI/testing/procfs-nfsd-reply_cache_stats +4 -7
index 57ed5f8e6597..7a22ac7c1ccd 100644
@@ -19,19 +19,16 @@ Description:
 		cache misses             s64     Requests not found in cache
 		not cached               s64     Idempotent requests that
 		                                 bypass the cache
-		payload misses           s64     XID matched but request
-		                                 checksum did not
 		longest chain len        u32     Longest hash chain observed
 		cachesize at longest     u32     Cache size when longest
 		                                 chain was recorded
 		=======================  ======  ==========================
 
 		Counter fields (cache hits, cache misses, not cached,
-		payload misses, mem usage) are maintained with per-cpu
-		counters and may briefly show stale values under
-		concurrent load. There is no way to reset these
-		counters; consumers should compute rates by sampling
-		over time.
+		mem usage) are maintained with per-cpu counters and
+		may briefly show stale values under concurrent load.
+		There is no way to reset these counters; consumers
+		should compute rates by sampling over time.
 
 		New fields may be appended in future kernels. Parsers
 		should match on field name, not line position.
fs/nfsd/cache.h +2 -7
index 894e0b61cbfc..3491a10524d7 100644
@@ -22,9 +22,7 @@ struct nfsd_net;
  */
 struct nfsd_cacherep {
 	struct {
-		/* Keep often-read xid, csum in the same cache line: */
 		__be32			k_xid;
-		__wsum			k_csum;
 		u32			k_proc;
 		u32			k_prot;
 		u32			k_vers;
@@ -80,15 +78,12 @@ enum {
 /* Cache entries expire after this time period */
 #define RC_EXPIRE		(120 * HZ)
 
-/* Checksum this amount of the request */
-#define RC_CSUMLEN		(256U)
-
 int	nfsd_drc_slab_create(void);
 void	nfsd_drc_slab_free(void);
 int	nfsd_reply_cache_init(struct nfsd_net *);
 void	nfsd_reply_cache_shutdown(struct nfsd_net *);
-int	nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start,
-			  unsigned int len, struct nfsd_cacherep **cacherep);
+int	nfsd_cache_lookup(struct svc_rqst *rqstp,
+			  struct nfsd_cacherep **cacherep);
 void	nfsd_cache_update(struct svc_rqst *rqstp, struct nfsd_cacherep *rp,
 			  int cachetype, __be32 *statp);
 void	nfsd_cache_reply_sent(struct svc_rqst *rqstp);
fs/nfsd/netns.h +0 -2
index fa3545963902..1d32846b1836 100644
@@ -39,8 +39,6 @@ enum nfsd_net_flag {
 };
 
 enum {
-	/* cache misses due only to checksum comparison failures */
-	NFSD_STATS_PAYLOAD_MISSES,
 	/* amount of memory (in bytes) currently consumed by the DRC */
 	NFSD_STATS_DRC_MEM_USAGE,
 	NFSD_STATS_RC_HITS,		/* repcache hits */
fs/nfsd/nfscache.c +16 -80

171 lines not on this page · raw message

fs/nfsd/nfssvc.c +1 -9
index 562bb06bb1ef..f504a9386557 100644
@@ -1074,7 +1074,6 @@ int nfsd_dispatch(struct svc_rqst *rqstp)
 	const struct svc_procedure *proc = rqstp->rq_procinfo;
 	__be32 *statp = rqstp->rq_accept_statp;
 	struct nfsd_cacherep *rp;
-	unsigned int start, len;
 	__be32 *nfs_reply;
 
 	/*
@@ -1083,13 +1082,6 @@ int nfsd_dispatch(struct svc_rqst *rqstp)
 	 */
 	ntli->ntli_cachetype = proc->pc_cachetype;
 
-	/*
-	 * ->pc_decode advances the argument stream past the NFS
-	 * Call header, so grab the header's starting location and
-	 * size now for the call to nfsd_cache_lookup().
-	 */
-	start = xdr_stream_pos(&rqstp->rq_arg_stream);
-	len = xdr_stream_remaining(&rqstp->rq_arg_stream);
 	if (!proc->pc_decode(rqstp, &rqstp->rq_arg_stream))
 		goto out_decode_err;
 
@@ -1103,7 +1095,7 @@ int nfsd_dispatch(struct svc_rqst *rqstp)
 	smp_store_release(&rqstp->rq_status_counter, rqstp->rq_status_counter | 1);
 
 	rp = NULL;
-	switch (nfsd_cache_lookup(rqstp, start, len, &rp)) {
+	switch (nfsd_cache_lookup(rqstp, &rp)) {
 	case RC_DOIT:
 		break;
 	case RC_REPLY:
fs/nfsd/stats.h +0 -5
index aabfbb1a9c71..4a556dfbf64a 100644
@@ -97,11 +97,6 @@ static inline void nfsd_stats_io_write_add(struct nfsd_net *nn,
 					 amount);
 }
 
-static inline void nfsd_stats_payload_misses_inc(struct nfsd_net *nn)
-{
-	percpu_counter_inc(&nn->counter[NFSD_STATS_PAYLOAD_MISSES]);
-}
-
 /**
  * nfsd_stats_drc_mem_usage_add - Add memory used by a cache item
  * @nn: target network namespace
fs/nfsd/trace.h +0 -24
index ad31d36c7c8d..f37f537f12fe 100644
@@ -1687,30 +1687,6 @@ TRACE_EVENT(nfsd_drc_found,
 
 );
 
-TRACE_EVENT(nfsd_drc_mismatch,
-	TP_PROTO(
-		const struct nfsd_net *nn,
-		const struct nfsd_cacherep *key,
-		const struct nfsd_cacherep *rp
-	),
-	TP_ARGS(nn, key, rp),
-	TP_STRUCT__entry(
-		__field(unsigned long long, boot_time)
-		__field(u32, xid)
-		__field(u32, cached)
-		__field(u32, ingress)
-	),
-	TP_fast_assign(
-		__entry->boot_time = nn->boot_time;
-		__entry->xid = be32_to_cpu(key->c_key.k_xid);
-		__entry->cached = (__force u32)key->c_key.k_csum;
-		__entry->ingress = (__force u32)rp->c_key.k_csum;
-	),
-	TP_printk("boot_time=%16llx xid=0x%08x cached-csum=0x%08x ingress-csum=0x%08x",
-		__entry->boot_time, __entry->xid, __entry->cached,
-		__entry->ingress)
-);
-
 DECLARE_EVENT_CLASS(nfsd_drc_entry_class,
 	TP_PROTO(
 		const struct nfsd_net *nn,
Signature · 1 line
2.56.0
author maintainer
#13 lore raw

PATCH v7 13/13 SUNRPC: Advance the acked position on TLS connections

In reply to Chuck Lever

The DRC retires an entry once its reply lies at or below the transport's xpt_acked_pos, the position svc_tcp_update_acked() advances as TCP acknowledgments arrive. When XPT_TLS_SESSION is set, svc_tcp_update_acked() leaves xpt_acked_pos alone, because kTLS can return a full count from sendmsg() while a record is still waiting to reach TCP. DRC entries on TLS connections therefore never retire by acknowledgment.

Advance xpt_acked_pos on a TLS connection when no record is open, awaiting encryption, or parked on a short TCP send. Every field the check reads is declared in include/net/tls.h, so the predicate lives beside its one caller rather than in net/tls. After the change, a TLS connection takes the socket lock once per reply, as a plain TCP connection already does. Record in svcsock_tcp_acked why a reply published no position.

On the receive side, copied_seq does not track plaintext under kTLS. tls_strp_msg_consume() advances copied_seq by the whole ciphertext record once the record is decrypted, before svc_tcp_recv_actor() sees any of its plaintext, and in copy mode the strparser advances copied_seq while pulling the ciphertext in. A record that spans the end of one call and the start of a retransmit leaves the retransmit's plaintext on rx_list, uncounted in xpt_recv_pending, with copied_seq already at the mark. Withhold the position while sk_is_readable() reports plaintext on rx_list or a record ready in the strparser.

The close_notify alert and the FIN sent at detach go out without xpt_mutex. Both are non-blocking and never drop the socket lock mid-push, so their bytes either reach write_seq and lower the position, or park and hold it.

---
 include/linux/sunrpc/svcsock.h |  9 ++++
 include/trace/events/sunrpc.h  | 23 ++++++++--
 net/sunrpc/svcsock.c           | 96 +++++++++++++++++++++++++++++++++++-------
 3 files changed, 109 insertions(+), 19 deletions(-)
include/linux/sunrpc/svcsock.h +9 -0
index 0fa62f9c6465..bd5cd79c1870 100644
@@ -49,12 +49,21 @@ struct svc_sock {
 	u32			sk_acked_rcv;
 
 	struct completion	sk_handshake_done;
+	/* tls ULP installed, recorded once; the ULP has no removal path */
+	bool			sk_tls_ulp;
 
 	/* received data */
 	unsigned long		sk_maxpages;
 	struct page *		sk_pages[] __counted_by(sk_maxpages);
 };
 
+/* Why svc_tcp_update_acked() did or did not publish a position */
+enum svc_tcp_acked_reason {
+	SVC_TCP_ACKED_PUBLISHED,
+	SVC_TCP_ACKED_NO_ULP,
+	SVC_TCP_ACKED_TX_PENDING,
+};
+
 static inline u32 svc_sock_reclen(struct svc_sock *svsk)
 {
 	return be32_to_cpu(svsk->sk_marker) & RPC_FRAGMENT_SIZE_MASK;
include/trace/events/sunrpc.h +19 -4
index a5d914376627..a7e96e49e0d0 100644
@@ -10,6 +10,7 @@
 #include <linux/sunrpc/svc.h>
 #include <linux/sunrpc/xprtsock.h>
 #include <linux/sunrpc/svc_xprt.h>
+#include <linux/sunrpc/svcsock.h>
 #include <net/tcp_states.h>
 #include <linux/net.h>
 #include <linux/tracepoint.h>
@@ -2349,6 +2350,16 @@ TRACE_EVENT(svcsock_tcp_recv_short,
 	)
 );
 
+TRACE_DEFINE_ENUM(SVC_TCP_ACKED_PUBLISHED);
+TRACE_DEFINE_ENUM(SVC_TCP_ACKED_NO_ULP);
+TRACE_DEFINE_ENUM(SVC_TCP_ACKED_TX_PENDING);
+
+#define show_svc_tcp_acked_reason(reason)				\
+	__print_symbolic(reason,					\
+		{ SVC_TCP_ACKED_PUBLISHED,	"published" },		\
+		{ SVC_TCP_ACKED_NO_ULP,		"no_ulp" },		\
+		{ SVC_TCP_ACKED_TX_PENDING,	"tx_pending" })
+
 TRACE_EVENT(svcsock_tcp_acked,
 	TP_PROTO(
 		const struct svc_xprt *xprt,
@@ -2356,10 +2367,12 @@ TRACE_EVENT(svcsock_tcp_acked,
 		u32 write_seq,
 		u32 snd_una,
 		u64 candidate,
-		u64 acked_pos
+		u64 acked_pos,
+		enum svc_tcp_acked_reason reason
 	),
 
-	TP_ARGS(xprt, send_pos, write_seq, snd_una, candidate, acked_pos),
+	TP_ARGS(xprt, send_pos, write_seq, snd_una, candidate, acked_pos,
+		reason),
 
 	TP_STRUCT__entry(
 		__field(u64, xpt_id)
@@ -2368,6 +2381,7 @@ TRACE_EVENT(svcsock_tcp_acked,
 		__field(u64, acked_pos)
 		__field(u32, write_seq)
 		__field(u32, snd_una)
+		__field(unsigned int, reason)
 		__string(addr, xprt->xpt_remotebuf)
 	),
 
@@ -2378,13 +2392,14 @@ TRACE_EVENT(svcsock_tcp_acked,
 		__entry->acked_pos = acked_pos;
 		__entry->write_seq = write_seq;
 		__entry->snd_una = snd_una;
+		__entry->reason = reason;
 		__assign_str(addr);
 	),
 
-	TP_printk("addr=%s xpt_id=%llu send_pos=%llu write_seq=%u snd_una=%u candidate=%llu acked_pos=%llu",
+	TP_printk("addr=%s xpt_id=%llu send_pos=%llu write_seq=%u snd_una=%u candidate=%llu acked_pos=%llu reason=%s",
 		__get_str(addr), __entry->xpt_id, __entry->send_pos,
 		__entry->write_seq, __entry->snd_una, __entry->candidate,
-		__entry->acked_pos
+		__entry->acked_pos, show_svc_tcp_acked_reason(__entry->reason)
 	)
 );
 
net/sunrpc/svcsock.c +81 -15

159 lines not on this page · raw message

Signature · 1 line
2.56.0