Improve the scalability of NFSD's classic DRC
Subsystems (4)
Chuck Lever
maintainerv4 cover Improve the scalability of NFSD's classic DRC
A completed DRC entry currently stays in its bucket for 120 seconds whether or not the client already holds the reply. On a busy server those entries lengthen every bucket walk, and under pressure they crowd out entries a retransmit could still hit.
After this series, the DRC retires an entry once the server transport has seen the client peer acknowledge its reply. Each transport publishes two positions, the reply it has just sent and the point up to which the peer has acknowledged, and the DRC compares only the two (patch 6). Nothing else about transport state is exposed to fs/nfsd. UDP entries and replies sent over a TLS session keep the time-based eviction.
Retention now ends at delivery rather than at consumption. A client whose RPC timer fires after the reply was acknowledged but before its RPC layer read it retransmits, and NFSD executes the call again. The client completes with the first reply and discards the second. Patch 9 works through the cases.
The XID checksum and the 256k entry cap were both sized for a cache full of stale replies. With acknowledged entries retired, the checksum guards only entries that wait out RC_EXPIRE, and the cap starves the partition case it was never meant for (patches 10-11). The "payload misses" line disappears from reply_cache_stats.
---
Changes in v4:
- Track reply positions instead of ack cookies (per Neil's review).
- Make xpt_id a 64-bit counter and drop the xarray (per Neil's review).
- Drop implied ACK and the last-request timestamp (per Neil's review).
- Keep an unsent reply until RC_EXPIRE (per Neil's review).
- Pin a DRC entry against eviction until its reply has been sent.
- svcrdma: Serialize Send posting so positions match completion order.
- Drop the nfsd_drc_reply_acked tracepoint.
- Link to v3: https://patch.msgid.link/[email protected]
Changes in v3:
- Add the reply-acknowledged callback and its TCP and RDMA reporters
ahead of implied ACK, which now defers to a pending report.
- Split the prune-loop restructure into its own patch.
- Move the eviction tracepoints ahead of the ack patches; the
implied-ACK event now lands with implied ACK.
- Keep err local to the pmap_register block in svc_setup_socket()
(per Jeff's review).
- Never move xpt_last_recv backwards when nfsd threads race.
- Fold the XPT_ORDERED patch into its consumer (per Jeff's review).
- State the re-execution exposure of implied-ACK eviction instead of
calling the DRC advisory (per Jeff's review).
- New patch: remove the DRC request checksum and payload_misses stat.
- New patch: remove the 256k-entry cap on the DRC size.
- Link to v2: https://patch.msgid.link/[email protected]
Changes in v2:
- Print the DRC eviction tracepoints' age field as unsigned.
- Link to v1: https://patch.msgid.link/[email protected]
---
Chuck Lever (11):
SUNRPC: Assign a unique identifier to each svc_xprt
NFSD: Track transport in DRC entries
NFSD: Prepare bucket pruning for out-of-order eviction
NFSD: Add tracepoints for DRC entry eviction
NFSD: Record DRC population in lookup tracepoints
SUNRPC: Publish reply positions for upper-layer consumers
SUNRPC: Publish TCP reply positions
svcrdma: Publish RDMA reply positions
NFSD: Evict acknowledged DRC entries
NFSD: Remove DRC checksum and payload_misses stat
NFSD: Remove hard cap on duplicate reply cache size
.../ABI/testing/procfs-nfsd-reply_cache_stats | 11 +- fs/nfsd/cache.h | 16 +- fs/nfsd/netns.h | 2 - fs/nfsd/nfscache.c | 243 +++++++++++---------- fs/nfsd/nfsd.h | 3 + fs/nfsd/nfssvc.c | 11 +- fs/nfsd/stats.h | 5 - fs/nfsd/trace.h | 47 ++-- include/linux/sunrpc/svc.h | 16 ++ include/linux/sunrpc/svc_rdma.h | 5 +- include/linux/sunrpc/svc_xprt.h | 3 + include/linux/sunrpc/svcsock.h | 3 + include/trace/events/sunrpc.h | 8 +- net/sunrpc/svc.c | 3 + net/sunrpc/svc_xprt.c | 19 +- net/sunrpc/svcsock.c | 29 +++ net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 2 +- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 27 ++- net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + 19 files changed, 286 insertions(+), 168 deletions(-)
---
base-commit: abf2077ee32058e60d3f49ecab265d3c4bd953d9
change-id: 20260325-duplicate-reply-cache-f0fe7c7b740c
Best regards,
Signature · 1 line
Chuck Lever <[email protected]>
Chuck Lever
maintainerPATCH v4 1/11 SUNRPC: Assign a unique identifier to each svc_xprt
A consumer that associates state with a transport without holding a reference cannot detect ABA collisions: once a transport is freed, SLUB may hand out a new svc_xprt at the same address.
Assign each transport a 32-bit identifier from a global sequential counter in svc_xprt_init(). An identifier is reused only after 2^32 further transports have been created, so a consumer whose association is shorter-lived than that needs no release when the transport is freed. The NFSD duplicate reply cache is the first consumer. A 32-bit value fits the padding in its cache entry, and a cache entry is retained for two minutes, not for 2^32 connections.
--- include/linux/sunrpc/svc_xprt.h | 1 + include/trace/events/sunrpc.h | 8 ++++++-- net/sunrpc/svc_xprt.c | 17 ++++++++++++++--- 3 files changed, 21 insertions(+), 5 deletions(-)
include/linux/sunrpc/svc_xprt.h +1 -0
@@ -56,6 +56,7 @@ struct svc_xprt {struct svc_xprt_class *xpt_class;const struct svc_xprt_ops *xpt_ops;struct kref xpt_ref; +u32 xpt_id;ktime_t xpt_qtime;struct list_head xpt_list;struct lwq_node xpt_ready;
include/trace/events/sunrpc.h +6 -2
@@ -1986,7 +1986,8 @@ TRACE_EVENT(svc_xprt_create_err, __sockaddr(server, (x)->xpt_locallen) \ __sockaddr(client, (x)->xpt_remotelen) \ __field(unsigned long, flags) \ -__field (unsigned int ,netns_ino ) +__field (unsigned int ,netns_ino ) \ +__field (u32, xpt_id) #define SVC_XPRT_ENDPOINT_ASSIGNMENTS(x) \ do { \
@@ -1996,13 +1997,15 @@ TRACE_EVENT(svc_xprt_create_err, (x)->xpt_remotelen); \ __entry->flags = (x)->xpt_flags; \ __entry->netns_ino = (x)->xpt_net->ns.inum; \ + __entry->xpt_id = (x)->xpt_id; \ } while (0)#define SVC_XPRT_ENDPOINT_FORMAT \ - "server=%pISpc client=%pISpc flags=%s" + "server=%pISpc client=%pISpc xpt_id=%u flags=%s"#define SVC_XPRT_ENDPOINT_VARARGS \ __get_sockaddr(server), __get_sockaddr(client), \ + __entry->xpt_id, \ show_svc_xprt_flags(__entry->flags) TRACE_EVENT(svc_xprt_enqueue,
@@ -2024,6 +2027,7 @@ TRACE_EVENT(svc_xprt_enqueue, xprt->xpt_remotelen); __entry->flags = flags; __entry->netns_ino = xprt->xpt_net->ns.inum; + __entry->xpt_id = xprt->xpt_id; ),TP_printk (SVC_XPRT_ENDPOINT_FORMAT ,SVC_XPRT_ENDPOINT_VARARGS )
net/sunrpc/svc_xprt.c +14 -3
@@ -26,6 +26,8 @@static unsigned int svc_rpc_per_connection_limit__read_mostly ;module_param (svc_rpc_per_connection_limit, uint,0644 ); +static atomic_t svc_xprt_id_seq; +static struct svc_deferred_req *svc_deferred_dequeue (struct svc_xprt *xprt);static int svc_deferred_recv (struct svc_rqst *rqstp);
@@ -190,9 +192,17 @@ void svc_xprt_put(struct svc_xprt *xprt) }EXPORT_SYMBOL_GPL (svc_xprt_put); -/* -* Called by transport drivers to initialize the transport independent -* portion of the transport instance. +/** +* svc_xprt_init - initialize transport-independent portion of a transport +* @net: network namespace in which the transport operates +* @xcl: transport class providing operations and metadata +* @xprt: svc_xprt to initialize +* @serv: RPC service that owns this transport +* +* @xprt->xpt_id is reused only after 2^32 further transports have +* been initialized. +* +* Context: Process context. May sleep. */ void svc_xprt_init (struct net *net,struct svc_xprt_class *xcl,struct svc_xprt *xprt,struct svc_serv *serv)
@@ -210,6 +220,7 @@ void svc_xprt_init(struct net *net, struct svc_xprt_class *xcl,set_bit (XPT_BUSY , &xprt->xpt_flags); xprt->xpt_net =get_net_track (net, &xprt->ns_tracker,GFP_ATOMIC );strcpy (xprt->xpt_remotebuf,"uninitialized" ); + xprt->xpt_id =atomic_inc_return (&svc_xprt_id_seq); }EXPORT_SYMBOL_GPL (svc_xprt_init);
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v4 2/11 NFSD: Track transport in DRC entries
In reply to Chuck LeverA cached reply carries no record of the transport its request arrived on. Transport-aware eviction needs that association.
Record the xprt id in each entry. It fits the padding after the flag bytes, so struct nfsd_cacherep stays at 128 bytes. No behavior change.
--- fs/nfsd/cache.h | 1 + fs/nfsd/nfscache.c | 1 + 2 files changed, 2 insertions(+)
fs/nfsd/cache.h +1 -0
@@ -37,6 +37,7 @@ struct nfsd_cacherep {unsigned char c_state,/* unused, inprog, done */ c_type,/* status, buffer */ c_secure : 1; /* req came from port < 1024 */ +u32 c_xprt;/* svc_xprt that carried req */ unsigned long c_timestamp;union {struct kvec u_vec;
fs/nfsd/nfscache.c +1 -0
@@ -109,6 +109,7 @@ nfsd_cacherep_alloc(struct svc_rqst *rqstp, __wsum csum, rp->c_key.k_vers = rqstp->rq_vers; rp->c_key.k_len = rqstp->rq_arg.len; rp->c_key.k_csum = csum; + rp->c_xprt = rqstp->rq_xprt->xpt_id; }return rp; }
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v4 3/11 NFSD: Prepare bucket pruning for out-of-order eviction
In reply to Chuck Levernfsd_prune_bucket_locked() evicts for memory pressure and RC_EXPIRE. Each reason, once it holds for an entry, holds for every older one, so the walk stops at the first entry it cannot evict. A reason that applies to one entry but not to an older one, such as confirmation that a particular reply reached the client, cannot be added to that loop.
Continue the walk past an entry no reason applies to. The scan no longer stops at the first entry it cannot evict, so bound the work per call to four times the eviction limit, and pass the shrinker's remaining nr_to_scan budget as that limit.
--- fs/nfsd/nfscache.c | 35 +++++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 12 deletions(-)
fs/nfsd/nfscache.c +23 -12
@@ -257,8 +257,8 @@ nfsd_cache_bucket_find(__be32 xid, struct nfsd_net *nn) }/* -* Remove and return no more than @max expired entries in bucket @b. -* If @max is zero, do not limit the number of removed entries. +* Remove and return no more than @max evictable entries in bucket @b, +* visiting at most 4 * @max entries. @max must not be zero. */ static void nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
@@ -266,20 +266,30 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b, {unsigned long expiry = jiffies -RC_EXPIRE ;struct nfsd_cacherep *rp, *tmp; -unsigned int freed =0 ; +unsigned int freed =0 , visited =0 ;lockdep_assert_held (&b->cache_lock);/* The bucket LRU is ordered oldest-first. */ list_for_each_entry_safe (rp, tmp, &b->lru_head, c_lru) { -if (atomic_read (&nn->num_drc_entries) <= nn->max_drc_entries && -time_before (expiry, rp->c_timestamp)) -break ; +if (atomic_read (&nn->num_drc_entries) > nn->max_drc_entries) + goto evict; +if (time_before_eq (rp->c_timestamp, expiry)) + goto evict; + goto next; +evict:nfsd_cacherep_unlink_locked (nn, b, rp);list_add (&rp->c_lru, dispose); + freed++; -if (max && ++freed >= max) +next: +/* +* A client controls its XIDs, so it can pack one bucket with +* entries that are not yet evictable and turn each miss into +* a full-bucket walk under cache_lock. +*/ +if (freed >= max || ++visited >= max *4 )break ; } }
@@ -307,9 +317,9 @@ nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc)* @shrink: our registered shrinker context * @sc: garbage collection parameters * -* Free expired entries on each bucket's LRU list until we've released -* nr_to_scan freed objects. Nothing will be released if the cache -* has not exceeded it's max_drc_entries limit. +* Free entries on each bucket's LRU list until nr_to_scan objects have been +* released. Entries are evicted when they have expired or the cache exceeds +* its max_drc_entries limit. * * Returns the number of entries released by this call. */
@@ -328,11 +338,12 @@ nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc)continue ;spin_lock (&b->cache_lock); -nfsd_prune_bucket_locked (nn, b,0 , &dispose); +nfsd_prune_bucket_locked (nn, b, sc->nr_to_scan - freed, + &dispose);spin_unlock (&b->cache_lock); freed +=nfsd_cacherep_dispose (&dispose); -if (freed > sc->nr_to_scan) +if (freed >= sc->nr_to_scan)break ; }return freed;
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v4 4/11 NFSD: Add tracepoints for DRC entry eviction
In reply to Chuck LeverThe DRC pruning path evicts entries for memory pressure and expiry, and neither emits a trace event, so there is no way to see which reason is retiring entries or how old they are when it happens. A further eviction reason needs the same visibility.
Add an event class that records the cache population, the XID, the transport, and the entry's age, and define an event for each eviction reason.
--- fs/nfsd/nfscache.c | 8 ++++++-- fs/nfsd/trace.h | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 2 deletions(-)
fs/nfsd/nfscache.c +6 -2
@@ -272,10 +272,14 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,/* The bucket LRU is ordered oldest-first. */ list_for_each_entry_safe (rp, tmp, &b->lru_head, c_lru) { -if (atomic_read (&nn->num_drc_entries) > nn->max_drc_entries) +if (atomic_read (&nn->num_drc_entries) > nn->max_drc_entries) { +trace_nfsd_drc_evict_pressure (nn, rp); goto evict; -if (time_before_eq (rp->c_timestamp, expiry)) + } +if (time_before_eq (rp->c_timestamp, expiry)) { +trace_nfsd_drc_evict_expired (nn, rp); goto evict; + } goto next; evict:
fs/nfsd/trace.h +37 -0
@@ -1557,6 +1557,43 @@ TRACE_EVENT(nfsd_drc_mismatch, __entry->ingress) ); +DECLARE_EVENT_CLASS (nfsd_drc_entry_class, +TP_PROTO ( + const struct nfsd_net *nn, + const struct nfsd_cacherep *rp + ), +TP_ARGS (nn, rp), +TP_STRUCT__entry ( + __field(unsigned long long, boot_time) + __field(unsigned int, num_drc_entries) + __field(u32, xid) + __field(u32, xprt) + __field(unsigned long, age) + ), +TP_fast_assign ( + __entry->boot_time = nn->boot_time; + __entry->num_drc_entries = atomic_read(&nn->num_drc_entries); + __entry->xid = be32_to_cpu(rp->c_key.k_xid); + __entry->xprt = rp->c_xprt; + __entry->age =time_is_after_jiffies (rp->c_timestamp) ? + 1 : jiffies - rp->c_timestamp; + ), +TP_printk ("boot_time=%16llx entries=%u xid=0x%08x xprt=%u age=%lu" , + __entry->boot_time, __entry->num_drc_entries, + __entry->xid, __entry->xprt, __entry->age) +); + +#define DEFINE_NFSD_DRC_ENTRY_EVENT (name) \ +DEFINE_EVENT(nfsd_drc_entry_class, nfsd_drc_##name, \ + TP_PROTO( \ + const struct nfsd_net *nn, \ + const struct nfsd_cacherep *rp \ + ), \ + TP_ARGS(nn, rp)) + +DEFINE_NFSD_DRC_ENTRY_EVENT (evict_pressure); +DEFINE_NFSD_DRC_ENTRY_EVENT (evict_expired); + TRACE_EVENT(nfsd_cb_args, TP_PROTO( const struct nfs4_client *clp,
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v4 5/11 NFSD: Record DRC population in lookup tracepoints
In reply to Chuck Levernfsd_drc_found reports the outcome of a lookup but not how full the cache was at the time, so a trace cannot show whether retransmits are being caught while the cache runs near its cap.
--- fs/nfsd/nfscache.c | 4 +++- fs/nfsd/trace.h | 11 +++++++---- 2 files changed, 10 insertions(+), 5 deletions(-)
fs/nfsd/nfscache.c +3 -1
@@ -492,6 +492,7 @@ int nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start,struct nfsd_drc_bucket *b;int type = ntli->ntli_cachetype;LIST_HEAD (dispose); +unsigned int entries;int rtn =RC_DOIT ;if (type ==RC_NOCACHE ) {
@@ -561,7 +562,8 @@ int nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start, } out_trace: -trace_nfsd_drc_found (nn, rqstp, rtn); + entries =atomic_read (&nn->num_drc_entries); +trace_nfsd_drc_found (nn, entries, rqstp, rtn); out_unlock:spin_unlock (&b->cache_lock); out:
fs/nfsd/trace.h +7 -4
@@ -1513,23 +1513,26 @@ TRACE_DEFINE_ENUM(RC_DOIT);TRACE_EVENT (nfsd_drc_found,TP_PROTO ( const struct nfsd_net *nn, + unsigned int num_drc_entries, const struct svc_rqst *rqstp, int result ), -TP_ARGS (nn, rqstp, result), +TP_ARGS (nn, num_drc_entries, rqstp, result),TP_STRUCT__entry ( __field(unsigned long long, boot_time) + __field(unsigned int, num_drc_entries) __field(unsigned long, result)__field (u32, xid) ),TP_fast_assign ( __entry->boot_time = nn->boot_time; + __entry->num_drc_entries = num_drc_entries; __entry->result = result; __entry->xid = be32_to_cpu(rqstp->rq_xid); ), -TP_printk ("boot_time=%16llx xid=0x%08x result=%s" , - __entry->boot_time, __entry->xid, -show_drc_retval (__entry->result)) +TP_printk ("boot_time=%16llx entries=%u xid=0x%08x result=%s" , + __entry->boot_time, __entry->num_drc_entries, + __entry->xid,show_drc_retval (__entry->result)) );
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v4 6/11 SUNRPC: Publish reply positions for upper-layer consumers
In reply to Chuck LeverAn RPC server's duplicate reply cache has to hold a cached reply until the client has received it. SUNRPC gives an upper layer no way to learn that, so a cache entry lives for the full retention interval even when the reply was delivered at once.
Add two transport-neutral values that an upper layer can compare:
the position of each reply on its transport, recorded in the svc_rqst by xpo_sendto(), and the position up to which the peer has acknowledged, recorded on the svc_xprt. The units are private to the transport (bytes for a stream, Send sequence for RDMA); a consumer learns only whether one is less than or equal to the other. A reply that xpo_sendto() does not hand to the wire keeps position zero and is never reported as acknowledged.
The acknowledged position is an atomic64_t. A transport publishes it outside any lock the consumer holds, and a torn 64-bit read on a 32-bit host could run ahead of the peer.
Add a per-service hook that runs once per request when its reply phase ends, whether svc_send() sent the reply or svc_process() dropped it, so an upper layer can record the reply's position against its own state either way. On the drop path the position is zero.
--- include/linux/sunrpc/svc.h | 16 ++++++++++++++++ include/linux/sunrpc/svc_xprt.h | 2 ++ net/sunrpc/svc.c | 3 +++ net/sunrpc/svc_xprt.c | 2 ++ 4 files changed, 23 insertions(+)
include/linux/sunrpc/svc.h +16 -0
@@ -59,6 +59,8 @@ enum { }; +struct svc_rqst ; +/* * RPC service. *
@@ -96,6 +98,13 @@ struct svc_serv {* connection */ bool sv_bc_enabled;/* service uses backchannel */ #endif /* CONFIG_SUNRPC_BACKCHANNEL */ + +/* +* Called once per request after its reply phase, whether +* xpo_sendto() succeeded, failed, or was never reached because +* svc_process() dropped the reply. See rq_reply_pos. +*/ +void (*sv_reply_sent)(struct svc_rqst *rqstp); };/* This is used by pool_stats to find and lock an svc */
@@ -267,6 +276,13 @@ struct svc_rqst {unsigned int bc_to_retries;unsigned int rq_status_counter;/* RPC processing counter */ void *rq_private;/* For use by the service thread */ + +/* +* Transport position of this request's reply, set by +* xpo_sendto(); zero when the reply did not reach the wire. +* Comparable only with the xpt_acked_pos of the same svc_xprt. +*/ +u64 rq_reply_pos; };/* bits for rq_flags */
include/linux/sunrpc/svc_xprt.h +2 -0
@@ -66,6 +66,8 @@ struct svc_xprt {atomic_t xpt_reserved;/* outq space rsvd, UDP only */ atomic_t xpt_nr_rqsts;/* Number of requests */ struct mutex xpt_mutex;/* to serialize sending data */ +atomic64_t xpt_acked_pos;/* replies up to this position +* are acknowledged; 0 = none */ spinlock_t xpt_lock;/* protects sk_deferred * and xpt_auth_cache */ void *xpt_auth_cache;/* auth cache */
net/sunrpc/svc.c +3 -0
@@ -1488,6 +1488,7 @@ svc_process_common(struct svc_rqst *rqstp)/* Reset the accept_stat for the RPC */ rqstp->rq_accept_statp =NULL ; + rqstp->rq_reply_pos =0 ;/* Will be turned off only when NFSv4 Sessions are used */ set_bit (RQ_USEDEFERRAL , &rqstp->rq_flags);
@@ -1733,6 +1734,8 @@ void svc_process(struct svc_rqst *rqstp) goto out_baddir;if (!svc_process_common (rqstp)) { +if (rqstp->rq_server->sv_reply_sent) + rqstp->rq_server->sv_reply_sent (rqstp);svc_release_rqst (rqstp); goto out_drop; }
net/sunrpc/svc_xprt.c +2 -0
@@ -1028,6 +1028,8 @@ void svc_send(struct svc_rqst *rqstp)trace_svc_stats_latency (rqstp); status = xprt->xpt_ops->xpo_sendto (rqstp); +if (xprt->xpt_server->sv_reply_sent) + xprt->xpt_server->sv_reply_sent (rqstp);trace_svc_send (rqstp, status); }
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v4 7/11 SUNRPC: Publish TCP reply positions
In reply to Chuck LeverAdd the svcsock side of reply-position reporting. A reply's position is the running count of bytes handed to the socket, kept per socket under xpt_mutex. A TCP sequence number cannot serve as the position:
the 32-bit space wraps in a few hundred milliseconds at 40 GbE, and a duplicate reply cache entry lives for two minutes.
The acknowledged position is derived from the socket's unacknowledged byte count, write_seq minus snd_una, after each successful send. The two are read without the socket lock, as tcp_ioctl() reads them for
SIOCOUTQ: write_seq advances only under xpt_mutex, and a stale snd_una only lowers the result. Refreshing the position only in the send path means a consumer sees the acknowledgment state as of the previous reply on that connection, stale by one round trip but never ahead of the peer. A failed or refused send leaves rq_reply_pos at zero, so the reply is never reported as acknowledged.
Under kTLS the socket counts ciphertext while svcsock counts plaintext, and tls_sw_sendmsg() can return the full plaintext count with part of a record still waiting for socket write space. The unacknowledged count then understates what the peer has yet to receive. Whether a record is pending is state private to net/tls, so a TLS session publishes no acknowledged position and its replies are retained for the full interval.
--- include/linux/sunrpc/svcsock.h | 3 +++ net/sunrpc/svcsock.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)
include/linux/sunrpc/svcsock.h +3 -0
@@ -41,6 +41,9 @@ struct svc_sock {struct page_frag_cache sk_frag_cache; +/* reply bytes handed to the socket; protected by xpt_mutex */ +u64 sk_send_pos; +struct completion sk_handshake_done;/* received data */
net/sunrpc/svcsock.c +29 -0
@@ -1390,6 +1390,32 @@ static int svc_tcp_sendmsg(struct svc_sock *svsk, struct svc_rqst *rqstp,return ret; } +/* +* Bytes the socket has not seen acknowledged all belong to the most +* recent replies, so sk_send_pos less that count is the acknowledged +* position. write_seq advances only under xpt_mutex, which the caller +* holds, and a stale snd_una only lowers the result. +* +* Under kTLS, tls_sw_sendmsg() can return the full plaintext count +* with part of a record still waiting for socket write space, leaving +* write_seq short of the reply. Whether a record is pending is private +* to net/tls, so a TLS session publishes no acknowledged position. +*/ +static void svc_tcp_update_acked (struct svc_sock *svsk) +{ +struct tcp_sock *tp =tcp_sk (svsk->sk_sk); +u64 unacked, acked; + +if (test_bit (XPT_TLS_SESSION , &svsk->sk_xprt.xpt_flags)) +return ; + unacked =READ_ONCE (tp->write_seq) -READ_ONCE (tp->snd_una); +if (unacked >= svsk->sk_send_pos) +return ; + acked = svsk->sk_send_pos - unacked; +if (acked >atomic64_read (&svsk->sk_xprt.xpt_acked_pos)) +atomic64_set (&svsk->sk_xprt.xpt_acked_pos, acked); +} +/** * svc_tcp_sendto - Send out a reply on a TCP socket * @rqstp: completed svc_rqst
@@ -1418,6 +1444,9 @@ static int svc_tcp_sendto(struct svc_rqst *rqstp)trace_svcsock_tcp_send (xprt, sent);if (sent <0 || sent != (xdr->len +sizeof (marker))) goto out_close; + svsk->sk_send_pos += sent; + rqstp->rq_reply_pos = svsk->sk_send_pos; +svc_tcp_update_acked (svsk);mutex_unlock (&xprt->xpt_mutex);return sent;
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v4 8/11 svcrdma: Publish RDMA reply positions
In reply to Chuck LeverAdd the svcrdma side of reply-position reporting. A reply's position is the sequence number of its Send on the transport, and the Send completion for that reply publishes the same number as the transport's acknowledged position. A successful Send completion means the peer's HCA has acknowledged the message, so a reply whose Send has completed is in the client's receive buffer.
Completions on one queue pair arrive in posting order. Several nfsd threads post on one queue pair with no serialization above the provider, so the order in which Sends are queued is decided inside the provider. Assign the sequence number and post the Send under one lock so positions match queuing order. ib_post_send() does not sleep; the Send Queue wait stays outside the lock.
Only a posted RPC Reply reports a position. The RDMA_ERROR message and backchannel calls post with no position pointer, and a Send that fails to post leaves rq_reply_pos at zero.
--- include/linux/sunrpc/svc_rdma.h | 5 ++++- net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 2 +- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 27 ++++++++++++++++++++++++--- net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + 4 files changed, 30 insertions(+), 5 deletions(-)
include/linux/sunrpc/svc_rdma.h +4 -1
@@ -96,6 +96,8 @@ struct svcxprt_rdma {spinlock_t sc_send_lock;struct llist_head sc_send_ctxts; +spinlock_t sc_post_lock;/* orders Send posting */ +u64 sc_post_seq;/* Sends posted so far */ spinlock_t sc_rw_ctxt_lock;struct llist_head sc_rw_ctxts;
@@ -242,6 +244,7 @@ struct svc_rdma_send_ctxt {struct ib_send_wr *sc_wr_chain;int sc_sqecount;struct ib_cqe sc_cqe; +u64 sc_pos;struct xdr_buf sc_hdrbuf;struct xdr_stream sc_stream;
@@ -305,7 +308,7 @@ extern struct svc_rdma_send_ctxt *extern void svc_rdma_send_ctxt_put (struct svcxprt_rdma *rdma,struct svc_rdma_send_ctxt *ctxt);extern int svc_rdma_post_send (struct svcxprt_rdma *rdma, -struct svc_rdma_send_ctxt *ctxt); +struct svc_rdma_send_ctxt *ctxt,u64 *pos); extern int svc_rdma_map_reply_msg(struct svcxprt_rdma *rdma, struct svc_rdma_send_ctxt *sctxt, const struct svc_rdma_pcl *write_pcl,
net/sunrpc/xprtrdma/svc_rdma_backchannel.c +1 -1
@@ -90,7 +90,7 @@ static int svc_rdma_bc_sendto(struct svcxprt_rdma *rdma,*/ get_page (virt_to_page (rqst->rq_buffer)); sctxt->sc_send_wr.opcode =IB_WR_SEND ; -return svc_rdma_post_send (rdma, sctxt); +return svc_rdma_post_send (rdma, sctxt,NULL ); }/* Server-side transport endpoint wants a whole page for its send
net/sunrpc/xprtrdma/svc_rdma_sendto.c +24 -3
@@ -222,6 +222,7 @@ struct svc_rdma_send_ctxt *svc_rdma_send_ctxt_get(struct svcxprt_rdma *rdma) ctxt->sc_page_count =0 ; ctxt->sc_wr_chain = &ctxt->sc_send_wr; ctxt->sc_sqecount =1 ; + ctxt->sc_pos =0 ;return ctxt;
@@ -471,6 +472,8 @@ static void svc_rdma_wc_send(struct ib_cq *cq, struct ib_wc *wc) goto flushed;trace_svcrdma_wc_send (&ctxt->sc_cid); +if (ctxt->sc_pos >atomic64_read (&rdma->sc_xprt.xpt_acked_pos)) +atomic64_set (&rdma->sc_xprt.xpt_acked_pos, ctxt->sc_pos);svc_rdma_send_ctxt_put (rdma, ctxt);return ;
@@ -487,23 +490,29 @@ static void svc_rdma_wc_send(struct ib_cq *cq, struct ib_wc *wc)* svc_rdma_post_send - Post a WR chain to the Send Queue * @rdma: transport context * @ctxt: WR chain to post +* @pos: OUT: position of this Send on @rdma, or NULL * * Copy fields in @ctxt to stack variables in order to guarantee * that these values remain available after the ib_post_send() call. * In some error flow cases, svc_rdma_wc_send() releases @ctxt. * +* @pos is written only when the chain was posted. Positions on one +* transport increase in posting order, and the Send completion of +* @ctxt publishes its position in xpt_acked_pos. +* * Return values: * %0: @ctxt's WR chain was posted successfully * %-ENOTCONN: The connection was lost */ int svc_rdma_post_send (struct svcxprt_rdma *rdma, -struct svc_rdma_send_ctxt *ctxt) +struct svc_rdma_send_ctxt *ctxt,u64 *pos) {struct ib_send_wr *first_wr = ctxt->sc_wr_chain;struct ib_send_wr *send_wr = &ctxt->sc_send_wr;const struct ib_send_wr *bad_wr = first_wr;struct rpc_rdma_cid cid = ctxt->sc_cid;int ret, sqecount = ctxt->sc_sqecount; +u64 seq;might_sleep ();
@@ -518,10 +527,22 @@ int svc_rdma_post_send(struct svcxprt_rdma *rdma,return ret;trace_svcrdma_post_send (ctxt); + +/* +* Assign the position and post under one lock so positions +* match the order the provider queues the Sends, and thus +* completion order. +*/ +spin_lock (&rdma->sc_post_lock); + seq = ++rdma->sc_post_seq; + ctxt->sc_pos = seq; ret =ib_post_send (rdma->sc_qp, first_wr, &bad_wr); +spin_unlock (&rdma->sc_post_lock);if (ret)return svc_rdma_post_send_err (rdma, &cid, bad_wr, first_wr, sqecount, ret); +if (pos) + *pos = seq;return 0 ; }
@@ -1052,7 +1073,7 @@ static int svc_rdma_send_reply_msg(struct svcxprt_rdma *rdma, send_wr->opcode =IB_WR_SEND ; } -return svc_rdma_post_send (rdma, sctxt); +return svc_rdma_post_send (rdma, sctxt, &rqstp->rq_reply_pos); }/**
@@ -1122,7 +1143,7 @@ void svc_rdma_send_error_msg(struct svcxprt_rdma *rdma,*/ sctxt->sc_wr_chain = &sctxt->sc_send_wr; sctxt->sc_sqecount =1 ; -if (svc_rdma_post_send (rdma, sctxt)) +if (svc_rdma_post_send (rdma, sctxt,NULL )) goto put_ctxt;return ;
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -0
@@ -207,6 +207,7 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,lockdep_set_class (&cma_xprt->sc_send_lock, &svcrdma_sctx_lock);spin_lock_init (&cma_xprt->sc_rw_ctxt_lock);lockdep_set_class (&cma_xprt->sc_rw_ctxt_lock, &svcrdma_rwctx_lock); +spin_lock_init (&cma_xprt->sc_post_lock);/* * Note that this implies that the underlying transport support
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v4 9/11 NFSD: Evict acknowledged DRC entries
In reply to Chuck LeverA duplicate reply cache entry exists to answer a retransmit. Once the client has received the reply, no retransmit of that XID can follow, yet the entry stays in the cache until RC_EXPIRE elapses or memory pressure forces it out. On a busy server the cache is mostly replies the client already has.
Record each reply's transport position in its cache entry from the sv_reply_sent hook, and compare it with the transport's acknowledged position when a later request prunes the same bucket. An entry whose position the peer has acknowledged is evicted before the age and pressure checks run. Only entries that arrived on the current request's transport are compared: the prune holds a reference on that transport alone, so its acknowledged position is the only one safe to read. The shrinker passes no transport and never evicts on this basis.
A transport acknowledgment means the peer's transport holds the reply, not that the RPC layer above it has read the reply. Two retransmits can therefore find the entry already gone. A client that discards unread socket data when it resets a connection retransmits the call over the new connection. A client whose RPC timer fires after the reply was acknowledged but before its RPC layer read it retransmits the call on the same connection. In the second case the call is executed again, but TCP delivers the first reply ahead of the second, so the client completes the call with the first reply and discards the second as an unknown XID. The re-execution is visible only to a third party that reused the name in between. A call still in progress when its retransmit arrives has sent no reply, so its entry is not acknowledged and the retransmit is dropped as before. Retention therefore ends at delivery rather than at consumption; only NFSv4.1 sessions close that gap.
Between nfsd_cache_update() and the hook the entry is pinned against eviction, and the nfsd thread keeps its pointer in thread-local state. Every nfsd_dispatch() path that leaves an entry in RC_DONE returns 1, so svc_process() runs the hook whether it sends the reply or drops it after svc_authorise() fails, and the pin is always cleared. The drop and encode-error paths update with RC_NOCACHE, which frees the entry without pinning it.
A reply that never reached the wire, or that went out over UDP, keeps position zero and is retained for the full RC_EXPIRE interval.
--- fs/nfsd/cache.h | 6 ++++- fs/nfsd/nfscache.c | 65 +++++++++++++++++++++++++++++++++++++++++++++++++++--- fs/nfsd/nfsd.h | 3 +++ fs/nfsd/nfssvc.c | 1 + fs/nfsd/trace.h | 1 + 5 files changed, 72 insertions(+), 4 deletions(-)
fs/nfsd/cache.h +5 -1
@@ -36,8 +36,11 @@ struct nfsd_cacherep {struct list_head c_lru;unsigned char c_state,/* unused, inprog, done */ c_type,/* status, buffer */ - c_secure : 1; /* req came from port < 1024 */ + c_secure : 1, /* req came from port < 1024 */ + c_inflight : 1; /* reply cached but not yet sent */u32 c_xprt;/* svc_xprt that carried req */ +u64 c_pos;/* reply's position on c_xprt; +* 0 = never acknowledged */ unsigned long c_timestamp;union {struct kvec u_vec;
@@ -88,6 +91,7 @@ int nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start, unsigned int len, struct nfsd_cacherep **cacherep);void nfsd_cache_update (struct svc_rqst *rqstp,struct nfsd_cacherep *rp,int cachetype,__be32 *statp); +void nfsd_cache_reply_sent (struct svc_rqst *rqstp);int nfsd_reply_cache_stats_show (struct seq_file *m,void *v);#endif /* NFSCACHE_H */
fs/nfsd/nfscache.c +62 -3
117 lines not on this page · raw message
fs/nfsd/nfsd.h +3 -0
@@ -53,9 +53,12 @@ extern atomic_t nfsd_th_cnt; /* number of available threads */extern const struct seq_operations nfs_exports_op; +struct nfsd_cacherep ; +struct nfsd_thread_local_info {struct nfs4_client **ntli_lease_breaker;int ntli_cachetype; +struct nfsd_cacherep *ntli_cacherep; };/*
fs/nfsd/nfssvc.c +1 -0
@@ -634,6 +634,7 @@ int nfsd_create_serv(struct net *net)percpu_ref_exit (&nn->nfsd_net_ref);return -ENOMEM ; } + serv->sv_reply_sent = nfsd_cache_reply_sent; error =svc_bind (serv, net);if (error <0 ) {
fs/nfsd/trace.h +1 -0
@@ -1596,6 +1596,7 @@ DEFINE_EVENT(nfsd_drc_entry_class, nfsd_drc_##name, \DEFINE_NFSD_DRC_ENTRY_EVENT (evict_pressure);DEFINE_NFSD_DRC_ENTRY_EVENT (evict_expired); +DEFINE_NFSD_DRC_ENTRY_EVENT (evict_acked); TRACE_EVENT(nfsd_cb_args, TP_PROTO(
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v4 10/11 NFSD: Remove DRC checksum and payload_misses stat
In reply to Chuck Levernfsd_cache_csum() costs CPU on every non-idempotent request. It also prevents the use of zero-copy RDMA receives for WRITE and SYMLINK because the payload must be in the server's memory before the DRC lookup can run.
Commit 01a7decf7593 ("nfsd: keep a checksum of the first 256 bytes of request") added the checksum when a growing cache made XID collisions easier to hit. It is the only guard against a reused
XID: two calls with the same procedure and equal-length arguments match on every other key field.
Acknowledgment-based eviction retires a TCP or RDMA entry once the transport confirms delivery and a later request from the same connection visits its bucket. Three kinds of entry outlast that: the entries a connection has issued since it last visited each bucket, entries from a connection the client has since closed, which no later request can evict because their transport is gone, and UDP entries. Those wait out RC_EXPIRE. A collision needs the same XID from the same address and port within that window. The Linux NFS client's XIDs are sequential from a seed drawn with get_random_u32(), so an XID recurs only after 2^32 calls, and a rebooted client does not replay its previous sequence. A client that restarts its sequence from a fixed value and rebinds its previous source port within RC_EXPIRE is unguarded, but a client that repeats a live XID cannot reliably match replies to its own calls anyway.
An acknowledged entry stays in its bucket until the next prune visits it, and a lookup matches it in the meantime. The client already holds that reply, so treat a call carrying its XID as a
miss: evict the entry and insert the new one in its place.
Remove the checksum, the nfsd_drc_mismatch tracepoint, and the payload_misses stat that counted checksum-detected collisions. nfsd_dispatch() no longer snapshots the argument stream before decoding, and with k_csum gone, struct nfsd_cacherep is 128 bytes again.
The "payload misses" line disappears from /proc/fs/nfsd/reply_cache_stats. No known userspace tool parses it.
--- .../ABI/testing/procfs-nfsd-reply_cache_stats | 11 +-- fs/nfsd/cache.h | 9 +- fs/nfsd/netns.h | 2 - fs/nfsd/nfscache.c | 103 +++++---------------- fs/nfsd/nfssvc.c | 10 +- fs/nfsd/stats.h | 5 - fs/nfsd/trace.h | 24 ----- 7 files changed, 28 insertions(+), 136 deletions(-)
Documentation/ABI/testing/procfs-nfsd-reply_cache_stats +4 -7
@@ -19,19 +19,16 @@ Description: cache misses s64 Requests not found in cache not cached s64 Idempotent requests that bypass the cache - payload misses s64 XID matched but request - checksum did not longest chain len u32 Longest hash chain observed cachesize at longest u32 Cache size when longest chain was recorded ======================= ====== ========================== Counter fields (cache hits, cache misses, not cached, - payload misses, mem usage) are maintained with per-cpu - counters and may briefly show stale values under - concurrent load. There is no way to reset these - counters; consumers should compute rates by sampling - over time. + mem usage) are maintained with per-cpu counters and + may briefly show stale values under concurrent load. + There is no way to reset these counters; consumers + should compute rates by sampling over time. New fields may be appended in future kernels. Parsers should match on field name, not line position.
fs/nfsd/cache.h +2 -7
@@ -22,9 +22,7 @@ struct nfsd_net;*/ struct nfsd_cacherep {struct { -/* Keep often-read xid, csum in the same cache line: */ __be32 k_xid; -__wsum k_csum;u32 k_proc;u32 k_prot;u32 k_vers;
@@ -80,15 +78,12 @@ enum {/* Cache entries expire after this time period */ #define RC_EXPIRE (120 * HZ) -/* Checksum this amount of the request */ -#define RC_CSUMLEN (256U) -int nfsd_drc_slab_create (void );void nfsd_drc_slab_free (void );int nfsd_reply_cache_init (struct nfsd_net *);void nfsd_reply_cache_shutdown (struct nfsd_net *); -int nfsd_cache_lookup (struct svc_rqst *rqstp,unsigned int start, -unsigned int len,struct nfsd_cacherep **cacherep); +int nfsd_cache_lookup (struct svc_rqst *rqstp, +struct nfsd_cacherep **cacherep);void nfsd_cache_update (struct svc_rqst *rqstp,struct nfsd_cacherep *rp,int cachetype,__be32 *statp);void nfsd_cache_reply_sent (struct svc_rqst *rqstp);
fs/nfsd/netns.h +0 -2
@@ -39,8 +39,6 @@ enum nfsd_net_flag { };enum { -/* cache misses due only to checksum comparison failures */ -NFSD_STATS_PAYLOAD_MISSES ,/* amount of memory (in bytes) currently consumed by the DRC */ NFSD_STATS_DRC_MEM_USAGE ,NFSD_STATS_RC_HITS ,/* repcache hits */
fs/nfsd/nfscache.c +21 -82
178 lines not on this page · raw message
fs/nfsd/nfssvc.c +1 -9
@@ -1005,7 +1005,6 @@ int nfsd_dispatch(struct svc_rqst *rqstp)const struct svc_procedure *proc = rqstp->rq_procinfo;__be32 *statp = rqstp->rq_accept_statp;struct nfsd_cacherep *rp; -unsigned int start, len;__be32 *nfs_reply;/*
@@ -1014,13 +1013,6 @@ int nfsd_dispatch(struct svc_rqst *rqstp)*/ ntli->ntli_cachetype = proc->pc_cachetype; -/* -* ->pc_decode advances the argument stream past the NFS -* Call header, so grab the header's starting location and -* size now for the call to nfsd_cache_lookup(). -*/ - start =xdr_stream_pos (&rqstp->rq_arg_stream); - len =xdr_stream_remaining (&rqstp->rq_arg_stream);if (!proc->pc_decode (rqstp, &rqstp->rq_arg_stream)) goto out_decode_err;
@@ -1034,7 +1026,7 @@ int nfsd_dispatch(struct svc_rqst *rqstp)smp_store_release (&rqstp->rq_status_counter, rqstp->rq_status_counter |1 ); rp =NULL ; -switch (nfsd_cache_lookup (rqstp, start, len, &rp)) { +switch (nfsd_cache_lookup (rqstp, &rp)) {case RC_DOIT :break ;case RC_REPLY :
fs/nfsd/stats.h +0 -5
@@ -97,11 +97,6 @@ static inline void nfsd_stats_io_write_add(struct nfsd_net *nn, amount); } -static inline void nfsd_stats_payload_misses_inc (struct nfsd_net *nn) -{ -percpu_counter_inc (&nn->counter[NFSD_STATS_PAYLOAD_MISSES ]); -} -/** * nfsd_stats_drc_mem_usage_add - Add memory used by a cache item * @nn: target network namespace
fs/nfsd/trace.h +0 -24
@@ -1536,30 +1536,6 @@ TRACE_EVENT(nfsd_drc_found, ); -TRACE_EVENT (nfsd_drc_mismatch, -TP_PROTO ( - const struct nfsd_net *nn, - const struct nfsd_cacherep *key, - const struct nfsd_cacherep *rp - ), -TP_ARGS (nn, key, rp), -TP_STRUCT__entry ( - __field(unsigned long long, boot_time) - __field(u32, xid) - __field(u32, cached) -__field (u32, ingress) - ), -TP_fast_assign ( - __entry->boot_time = nn->boot_time; - __entry->xid = be32_to_cpu(key->c_key.k_xid); - __entry->cached = (__force u32)key->c_key.k_csum; - __entry->ingress = (__force u32)rp->c_key.k_csum; - ), -TP_printk ("boot_time=%16llx xid=0x%08x cached-csum=0x%08x ingress-csum=0x%08x" , - __entry->boot_time, __entry->xid, __entry->cached, - __entry->ingress) -); - DECLARE_EVENT_CLASS(nfsd_drc_entry_class, TP_PROTO( const struct nfsd_net *nn,
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v4 11/11 NFSD: Remove hard cap on duplicate reply cache size
In reply to Chuck LeverThe DRC matters most during a network partition, when clients cannot receive replies and NFSD must hold them for retransmission after reconnect. Acknowledgment-based eviction is inactive then, leaving only RC_EXPIRE, and a 256k entry ceiling can be too small for a large client cohort through a long partition.
During normal operation over TCP and RDMA, acknowledgment-based eviction is bucket-local. A lookup evicts acknowledged entries only from the bucket it lands in, and only those that arrived on its own connection. Each active connection keeps about one entry per bucket resident, the calls it issued since it last visited that bucket, which is an eighth of the maximum. With more than a handful of busy connections the maximum governs in normal operation as well, and pressure eviction then removes oldest-first entries that are already acknowledged. The cap bounds memory in normal operation and starves the partition case. The formula's maximum serves both. UDP entries wait out RC_EXPIRE as before, and the maximum bounds a UDP workload as it always has.
Remove the cap and let the square-root formula govern sizing. It scales sub-linearly with memory, and the hash table already uses kvzalloc, so larger sizes need no physical contiguity. The limit is unchanged at 16 GB and below. At 1 TB the formula yields 1048576 entries, four times the old cap. Even at 1 KB per entry, the worst case the old comment assumed, a full cache is 1 GB, 0.1% of that host's memory. That memory is not reclaimable: the shrinker frees only entries older than RC_EXPIRE, and each net namespace sizes its own cache.
--- fs/nfsd/nfscache.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-)
fs/nfsd/nfscache.c +18 -15
@@ -45,34 +45,37 @@ static unsigned long nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc);/* -* Put a cap on the size of the DRC based on the amount of available -* low memory in the machine. +* Size the DRC by the amount of low memory in the machine. The +* limit scales with the square root of available pages; with 4 KB +* pages: * * 64MB: 8192 -* 128MB: 11585 +* 128MB: 11584 * 256MB: 16384 -* 512MB: 23170 +* 512MB: 23168 * 1GB: 32768 -* 2GB: 46340 +* 2GB: 46336 * 4GB: 65536 -* 8GB: 92681 +* 8GB: 92672 * 16GB: 131072 +* 32GB: 185344 +* 64GB: 262144 +* 128GB: 370688 +* 256GB: 524288 +* 512GB: 741440 +* 1TB: 1048576 * -* ...with a hard cap of 256k entries. In the worst case, each entry will be -* ~1k, so the above numbers should give a rough max of the amount of memory -* used in k. -* -* XXX: these limits are per-container, so memory used will increase -* linearly with number of containers. Maybe that's OK. +* These limits are per-net-namespace, so memory used increases +* linearly with the number of namespaces. The shrinker frees only +* entries older than RC_EXPIRE, so a cache below its limit is not +* reclaimable under memory pressure. */ static unsigned int nfsd_cache_size_limit (void ) { -unsigned int limit;unsigned long low_pages =totalram_pages () -totalhigh_pages (); - limit = (16 *int_sqrt (low_pages)) << (PAGE_SHIFT -10 ); - return min_t(unsigned int, limit, 256*1024); +return (16 *int_sqrt (low_pages)) << (PAGE_SHIFT -10 ); }/*
Signature · 1 line
2.55.0
Quoted 25 lines · Chuck Lever
A completed DRC entry currently stays in its bucket for 120 seconds whether or not the client already holds the reply. On a busy server those entries lengthen every bucket walk, and under pressure they crowd out entries a retransmit could still hit.
After this series, the DRC retires an entry once the server transport has seen the client peer acknowledge its reply. Each transport publishes two positions, the reply it has just sent and the point up to which the peer has acknowledged, and the DRC compares only the two (patch 6). Nothing else about transport state is exposed to fs/nfsd. UDP entries and replies sent over a TLS session keep the time-based eviction.
Retention now ends at delivery rather than at consumption. A client whose RPC timer fires after the reply was acknowledged but before its RPC layer read it retransmits, and NFSD executes the call again. The client completes with the first reply and discards the second. Patch 9 works through the cases.
The XID checksum and the 256k entry cap were both sized for a cache full of stale replies. With acknowledged entries retired, the checksum guards only entries that wait out
RC_EXPIRE, and the cap starves the partition case it was never meant for (patches 10-11). The "payload misses" line disappears fromreply_cache_stats.
Quoted 4 lines
---
Changes in v4:
- Track reply positions instead of ack cookies (per Neil's review).
- Makexpt_ida 64-bit counter and drop the xarray (per Neil's review).
You actually made the xpt_id 32-bit and the reply-position 64 bit. I would swap those, or at least make them 48bits each.
As a connection can exist for an arbitrarily long time, if the xpt_id counter ever wraps you can get a collision. The age of cache entries isn't the important age - it is the age of the connection. 2^32 seconds is 136 years, so it probably won't wrap (connections tend to last minutes to hours, not seconds).
2^48 milliseconds is nearly 9 millennia so that certainly won't wrap.
The reply position only needs to outlast the cache entry, not the connection. But it counts in bytes.
10Gbit is 1Gbyte/sec so 32 bits could cycle in 4 seconds :-) (but that is what TCP uses!)
Your code doesn't depend on the underlying size, which is nice. 48 bits would only cycle after 3 days at 10Gbit, which is much longer than the cache residency time.
(I haven't properly looked at the pruning algorithms yet, but I will).
Thanks,
NeilBrown
Quoted 70 lines
- Drop implied ACK and the last-request timestamp (per Neil's review).
- Keep an unsent reply untilRC_EXPIRE(per Neil's review).
- Pin a DRC entry against eviction until its reply has been sent.
- svcrdma: Serialize Send posting so positions match completion order.
- Drop thenfsd_drc_reply_ackedtracepoint.
- Link to v3: https://patch.msgid.link/[email protected]Changes in v3:
- Add the reply-acknowledged callback and its TCP and RDMA reporters
ahead of implied ACK, which now defers to a pending report.
- Split the prune-loop restructure into its own patch.
- Move the eviction tracepoints ahead of the ack patches; the
implied-ACK event now lands with implied ACK.
- Keep err local to thepmap_registerblock insvc_setup_socket()
(per Jeff's review).
- Never movexpt_last_recvbackwards when nfsd threads race.
- Fold theXPT_ORDEREDpatch into its consumer (per Jeff's review).
- State the re-execution exposure of implied-ACK eviction instead of
calling the DRC advisory (per Jeff's review).
- New patch: remove the DRC request checksum andpayload_missesstat.
- New patch: remove the 256k-entry cap on the DRC size.
- Link to v2: https://patch.msgid.link/[email protected]Changes in v2:
- Print the DRC eviction tracepoints' age field as unsigned.
- Link to v1: https://patch.msgid.link/[email protected]---
Chuck Lever (11):
SUNRPC: Assign a unique identifier to eachsvc_xprt
NFSD: Track transport in DRC entries
NFSD: Prepare bucket pruning for out-of-order eviction
NFSD: Add tracepoints for DRC entry eviction
NFSD: Record DRC population in lookup tracepoints
SUNRPC: Publish reply positions for upper-layer consumers
SUNRPC: Publish TCP reply positions
svcrdma: Publish RDMA reply positions
NFSD: Evict acknowledged DRC entries
NFSD: Remove DRC checksum andpayload_missesstat
NFSD: Remove hard cap on duplicate reply cache size.../ABI/testing/procfs-nfsd-reply_cache_stats | 11 +- fs/nfsd/cache.h | 16 +- fs/nfsd/netns.h | 2 - fs/nfsd/nfscache.c | 243 +++++++++++---------- fs/nfsd/nfsd.h | 3 + fs/nfsd/nfssvc.c | 11 +- fs/nfsd/stats.h | 5 - fs/nfsd/trace.h | 47 ++-- include/linux/sunrpc/svc.h | 16 ++ include/linux/sunrpc/svc_rdma.h | 5 +- include/linux/sunrpc/svc_xprt.h | 3 + include/linux/sunrpc/svcsock.h | 3 + include/trace/events/sunrpc.h | 8 +- net/sunrpc/svc.c | 3 + net/sunrpc/svc_xprt.c | 19 +- net/sunrpc/svcsock.c | 29 +++ net/sunrpc/xprtrdma/svc_rdma_backchannel.c | 2 +- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 27 ++- net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + 19 files changed, 286 insertions(+), 168 deletions(-)---
base-commit:abf2077ee32058e60d3f49ecab265d3c4bd953d9
change-id: 20260325-duplicate-reply-cache-f0fe7c7b740cBest regards,
Signature · 1 line
Chuck Lever <[email protected]>
Chuck Lever
maintainerQuoted 2 lines · NeilBrown
You actually made the
xpt_id32-bit and the reply-position 64 bit. I would swap those, or at least make them 48bits each.
The changelog described what I intended rather than what I built. Oops.
Quoted 3 lines
As a connection can exist for an arbitrarily long time, if the
xpt_idcounter ever wraps you can get a collision. The age of cache entries isn't the important age - it is the age of the connection.
Agreed. Two live connections sharing an id would let one connection's acknowledged position evict the other's entries for as long as both live. I’ll make xpt_id a u64. A 64-bit counter cannot wrap within the life of a connection, which is the property the v3 xarray provided. The DRC entry will carry the full 64 bits.
Quoted 2 lines
48 bits would only cycle after 3 days at 10Gbit, which is much longer than the cache residency time.
I'll keep the reply position at 64 bits as well. Nothing depends on the width, and packing both into 48 bits saves four bytes per entry at the cost of a comparison that needs explaining.
Quoted 1 line
(I haven't properly looked at the pruning algorithms yet, but I will).
Thanks. I'll hold posting v5 for that.
Signature · 1 line
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)
Quoted 4 lines · Chuck Lever
A completed DRC entry currently stays in its bucket for 120 seconds whether or not the client already holds the reply. On a busy server those entries lengthen every bucket walk, and under pressure they crowd out entries a retransmit could still hit.
120 seconds .... that isn't very long.
The linux NFS client configures a TCP connection to give up after about 180 seconds. So by the time NFS has decided the connection is broken, closes it, and attempts a new connection, the DRC will already be empty.
TCP connection can hang around for an arbitrarily long time. Default configurations seems to be the order of an hour. Linux/NFS deliberately sets the timeout much shorter.
This happened in
Commit: 7f260e8575bf ("SUNRPC: Enable the keepalive option for TCP sockets")
with a justification that it was needed for NFSv4. This is because NFSv4 never retransmits on the same connection. I think the purpose of this is to detect a server restart quickly. The restarted server won't report RST for an unknown TCP connection if it isn't getting any traffic on it. So keep-alives with associated timeouts were added. Adding keep-alives is sensible to trigger an early RST. I'm not sure the low total timeout is justified...
In any case, There is zero chance an NFS server will get a retransmission of an NFS request over TCP within 120 seconds unless the admin has significantly reduced the timeo / retrans options. So maybe there is no point caching these at all???
Or maybe once we get this more efficient pruning working, we can extend the lifetime for cached TCP replies to something potentially useful, like 15 minutes. Or more.
I lean towards not caching them, and guiding admins to choose a large retrans time, or possibly negotiating with client maintainers to allow more missed keep-alives (tcp_sock_set_keepcnt()) before closing the connection, so that the TCP send buffer on the server serves as the reply cache.
NeilBrown
Chuck Lever
maintainerQuoted 4 lines · NeilBrown
In any case, There is zero chance an NFS server will get a retransmission of an NFS request over TCP within 120 seconds unless the admin has significantly reduced the timeo / retrans options. So maybe there is no point caching these at all???
Agreed that 120 seconds serves nobody on TCP. But the retransmit that matters arrives on a new connection, after a partition or a client timeout, and by then the old connection's send buffer is gone. The DRC is the only mechanism that can handle it.
Quoted 3 lines
Or maybe once we get this more efficient pruning working, we can extend the lifetime for cached TCP replies to something potentially useful, like 15 minutes. Or more.
That's the direction I would prefer to take. We spent a lot of effort on the courteous server so NFSv4 state survives partitions that might last hours. The DRC should cover the same window for NFSv3, if that is at all feasible.
With acknowledged entries retired, that costs only the entries whose reply the peer never acknowledged, which is the set a retransmit can still hit, and the shrinker still bounds the total. UDP, being unconnected, would keep the short timer, since nothing else constrains it.
TLS will need some attention, since a TLS session does not publish an acknowledged receive position. That will have to be follow-on work.
Signature · 1 line
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)
Quoted 12 lines · Chuck Lever
Quoted 4 lines · NeilBrown
In any case, There is zero chance an NFS server will get a retransmission of an NFS request over TCP within 120 seconds unless the admin has significantly reduced the timeo / retrans options. So maybe there is no point caching these at all???
Agreed that 120 seconds serves nobody on TCP. But the retransmit that matters arrives on a new connection, after a partition or a client timeout, and by then the old connection's send buffer is gone. The DRC is the only mechanism that can handle it.
Quoted 8 lines
Quoted 3 lines
Or maybe once we get this more efficient pruning working, we can extend the lifetime for cached TCP replies to something potentially useful, like 15 minutes. Or more.
That's the direction I would prefer to take. We spent a lot of effort on the courteous server so NFSv4 state survives partitions that might last hours. The DRC should cover the same window for NFSv3, if that is at all feasible.
Thanks - this is useful context and feeds into the design I think. Particularly the LRU and pruning.
I imagine having 3 separate LRUs for cache entries.
1/ the current per-bucket LRU - entries on this are prune after
a timeout and when the size gets too big, exactly as present.
2/ a new per-xprt LRU which tracks all the request on that xprt
which are awating an ack. Entries are removed when the ack arrives. This means a per-xprt lock for the queue, and taking a per bucket lock for each entry we need to remove.
When the xprt closes, everything needs to be moved to the third queue.
3/ a second per-bucket LRU much like the first one but with a much
longer timeout - several hours. This is where entries go when the xprt is closed. If a new request arrives which matches the entry, it is moved to the relevant per-xprt LRU. Probably it is removed from this LRU, then added to the per-xprt one when it is sent ... maybe.
I had concerns about the changes you made to the LRU code - not a concrete sense of a problem, but more a sense that I couldn't understand what was happening enough to know if there was a problem.
I think the above design, particularly keeping the old LRU as-is and adding new LRUs for new needs, addresses those concerns.
Quoted 9 lines
With acknowledged entries retired, that costs only the entries whose reply the peer never acknowledged, which is the set a retransmit can still hit, and the shrinker still bounds the total. UDP, being unconnected, would keep the short timer, since nothing else constrains it.
TLS will need some attention, since a TLS session does not publish an acknowledged receive position. That will have to be follow-on work.
That is reasonable. There must some way to get the detail out, but leaving it for follow-on seems a good choice.
Thanks,
NeilBrown
Chuck Lever
maintainerQuoted 41 lines · NeilBrown
Quoted 12 lines · Chuck Lever
Quoted 4 lines · NeilBrown
In any case, There is zero chance an NFS server will get a retransmission of an NFS request over TCP within 120 seconds unless the admin has significantly reduced the timeo / retrans options. So maybe there is no point caching these at all???
Agreed that 120 seconds serves nobody on TCP. But the retransmit that matters arrives on a new connection, after a partition or a client timeout, and by then the old connection's send buffer is gone. The DRC is the only mechanism that can handle it.
Quoted 8 lines
Quoted 3 lines
Or maybe once we get this more efficient pruning working, we can extend the lifetime for cached TCP replies to something potentially useful, like 15 minutes. Or more.
That's the direction I would prefer to take. We spent a lot of effort on the courteous server so NFSv4 state survives partitions that might last hours. The DRC should cover the same window for NFSv3, if that is at all feasible.
Thanks - this is useful context and feeds into the design I think. Particularly the LRU and pruning.
I imagine having 3 separate LRUs for cache entries.
1/ the current per-bucket LRU - entries on this are prune after
a timeout and when the size gets too big, exactly as present.
2/ a new per-xprt LRU which tracks all the request on that xprt
which are awating an ack. Entries are removed when the ack arrives. This means a per-xprt lock for the queue, and taking a per bucket lock for each entry we need to remove.
When the xprt closes, everything needs to be moved to the third queue.
3/ a second per-bucket LRU much like the first one but with a much
longer timeout - several hours. This is where entries go when the xprt is closed. If a new request arrives which matches the entry, it is moved to the relevant per-xprt LRU. Probably it is removed from this LRU, then added to the per-xprt one when it is sent ... maybe.
Quoted 5 lines
I had concerns about the changes you made to the LRU code - not a concrete sense of a problem, but more a sense that I couldn't understand what was happening enough to know if there was a problem.
I think the above design, particularly keeping the old LRU as-is and adding new LRUs for new needs, addresses those concerns.
You're jumping ahead of me a little. The next series I have in this area implements per-xprt DRCs for connected transports, and it provides some of what you describe above.
So shall I post a v5 that includes that work, or do you think merging the v4 work now (with adjustments) is acceptable?
Signature · 1 line
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)
Quoted 49 lines · Chuck Lever
Quoted 41 lines · NeilBrown
Quoted 12 lines · Chuck Lever
Quoted 4 lines · NeilBrown
In any case, There is zero chance an NFS server will get a retransmission of an NFS request over TCP within 120 seconds unless the admin has significantly reduced the timeo / retrans options. So maybe there is no point caching these at all???
Agreed that 120 seconds serves nobody on TCP. But the retransmit that matters arrives on a new connection, after a partition or a client timeout, and by then the old connection's send buffer is gone. The DRC is the only mechanism that can handle it.
Quoted 8 lines
Quoted 3 lines
Or maybe once we get this more efficient pruning working, we can extend the lifetime for cached TCP replies to something potentially useful, like 15 minutes. Or more.
That's the direction I would prefer to take. We spent a lot of effort on the courteous server so NFSv4 state survives partitions that might last hours. The DRC should cover the same window for NFSv3, if that is at all feasible.
Thanks - this is useful context and feeds into the design I think. Particularly the LRU and pruning.
I imagine having 3 separate LRUs for cache entries.
1/ the current per-bucket LRU - entries on this are prune after
a timeout and when the size gets too big, exactly as present.
2/ a new per-xprt LRU which tracks all the request on that xprt
which are awating an ack. Entries are removed when the ack arrives. This means a per-xprt lock for the queue, and taking a per bucket lock for each entry we need to remove.
When the xprt closes, everything needs to be moved to the third queue.
3/ a second per-bucket LRU much like the first one but with a much
longer timeout - several hours. This is where entries go when the xprt is closed. If a new request arrives which matches the entry, it is moved to the relevant per-xprt LRU. Probably it is removed from this LRU, then added to the per-xprt one when it is sent ... maybe.Quoted 5 lines
I had concerns about the changes you made to the LRU code - not a concrete sense of a problem, but more a sense that I couldn't understand what was happening enough to know if there was a problem.
I think the above design, particularly keeping the old LRU as-is and adding new LRUs for new needs, addresses those concerns.
Quoted 6 lines
You're jumping ahead of me a little. The next series I have in this area implements per-xprt DRCs for connected transports, and it provides some of what you describe above.
So shall I post a v5 that includes that work, or do you think merging the v4 work now (with adjustments) is acceptable?
I don't think the intermediate LRU makes any sense.
The change to the eviction walk to only 4 times the maximum number to free is arbitrary and unjustified.
I understand that for evicting acknowledged replies you need to move past unacknowledged replies (for another transport), and that you are cautious about walking the whole list because you don't have control of the length, but the "4 times" doesn't even seem like good intermediate solution, and certainly not a good final solution.
If you want to keep the per-xprt LRU for a separate series (and I understand the appeal of that), I would rather you leave the LRU alone, only adding the extra "acknowledged replies can be freed if you happen to find them" case without changing how the LRU is handled in general. The commit which adds this case would include a note that this will miss a lot of replies, but that future work will change the LRU.
The removal of the hard cap is, I think, independent of the other changes - is that right? In that case I would probably put is first to make this clear.
Thanks,
NeilBrown
Quoted 5 lines
Signature · 2 lines
-- Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)