Improve the scalability of NFSD's classic DRC
Subsystems (4)
Chuck Lever
maintainerv3 cover Improve the scalability of NFSD's classic DRC
A completed DRC entry currently stays in its bucket for 120 seconds whether or not the client already holds the reply. On a busy server those entries lengthen every bucket walk, and under pressure they crowd out entries a retransmit could still hit.
This series modifies the DRC to retire an entry once there is reason to believe its reply was delivered.
1. The transport reports delivery outright where it can. TCP now reports when snd_una passes the reply's sequence number, and RDMA reports on each Send completion.
2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).
UDP continues to use a traditional time-based eviction mechanism.
The first mechanism is the more reliable of the two, and the backup mechanism is more of an informed guess. A pipelined client sends its next request before the previous reply lands, and an entry evicted in that window is lost if the connection then drops and the client retransmits. In the current NFSD code, memory pressure and RC_EXPIRE already open this same window. For instance, on a server with fast networking and storage, pressure eviction already evicts entries far younger than 120 seconds.
Two DRC capacity guards that were sized for a cache full of stale replies can be removed, now that reply delivery reports keep the DRC small. The commit messages for those patches explain the rationale in detail.
v2 of this series (implied ACK only) was profiled with "perf record -e cycles -e cpu-clock -e LLC-load-misses -e branch-misses" during an NFSv3/RDMA 4KB random-write workload. v3 has not been re-profiled. nfsd_cache_lookup overhead dropped from 1.53% to 0.76% of CPU cycles during this test. The rb-tree operations (rb_erase, rb_insert_color) that dominated LLC cache misses fell from a combined 13.2% to 1.1% of all LLC-load-misses, because shorter-lived entries keep the per-bucket trees small.
---
Changes in v3:
- Add the reply-acknowledged callback and its TCP and RDMA reporters
ahead of implied ACK, which now defers to a pending report.
- Split the prune-loop restructure into its own patch.
- Move the eviction tracepoints ahead of the ack patches; the
implied-ACK event now lands with implied ACK.
- Keep err local to the pmap_register block in svc_setup_socket()
(per Jeff's review).
- Never move xpt_last_recv backwards when nfsd threads race.
- Fold the XPT_ORDERED patch into its consumer (per Jeff's review).
- State the re-execution exposure of implied-ACK eviction instead of
calling the DRC advisory (per Jeff's review).
- New patch: remove the DRC request checksum and payload_misses stat.
- New patch: remove the 256k-entry cap on the DRC size.
- Link to v2: https://patch.msgid.link/[email protected]
Changes in v2:
- Print the DRC eviction tracepoints' age field as unsigned.
- Link to v1: https://patch.msgid.link/[email protected]
---
Chuck Lever (12):
SUNRPC: Assign a unique identifier to each svc_xprt
NFSD: Track transport in DRC entries
NFSD: Prepare bucket pruning for out-of-order eviction
NFSD: Add tracepoints for DRC entry eviction
NFSD: Record DRC population in lookup tracepoints
NFSD: Add reply-acknowledged callback infrastructure
SUNRPC: Add TCP sequence-number ACK tracking for reply delivery
svcrdma: Fire reply-acknowledged callback on Send completion
SUNRPC: Record last-request timestamp on svc_xprt
NFSD: Evict unacknowledged DRC entries via implied ACK
NFSD: Remove DRC checksum and payload_misses stat
NFSD: Remove hard cap on duplicate reply cache size
.../ABI/testing/procfs-nfsd-reply_cache_stats | 11 +- fs/nfsd/cache.h | 20 +- fs/nfsd/netns.h | 2 - fs/nfsd/nfscache.c | 310 +++++++++++++-------- fs/nfsd/nfssvc.c | 20 +- fs/nfsd/stats.h | 5 - fs/nfsd/trace.h | 81 +++++- include/linux/sunrpc/svc.h | 51 ++++ include/linux/sunrpc/svc_rdma.h | 1 + include/linux/sunrpc/svc_xprt.h | 6 +- include/linux/sunrpc/svcsock.h | 10 + include/trace/events/sunrpc.h | 10 +- net/sunrpc/netns.h | 4 + net/sunrpc/sunrpc_syms.c | 2 + net/sunrpc/svc.c | 1 + net/sunrpc/svc_xprt.c | 53 +++- net/sunrpc/svcsock.c | 119 +++++++- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 14 + net/sunrpc/xprtrdma/svc_rdma_transport.c | 7 +- 19 files changed, 541 insertions(+), 186 deletions(-)
---
base-commit: abf2077ee32058e60d3f49ecab265d3c4bd953d9
change-id: 20260325-duplicate-reply-cache-f0fe7c7b740c
Best regards,
Signature · 1 line
Chuck Lever <[email protected]>
Chuck Lever
maintainerPATCH v3 1/12 SUNRPC: Assign a unique identifier to each svc_xprt
A consumer that associates state with a transport without holding a reference cannot detect ABA collisions: once a transport is freed, SLUB may hand out a new svc_xprt at the same address.
Allocate a per-netns identifier for each transport in svc_xprt_init() with xa_alloc_cyclic(), which delays reuse of an identifier after its transport is freed. svc_xprt_init() now returns a boolean, and its callers unwind when allocation fails. The NFSD duplicate reply cache is the first consumer.
--- include/linux/sunrpc/svc_xprt.h | 3 ++- include/trace/events/sunrpc.h | 8 +++++-- net/sunrpc/netns.h | 4 ++++ net/sunrpc/sunrpc_syms.c | 2 ++ net/sunrpc/svc_xprt.c | 39 ++++++++++++++++++++++++++++---- net/sunrpc/svcsock.c | 33 +++++++++++++++++++-------- net/sunrpc/xprtrdma/svc_rdma_transport.c | 5 +++- 7 files changed, 75 insertions(+), 19 deletions(-)
include/linux/sunrpc/svc_xprt.h +2 -1
@@ -56,6 +56,7 @@ struct svc_xprt {struct svc_xprt_class *xpt_class;const struct svc_xprt_ops *xpt_ops;struct kref xpt_ref; +unsigned int xpt_id;ktime_t xpt_qtime;struct list_head xpt_list;struct lwq_node xpt_ready;
@@ -162,7 +163,7 @@ static inline bool svc_xprt_is_dead(const struct svc_xprt *xprt)int svc_reg_xprt_class (struct svc_xprt_class *);void svc_unreg_xprt_class (struct svc_xprt_class *); -void svc_xprt_init (struct net *,struct svc_xprt_class *,struct svc_xprt *, +bool svc_xprt_init (struct net *,struct svc_xprt_class *,struct svc_xprt *,struct svc_serv *); int svc_xprt_create_from_sa(struct svc_serv *serv, const char *xprt_name, struct net *net, struct sockaddr *sap,
include/trace/events/sunrpc.h +6 -2
@@ -1986,7 +1986,8 @@ TRACE_EVENT(svc_xprt_create_err, __sockaddr(server, (x)->xpt_locallen) \ __sockaddr(client, (x)->xpt_remotelen) \ __field(unsigned long, flags) \ -__field (unsigned int ,netns_ino ) +__field (unsigned int ,netns_ino ) \ + __field(unsigned int, xpt_id) #define SVC_XPRT_ENDPOINT_ASSIGNMENTS(x) \ do { \
@@ -1996,13 +1997,15 @@ TRACE_EVENT(svc_xprt_create_err, (x)->xpt_remotelen); \ __entry->flags = (x)->xpt_flags; \ __entry->netns_ino = (x)->xpt_net->ns.inum; \ + __entry->xpt_id = (x)->xpt_id; \ } while (0)#define SVC_XPRT_ENDPOINT_FORMAT \ - "server=%pISpc client=%pISpc flags=%s" + "server=%pISpc client=%pISpc xpt_id=%u flags=%s"#define SVC_XPRT_ENDPOINT_VARARGS \ __get_sockaddr(server), __get_sockaddr(client), \ + __entry->xpt_id, \ show_svc_xprt_flags(__entry->flags) TRACE_EVENT(svc_xprt_enqueue,
@@ -2024,6 +2027,7 @@ TRACE_EVENT(svc_xprt_enqueue, xprt->xpt_remotelen); __entry->flags = flags; __entry->netns_ino = xprt->xpt_net->ns.inum; + __entry->xpt_id = xprt->xpt_id; ),TP_printk (SVC_XPRT_ENDPOINT_FORMAT ,SVC_XPRT_ENDPOINT_VARARGS )
net/sunrpc/netns.h +4 -0
@@ -2,6 +2,7 @@#ifndef __SUNRPC_NETNS_H__#define __SUNRPC_NETNS_H__ +#include <linux/xarray.h> #include <net/net_namespace.h> #include <net/netns/generic.h>
@@ -34,6 +35,9 @@ struct sunrpc_net {atomic_t pipe_users;struct proc_dir_entry *use_gssp_proc;struct proc_dir_entry *gss_krb5_enctypes; + +struct xarray svc_xprt_ids; +u32 svc_xprt_id_next; };extern unsigned int sunrpc_net_id;
net/sunrpc/sunrpc_syms.c +2 -0
@@ -58,6 +58,7 @@ static __net_init int sunrpc_init_net(struct net *net)spin_lock_init (&sn->rpc_client_lock);spin_lock_init (&sn->rpcb_clnt_lock);mutex_init (&sn->gssp_lock); +xa_init_flags (&sn->svc_xprt_ids,XA_FLAGS_ALLOC1 );return 0 ; err_pipefs:
@@ -74,6 +75,7 @@ static __net_exit void sunrpc_exit_net(struct net *net) {struct sunrpc_net *sn =net_generic (net, sunrpc_net_id); +xa_destroy (&sn->svc_xprt_ids);rpc_pipefs_exit_net (net);unix_gid_cache_destroy (net);ip_map_cache_destroy (net);
net/sunrpc/svc_xprt.c +34 -5
@@ -21,6 +21,8 @@#include <linux/netdevice.h> #include <trace/events/sunrpc.h> +#include "netns.h" +#define RPCDBG_FACILITY RPCDBG_SVCXPRTstatic unsigned int svc_rpc_per_connection_limit__read_mostly ;
@@ -169,7 +171,10 @@ static void svc_xprt_free(struct kref *kref) {struct svc_xprt *xprt = container_of(kref, struct svc_xprt, xpt_ref); +struct sunrpc_net *sn =net_generic (xprt->xpt_net, sunrpc_net_id);struct module *owner = xprt->xpt_class->xcl_owner; + +xa_erase (&sn->svc_xprt_ids, xprt->xpt_id);if (test_bit (XPT_CACHE_AUTH , &xprt->xpt_flags))svcauth_unix_info_release (xprt);put_cred (xprt->xpt_cred);
@@ -190,13 +195,28 @@ void svc_xprt_put(struct svc_xprt *xprt) }EXPORT_SYMBOL_GPL (svc_xprt_put); -/* -* Called by transport drivers to initialize the transport independent -* portion of the transport instance. +/** +* svc_xprt_init - initialize transport-independent portion of a transport +* @net: network namespace in which the transport operates +* @xcl: transport class providing operations and metadata +* @xprt: svc_xprt to initialize +* @serv: RPC service that owns this transport +* +* Assigns @xprt->xpt_id, unique among the transports live in @net. The +* id value can be reused once @xprt is freed. +* +* Context: Process context. May sleep. +* +* Return: +* %true: initialization succeeded +* %false: initialization failed */ -void svc_xprt_init (struct net *net,struct svc_xprt_class *xcl, +bool svc_xprt_init (struct net *net,struct svc_xprt_class *xcl,struct svc_xprt *xprt,struct svc_serv *serv) { +struct sunrpc_net *sn =net_generic (net, sunrpc_net_id); +u32 id; +memset (xprt,0 ,sizeof (*xprt)); xprt->xpt_class = xcl; xprt->xpt_ops = xcl->xcl_ops;
@@ -208,8 +228,17 @@ void svc_xprt_init(struct net *net, struct svc_xprt_class *xcl,mutex_init (&xprt->xpt_mutex);spin_lock_init (&xprt->xpt_lock);set_bit (XPT_BUSY , &xprt->xpt_flags); - xprt->xpt_net =get_net_track (net, &xprt->ns_tracker,GFP_ATOMIC ); + xprt->xpt_net =get_net_track (net, &xprt->ns_tracker,GFP_KERNEL );strcpy (xprt->xpt_remotebuf,"uninitialized" ); + +if (xa_alloc_cyclic (&sn->svc_xprt_ids, &id, xprt, +XA_LIMIT (1 ,UINT_MAX ), &sn->svc_xprt_id_next, +GFP_KERNEL ) <0 ) { +put_net_track (xprt->xpt_net, &xprt->ns_tracker); +return false; + } + xprt->xpt_id = id; +return true; }EXPORT_SYMBOL_GPL (svc_xprt_init);
net/sunrpc/svcsock.c +23 -10
@@ -803,10 +803,11 @@ static struct svc_xprt_class svc_udp_class = { .xcl_flags = SVC_XPRT_FLAG_WSPACE_RESERVE, }; -static void svc_udp_init (struct svc_sock *svsk,struct svc_serv *serv) +static bool svc_udp_init (struct svc_sock *svsk,struct svc_serv *serv) { -svc_xprt_init (sock_net (svsk->sk_sock->sk), &svc_udp_class, - &svsk->sk_xprt, serv); +if (!svc_xprt_init (sock_net (svsk->sk_sock->sk), &svc_udp_class, + &svsk->sk_xprt, serv)) +return false;clear_bit (XPT_CACHE_AUTH , &svsk->sk_xprt.xpt_flags); svsk->sk_sk->sk_data_ready = svc_data_ready; svsk->sk_sk->sk_write_space = svc_write_space;
@@ -833,6 +834,7 @@ static void svc_udp_init(struct svc_sock *svsk, struct svc_serv *serv)default :BUG (); } +return true; }/*
@@ -1476,12 +1478,13 @@ void svc_cleanup_xprt_sock(void)svc_unreg_xprt_class (&svc_udp_class); } -static void svc_tcp_init (struct svc_sock *svsk,struct svc_serv *serv) +static bool svc_tcp_init (struct svc_sock *svsk,struct svc_serv *serv) {struct sock *sk = svsk->sk_sk; -svc_xprt_init (sock_net (svsk->sk_sock->sk), &svc_tcp_class, - &svsk->sk_xprt, serv); +if (!svc_xprt_init (sock_net (svsk->sk_sock->sk), &svc_tcp_class, + &svsk->sk_xprt, serv)) +return false;set_bit (XPT_CACHE_AUTH , &svsk->sk_xprt.xpt_flags);set_bit (XPT_CONG_CTRL , &svsk->sk_xprt.xpt_flags);if (sk->sk_state ==TCP_LISTEN ) {
@@ -1512,6 +1515,7 @@ static void svc_tcp_init(struct svc_sock *svsk, struct svc_serv *serv)svc_xprt_deferred_close (&svsk->sk_xprt); } } +return true; }void svc_sock_update_bufs (struct svc_serv *serv)
@@ -1603,13 +1607,22 @@ static struct svc_sock *svc_setup_socket(struct svc_serv *serv, inet->sk_user_data = svsk;/* Initialize the socket */ -if (sock->type ==SOCK_DGRAM ) -svc_udp_init (svsk, serv); -else -svc_tcp_init (svsk, serv); +if (sock->type ==SOCK_DGRAM ) { +if (!svc_udp_init (svsk, serv)) + goto out_free; + }else { +if (!svc_tcp_init (svsk, serv)) + goto out_free; + }trace_svcsock_new (svsk, sock);return svsk; + +out_free: + inet->sk_user_data =NULL ; +kfree (svsk->sk_bvec); +kfree (svsk); +return ERR_PTR (-ENOMEM ); }/**
net/sunrpc/xprtrdma/svc_rdma_transport.c +4 -1
@@ -189,7 +189,10 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,if (!cma_xprt)return NULL ; -svc_xprt_init (net, &svc_rdma_class, &cma_xprt->sc_xprt, serv); +if (!svc_xprt_init (net, &svc_rdma_class, &cma_xprt->sc_xprt, serv)) { +kfree (cma_xprt); +return NULL ; + }INIT_LIST_HEAD (&cma_xprt->sc_accept_q);INIT_LIST_HEAD (&cma_xprt->sc_rq_dto_q);INIT_LIST_HEAD (&cma_xprt->sc_read_complete_q);
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 2/12 NFSD: Track transport in DRC entries
In reply to Chuck LeverA cached reply carries no record of the transport its request arrived on. Transport-aware eviction needs that association.
Record the xprt id in each entry. No behavior change.
--- fs/nfsd/cache.h | 1 + fs/nfsd/nfscache.c | 1 + 2 files changed, 2 insertions(+)
fs/nfsd/cache.h +1 -0
@@ -37,6 +37,7 @@ struct nfsd_cacherep {unsigned char c_state,/* unused, inprog, done */ c_type,/* status, buffer */ c_secure : 1; /* req came from port < 1024 */ +unsigned int c_xprt;/* svc_xprt that carried req */ unsigned long c_timestamp;union {struct kvec u_vec;
fs/nfsd/nfscache.c +1 -0
@@ -109,6 +109,7 @@ nfsd_cacherep_alloc(struct svc_rqst *rqstp, __wsum csum, rp->c_key.k_vers = rqstp->rq_vers; rp->c_key.k_len = rqstp->rq_arg.len; rp->c_key.k_csum = csum; + rp->c_xprt = rqstp->rq_xprt->xpt_id; }return rp; }
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 3/12 NFSD: Prepare bucket pruning for out-of-order eviction
In reply to Chuck Levernfsd_prune_bucket_locked() evicts for memory pressure and RC_EXPIRE. Both reasons hold for every entry older than the first one they hold for, so the walk stops at the first entry it cannot evict. An eviction reason that applies to one entry but not to an older one, such as confirmation that a particular reply reached the client, cannot be added to that loop.
Restructure the loop so each reason is a separate test that jumps to a shared eviction label, and continue past an entry no reason applies to. The restructured scan no longer stops at the first entry it cannot evict, so bound the work per call to four times the eviction limit. Pass the shrinker's remaining nr_to_scan budget as that limit, bounding its walk of each bucket the same way.
--- fs/nfsd/nfscache.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-)
fs/nfsd/nfscache.c +24 -12
@@ -257,8 +257,8 @@ nfsd_cache_bucket_find(__be32 xid, struct nfsd_net *nn) }/* -* Remove and return no more than @max expired entries in bucket @b. -* If @max is zero, do not limit the number of removed entries. +* Remove and return no more than @max evictable entries in bucket @b, +* visiting at most 4 * @max entries. @max must not be zero. */ static void nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,
@@ -266,20 +266,31 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b, {unsigned long expiry = jiffies -RC_EXPIRE ;struct nfsd_cacherep *rp, *tmp; -unsigned int freed =0 ; +unsigned int freed =0 , visited =0 ;lockdep_assert_held (&b->cache_lock);/* The bucket LRU is ordered oldest-first. */ list_for_each_entry_safe (rp, tmp, &b->lru_head, c_lru) { -if (atomic_read (&nn->num_drc_entries) <= nn->max_drc_entries && -time_before (expiry, rp->c_timestamp)) -break ; +if (atomic_read (&nn->num_drc_entries) > nn->max_drc_entries) + goto evict; +if (time_before_eq (rp->c_timestamp, expiry)) + goto evict; + goto next; +evict:nfsd_cacherep_unlink_locked (nn, b, rp);list_add (&rp->c_lru, dispose); + freed++; -if (max && ++freed >= max) +next: +/* +* A client controls its XIDs, so it can pack one bucket with +* entries that are not yet evictable and turn each miss into +* a full-bucket walk under cache_lock. A skipped entry is +* left for a later prune. +*/ +if (freed >= max || ++visited >= max *4 )break ; } }
@@ -307,9 +318,9 @@ nfsd_reply_cache_count(struct shrinker *shrink, struct shrink_control *sc)* @shrink: our registered shrinker context * @sc: garbage collection parameters * -* Free expired entries on each bucket's LRU list until we've released -* nr_to_scan freed objects. Nothing will be released if the cache -* has not exceeded it's max_drc_entries limit. +* Free entries on each bucket's LRU list until nr_to_scan objects have been +* released. Entries are evicted when they have expired or the cache exceeds +* its max_drc_entries limit. * * Returns the number of entries released by this call. */
@@ -328,11 +339,12 @@ nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc)continue ;spin_lock (&b->cache_lock); -nfsd_prune_bucket_locked (nn, b,0 , &dispose); +nfsd_prune_bucket_locked (nn, b, sc->nr_to_scan - freed, + &dispose);spin_unlock (&b->cache_lock); freed +=nfsd_cacherep_dispose (&dispose); -if (freed > sc->nr_to_scan) +if (freed >= sc->nr_to_scan)break ; }return freed;
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 4/12 NFSD: Add tracepoints for DRC entry eviction
In reply to Chuck LeverThe DRC pruning path evicts entries for memory pressure and expiry, and neither emits a trace event, so there is no way to see which reason is retiring entries or how old they are when it happens. Eviction reasons that follow will need the same visibility.
Add an event class that records the cache population, the XID, the transport, and the entry's age, and define an event for each eviction reason.
--- fs/nfsd/nfscache.c | 8 ++++++-- fs/nfsd/trace.h | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 2 deletions(-)
fs/nfsd/nfscache.c +6 -2
@@ -272,10 +272,14 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,/* The bucket LRU is ordered oldest-first. */ list_for_each_entry_safe (rp, tmp, &b->lru_head, c_lru) { -if (atomic_read (&nn->num_drc_entries) > nn->max_drc_entries) +if (atomic_read (&nn->num_drc_entries) > nn->max_drc_entries) { +trace_nfsd_drc_evict_pressure (nn, rp); goto evict; -if (time_before_eq (rp->c_timestamp, expiry)) + } +if (time_before_eq (rp->c_timestamp, expiry)) { +trace_nfsd_drc_evict_expired (nn, rp); goto evict; + } goto next; evict:
fs/nfsd/trace.h +37 -0
@@ -1557,6 +1557,43 @@ TRACE_EVENT(nfsd_drc_mismatch, __entry->ingress) ); +DECLARE_EVENT_CLASS (nfsd_drc_entry_class, +TP_PROTO ( + const struct nfsd_net *nn, + const struct nfsd_cacherep *rp + ), +TP_ARGS (nn, rp), +TP_STRUCT__entry ( + __field(unsigned long long, boot_time) + __field(unsigned int, num_drc_entries) + __field(u32, xid) + __field(unsigned int, xprt) + __field(unsigned long, age) + ), +TP_fast_assign ( + __entry->boot_time = nn->boot_time; + __entry->num_drc_entries = atomic_read(&nn->num_drc_entries); + __entry->xid = be32_to_cpu(rp->c_key.k_xid); + __entry->xprt = rp->c_xprt; + __entry->age =time_is_after_jiffies (rp->c_timestamp) ? + 1 : jiffies - rp->c_timestamp; + ), +TP_printk ("boot_time=%16llx entries=%u xid=0x%08x xprt=%u age=%lu" , + __entry->boot_time, __entry->num_drc_entries, + __entry->xid, __entry->xprt, __entry->age) +); + +#define DEFINE_NFSD_DRC_ENTRY_EVENT (name) \ +DEFINE_EVENT(nfsd_drc_entry_class, nfsd_drc_##name, \ + TP_PROTO( \ + const struct nfsd_net *nn, \ + const struct nfsd_cacherep *rp \ + ), \ + TP_ARGS(nn, rp)) + +DEFINE_NFSD_DRC_ENTRY_EVENT (evict_pressure); +DEFINE_NFSD_DRC_ENTRY_EVENT (evict_expired); + TRACE_EVENT(nfsd_cb_args, TP_PROTO( const struct nfs4_client *clp,
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 5/12 NFSD: Record DRC population in lookup tracepoints
In reply to Chuck Levernfsd_drc_found reports the outcome of a lookup but not how full the cache was at the time, so a trace cannot show whether retransmits are being caught while the cache runs near its cap.
--- fs/nfsd/nfscache.c | 3 ++- fs/nfsd/trace.h | 11 +++++++---- 2 files changed, 9 insertions(+), 5 deletions(-)
fs/nfsd/nfscache.c +2 -1
@@ -562,7 +562,8 @@ int nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start, } out_trace: -trace_nfsd_drc_found (nn, rqstp, rtn); +trace_nfsd_drc_found (nn,atomic_read (&nn->num_drc_entries), + rqstp, rtn); out_unlock:spin_unlock (&b->cache_lock); out:
fs/nfsd/trace.h +7 -4
@@ -1513,23 +1513,26 @@ TRACE_DEFINE_ENUM(RC_DOIT);TRACE_EVENT (nfsd_drc_found,TP_PROTO ( const struct nfsd_net *nn, + unsigned int num_drc_entries, const struct svc_rqst *rqstp, int result ), -TP_ARGS (nn, rqstp, result), +TP_ARGS (nn, num_drc_entries, rqstp, result),TP_STRUCT__entry ( __field(unsigned long long, boot_time) + __field(unsigned int, num_drc_entries) __field(unsigned long, result)__field (u32, xid) ),TP_fast_assign ( __entry->boot_time = nn->boot_time; + __entry->num_drc_entries = num_drc_entries; __entry->result = result; __entry->xid = be32_to_cpu(rqstp->rq_xid); ), -TP_printk ("boot_time=%16llx xid=0x%08x result=%s" , - __entry->boot_time, __entry->xid, -show_drc_retval (__entry->result)) +TP_printk ("boot_time=%16llx entries=%u xid=0x%08x result=%s" , + __entry->boot_time, __entry->num_drc_entries, + __entry->xid,show_drc_retval (__entry->result)) );
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 6/12 NFSD: Add reply-acknowledged callback infrastructure
In reply to Chuck LeverThe DRC retains an entry for RC_EXPIRE whether or not the client has received the reply. TCP and RDMA can confirm delivery, which would allow the entry to be evicted much earlier.
Early eviction is safe because a client retransmits a request only when its reply never arrived. Once the transport confirms the reply reached the client, no retransmit of that XID can follow, so the entry can never again satisfy a lookup. A reply still in flight when a connection drops is never confirmed, so the entries a reconnecting client retransmits against keep their full RC_EXPIRE retention.
Add a callback and opaque private pointer on struct svc_serv so a transport can report reply delivery without depending on NFS types. The report carries an opaque cookie naming the entry by XID, transport identifier, and generation. The generation guards XID
reuse: without it a stale report could mark a successor entry delivered, evicting a reply the client never received. The handler caps its newest-first bucket walk because a client chooses which bucket its XIDs hash to and could otherwise pin cache_lock with an unbounded walk.
A transport that reports on replies sets XPT_REPLY_ACK, and its entries are marked c_ack_pending until the report arrives, so an eviction reason that only guesses at delivery can defer to it.
--- fs/nfsd/cache.h | 10 +++-- fs/nfsd/nfscache.c | 86 ++++++++++++++++++++++++++++++++++++++++- fs/nfsd/nfssvc.c | 10 ++--- fs/nfsd/trace.h | 34 ++++++++++++++++ include/linux/sunrpc/svc.h | 48 +++++++++++++++++++++++ include/linux/sunrpc/svc_xprt.h | 1 + include/trace/events/sunrpc.h | 1 + 7 files changed, 180 insertions(+), 10 deletions(-)
fs/nfsd/cache.h +7 -3
@@ -36,8 +36,11 @@ struct nfsd_cacherep {struct list_head c_lru;unsigned char c_state,/* unused, inprog, done */ c_type,/* status, buffer */ - c_secure : 1; /* req came from port < 1024 */ + c_secure : 1, /* req came from port < 1024 */ + c_acked : 1, /* reply delivery confirmed */ + c_ack_pending : 1; /* transport will report */unsigned int c_xprt;/* svc_xprt that carried req */ +u32 c_ack_gen;/* uniquifies the ack cookie */ unsigned long c_timestamp;union {struct kvec u_vec;
@@ -80,10 +83,11 @@ enum {/* Checksum this amount of the request */ #define RC_CSUMLEN (256U) +svc_ack_cookie_t nfsd_cache_ack_cookie (const struct nfsd_cacherep *rp);int nfsd_drc_slab_create (void );void nfsd_drc_slab_free (void ); -int nfsd_reply_cache_init (struct nfsd_net *); -void nfsd_reply_cache_shutdown (struct nfsd_net *); +int nfsd_reply_cache_init (struct nfsd_net *,struct svc_serv *); +void nfsd_reply_cache_shutdown (struct nfsd_net *,struct svc_serv *);int nfsd_cache_lookup (struct svc_rqst *rqstp,unsigned int start,unsigned int len,struct nfsd_cacherep **cacherep); void nfsd_cache_update(struct svc_rqst *rqstp, struct nfsd_cacherep *rp,
fs/nfsd/nfscache.c +84 -2
152 lines not on this page · raw message
fs/nfsd/nfssvc.c +5 -5
@@ -389,7 +389,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)if (ret) goto out_lockd; - ret =nfsd_reply_cache_init (nn); + ret =nfsd_reply_cache_init (nn, nn->nfsd_serv);if (ret) goto out_filecache;
@@ -404,7 +404,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)return 0 ; out_reply_cache: -nfsd_reply_cache_shutdown (nn); +nfsd_reply_cache_shutdown (nn, nn->nfsd_serv); out_filecache:nfsd_file_cache_shutdown_net (net); out_lockd:
@@ -417,7 +417,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred)return ret; } -static void nfsd_shutdown_net (struct net *net) +static void nfsd_shutdown_net (struct net *net,struct svc_serv *serv) {struct nfsd_net *nn =net_generic (net, nfsd_net_id);
@@ -427,7 +427,7 @@ static void nfsd_shutdown_net(struct net *net)nfsd_export_flush (net);nfs4_state_shutdown_net (net); -nfsd_reply_cache_shutdown (nn); +nfsd_reply_cache_shutdown (nn, serv);nfsd_file_cache_shutdown_net (net);if (test_bit (NFSD_NET_LOCKD_UP , &nn->flags)) {lockd_down (net);
@@ -539,7 +539,7 @@ void nfsd_destroy_serv(struct net *net)* other initialization has been done except the rpcb information. */ svc_xprt_destroy_all (serv, net, true); -nfsd_shutdown_net (net); +nfsd_shutdown_net (net, serv);svc_destroy (&serv); }
fs/nfsd/trace.h +34 -0
@@ -1594,9 +1594,43 @@ DEFINE_EVENT(nfsd_drc_entry_class, nfsd_drc_##name, \ ), \ TP_ARGS(nn, rp)) +DEFINE_NFSD_DRC_ENTRY_EVENT (evict_acked);DEFINE_NFSD_DRC_ENTRY_EVENT (evict_pressure);DEFINE_NFSD_DRC_ENTRY_EVENT (evict_expired); +TRACE_EVENT (nfsd_drc_reply_acked, +TP_PROTO ( + const struct nfsd_net *nn, + unsigned int num_drc_entries, + u32 xid, + unsigned int xprt, + bool delivered, + bool found + ), +TP_ARGS (nn, num_drc_entries, xid, xprt, delivered, found), +TP_STRUCT__entry ( + __field(unsigned long long, boot_time) + __field(unsigned int, num_drc_entries) + __field(u32, xid) + __field(unsigned int, xprt) + __field(bool, delivered) +__field (bool, found) + ), +TP_fast_assign ( + __entry->boot_time = nn->boot_time; + __entry->num_drc_entries = num_drc_entries; + __entry->xid = xid; + __entry->xprt = xprt; + __entry->delivered = delivered; + __entry->found = found; + ), +TP_printk ("boot_time=%16llx entries=%u xid=0x%08x xprt=%u %s %s" , + __entry->boot_time, __entry->num_drc_entries, + __entry->xid, __entry->xprt, + __entry->delivered ?"delivered" :"untracked" , + __entry->found ?"found" :"stale" ) +); + TRACE_EVENT(nfsd_cb_args, TP_PROTO( const struct nfs4_client *clp,
include/linux/sunrpc/svc.h +48 -0
66 lines not on this page · raw message
include/linux/sunrpc/svc_xprt.h +1 -0
@@ -101,6 +101,7 @@ enum {XPT_LOCAL ,/* connection from loopback interface */ XPT_KILL_TEMP ,/* call xpo_kill_temp_xprt before closing */ XPT_CONG_CTRL ,/* has congestion control */ +XPT_REPLY_ACK ,/* reports reply delivery via svc_reply_acked */ XPT_HANDSHAKE ,/* xprt requests a handshake */ XPT_TLS_SESSION ,/* transport-layer security established */ XPT_PEER_AUTH, /* peer has been authenticated */
include/trace/events/sunrpc.h +1 -0
@@ -1931,6 +1931,7 @@ TRACE_EVENT(svc_stats_latency, svc_xprt_flag(LOCAL) \ svc_xprt_flag(KILL_TEMP) \ svc_xprt_flag(CONG_CTRL) \ + svc_xprt_flag(REPLY_ACK) \ svc_xprt_flag(HANDSHAKE) \ svc_xprt_flag(TLS_SESSION) \ svc_xprt_flag(PEER_AUTH) \
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 7/12 SUNRPC: Add TCP sequence-number ACK tracking for reply delivery
In reply to Chuck LeverA DRC entry for a reply sent over TCP lingers until RC_EXPIRE even though the peer's TCP acknowledgment has already confirmed that the reply arrived.
The acknowledgment confirms only that the reply reached the peer's TCP stack. If the connection drops before the NFS client reads it, the client retransmits the call and the evicted entry cannot answer it. Memory pressure and RC_EXPIRE already open the same window.
When the upper layer caches a reply it stores a cookie on the svc_rqst. After svc_tcp_sendto() transmits the reply, record the cookie with the current write_seq in a fixed-size per-socket ring. Before appending, drain entries whose sequence number is at or before snd_una and report each as delivered. The send path is single-threaded, so the ring needs no locking. sk_wmem_queued cannot be used for this because it counts per-skb truesize, not payload bytes.
On a kTLS session the snapshot needs one more check. A short push inside the TLS layer leaves the rest of the record as partially_sent_record, and tls_sw_sendmsg() still returns the full plaintext count, so write_seq can fall short of the reply. Record the reply only when no partially_sent_record remains after the send.
Some replies get no ring entry: the ring is full, the send fails, the transport is already dead, or a kTLS record is still partly unsent. Report each of these as untracked at once, so the upper layer knows no delivery report will follow. When the socket is freed, report entries still in the ring as undelivered. Set XPT_REPLY_ACK on TCP transports so the upper layer expects these reports.
--- fs/nfsd/nfscache.c | 15 +++++--- include/linux/sunrpc/svc.h | 3 ++ include/linux/sunrpc/svcsock.h | 10 +++++ net/sunrpc/svc.c | 1 + net/sunrpc/svcsock.c | 85 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 108 insertions(+), 6 deletions(-)
fs/nfsd/nfscache.c +9 -6
@@ -289,12 +289,14 @@ nfsd_cache_bucket_find(__be32 xid, struct nfsd_net *nn) }/* -* The generation match keeps a stale cookie from acknowledging a -* later entry that reuses the same XID and transport. The walk -* starts at the MRU end of the bucket LRU, where the entry for a -* just-sent reply sits. The visit cap bounds the time spent under -* cache_lock when a client packs the bucket; an entry missed under -* the cap is left to the other eviction reasons. +* The generation match keeps a stale cookie from acknowledging a later +* entry that reuses the same XID and transport. +* +* Average bucket occupancy is small (TARGET_BUCKET_SIZE), so a linear walk +* of the bucket LRU beats an rb-tree lookup. The walk starts at the MRU +* end, where the entry for a just-sent reply sits. The visit cap bounds +* the time spent under cache_lock when a client packs the bucket; an entry +* missed under the cap is left to the other eviction reasons. */ static void nfsd_reply_ack (void *data,const svc_ack_cookie_t *cookie,bool delivered)
@@ -723,6 +725,7 @@ void nfsd_cache_update(struct svc_rqst *rqstp, struct nfsd_cacherep *rp, rp->c_ack_gen =nfsd_cache_next_ack_gen (); rp->c_type = cachetype; rp->c_state =RC_DONE ; + rqstp->rq_ack_cookie =nfsd_cache_ack_cookie (rp);spin_unlock (&b->cache_lock);return ; }
include/linux/sunrpc/svc.h +3 -0
@@ -315,6 +315,9 @@ struct svc_rqst {unsigned int bc_to_retries;unsigned int rq_status_counter;/* RPC processing counter */ void *rq_private;/* For use by the service thread */ + +svc_ack_cookie_t rq_ack_cookie;/* zero when no reply-ack +* tracking is requested */ };/* bits for rq_flags */
include/linux/sunrpc/svcsock.h +10 -0
@@ -41,6 +41,16 @@ struct svc_sock {struct page_frag_cache sk_frag_cache; +/* TCP reply-ACK tracking (single-threaded send path) */ +#define SVC_ACK_RING_BITS 6 +#define SVC_ACK_RING_SIZE (1 << SVC_ACK_RING_BITS) +struct { +u32 ae_pos;/* write_seq after send */ +svc_ack_cookie_t ae_cookie;/* opaque DRC cookie */ + } sk_ack_ring[SVC_ACK_RING_SIZE ]; +unsigned int sk_ack_head; +unsigned int sk_ack_tail; +struct completion sk_handshake_done;/* received data */
net/sunrpc/svc.c +1 -0
@@ -1488,6 +1488,7 @@ svc_process_common(struct svc_rqst *rqstp)/* Reset the accept_stat for the RPC */ rqstp->rq_accept_statp =NULL ; + rqstp->rq_ack_cookie = (svc_ack_cookie_t ){};/* Will be turned off only when NFSv4 Sessions are used */ set_bit (RQ_USEDEFERRAL , &rqstp->rq_flags);
net/sunrpc/svcsock.c +85 -0
149 lines not on this page · raw message
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 8/12 svcrdma: Fire reply-acknowledged callback on Send completion
In reply to Chuck Leversvcrdma posts RDMA Write work requests ahead of the Send, so a successful Send completion confirms that the transport header and the reply payload have both reached the client.
Store rq_ack_cookie in the send context before posting and report the reply as delivered from the Send completion handler, so the DRC can evict the entry without waiting for RC_EXPIRE. Set XPT_REPLY_ACK on RDMA transports so the upper layer expects these reports. A flushed Send reports the reply as undelivered: the connection is closing, and the entry stays cached for a retransmit on the next connection.
--- include/linux/sunrpc/svc_rdma.h | 1 + net/sunrpc/xprtrdma/svc_rdma_sendto.c | 14 ++++++++++++++ net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + 3 files changed, 16 insertions(+)
include/linux/sunrpc/svc_rdma.h +1 -0
@@ -248,6 +248,7 @@ struct svc_rdma_send_ctxt {struct list_head sc_write_info_list;struct svc_rdma_write_info sc_reply_info; +svc_ack_cookie_t sc_ack_cookie;void *sc_xprt_buf;int sc_page_count;int sc_cur_sge_no;
net/sunrpc/xprtrdma/svc_rdma_sendto.c +14 -0
@@ -220,6 +220,7 @@ struct svc_rdma_send_ctxt *svc_rdma_send_ctxt_get(struct svcxprt_rdma *rdma) ctxt->sc_send_wr.num_sge =0 ; ctxt->sc_cur_sge_no =0 ; ctxt->sc_page_count =0 ; + ctxt->sc_ack_cookie = (svc_ack_cookie_t ){}; ctxt->sc_wr_chain = &ctxt->sc_send_wr; ctxt->sc_sqecount =1 ;
@@ -470,11 +471,17 @@ static void svc_rdma_wc_send(struct ib_cq *cq, struct ib_wc *wc)if (unlikely (wc->status !=IB_WC_SUCCESS )) goto flushed; +if (svc_ack_cookie_present (&ctxt->sc_ack_cookie)) +svc_reply_acked (rdma->sc_xprt.xpt_server, + &ctxt->sc_ack_cookie, true);trace_svcrdma_wc_send (&ctxt->sc_cid);svc_rdma_send_ctxt_put (rdma, ctxt);return ; flushed: +if (svc_ack_cookie_present (&ctxt->sc_ack_cookie)) +svc_reply_acked (rdma->sc_xprt.xpt_server, + &ctxt->sc_ack_cookie, false);if (wc->status !=IB_WC_WR_FLUSH_ERR )trace_svcrdma_wc_send_err (wc, &ctxt->sc_cid);else
@@ -1198,6 +1205,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)if (ret <0 ) goto put_ctxt; + sctxt->sc_ack_cookie = rqstp->rq_ack_cookie; ret =svc_rdma_send_reply_msg (rdma, sctxt, rctxt, rqstp);if (ret <0 ) goto send_err;
@@ -1207,6 +1215,12 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)if (ret != -E2BIG && ret != -EINVAL ) goto put_ctxt; +/* The sctxt is reused for the RDMA_ERROR message. Clear the +* ack cookie so that message's Send completion does not +* report the unsent reply as delivered. +*/ + sctxt->sc_ack_cookie = (svc_ack_cookie_t ){}; +/* Send completion releases payload pages that were part * of previously posted RDMA Writes. */
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -0
@@ -218,6 +218,7 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,* transports are suitable here. */ set_bit (XPT_CONG_CTRL , &cma_xprt->sc_xprt.xpt_flags); +set_bit (XPT_REPLY_ACK , &cma_xprt->sc_xprt.xpt_flags);return cma_xprt; }
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 9/12 SUNRPC: Record last-request timestamp on svc_xprt
In reply to Chuck LeverOn a connection-oriented transport, the arrival of a later request suggests the client received the prior reply. It is not proof: a client with several requests outstanding sends the next one without waiting for a reply.
Add xpt_last_recv to struct svc_xprt. svc_handle_xprt() samples jiffies before calling xpo_recvfrom() and publishes the sample after a successful receive. The DRC reads this value as a best-effort signal that a completed reply has been delivered.
--- include/linux/sunrpc/svc_xprt.h | 1 + net/sunrpc/svc_xprt.c | 14 ++++++++++++++ 2 files changed, 15 insertions(+)
include/linux/sunrpc/svc_xprt.h +1 -0
@@ -81,6 +81,7 @@ struct svc_xprt {struct net *xpt_net;netns_tracker ns_tracker;const struct cred *xpt_cred; +unsigned long xpt_last_recv;/* jiffies of last request */ struct rpc_xprt *xpt_bc_xprt;/* NFSv4.1 backchannel */ struct rpc_xprt_switch *xpt_bc_xps;/* NFSv4.1 backchannel */ };
net/sunrpc/svc_xprt.c +14 -0
@@ -230,6 +230,7 @@ bool svc_xprt_init(struct net *net, struct svc_xprt_class *xcl,set_bit (XPT_BUSY , &xprt->xpt_flags); xprt->xpt_net =get_net_track (net, &xprt->ns_tracker,GFP_KERNEL );strcpy (xprt->xpt_remotebuf,"uninitialized" ); + xprt->xpt_last_recv = jiffies; if (xa_alloc_cyclic(&sn->svc_xprt_ids, &id, xprt, XA_LIMIT(1, UINT_MAX), &sn->svc_xprt_id_next,
@@ -901,6 +902,8 @@ static void svc_handle_xprt(struct svc_rqst *rqstp, struct svc_xprt *xprt)svc_xprt_received (xprt); }else if (svc_xprt_reserve_slot (rqstp, xprt)) {/* XPT_DATA|XPT_DEFERRED case: */ +unsigned long recv_time = jiffies; + rqstp->rq_deferred =svc_deferred_dequeue (xprt);if (rqstp->rq_deferred) len =svc_deferred_recv (rqstp);
@@ -916,6 +919,17 @@ static void svc_handle_xprt(struct svc_rqst *rqstp, struct svc_xprt *xprt)clear_bit (XPT_OLD , &xprt->xpt_flags); +/* +* A deferred request arrived before its deferral, so its +* replay does not advance the timestamp. recv_time was +* sampled before xpo_recvfrom() released XPT_BUSY. +* time_after() keeps a slow thread from pushing the +* timestamp past requests received since then. +*/ +if (!rqstp->rq_deferred && +time_after (recv_time,READ_ONCE (xprt->xpt_last_recv))) +WRITE_ONCE (xprt->xpt_last_recv, recv_time); + rqstp->rq_chandle.defer = svc_defer;if (serv->sv_stats)
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 10/12 NFSD: Evict unacknowledged DRC entries via implied ACK
In reply to Chuck LeverA transport reports reply delivery only while it is tracking the reply. The TCP ring has a fixed size, so a burst leaves some replies untracked, and a kTLS reply is untracked when part of its record is still unsent. Entries for those replies stay in their bucket until RC_EXPIRE elapses or the cache exceeds max_drc_entries, lengthening every lookup that hashes there.
RFC 1813 Section 4.5 observes that on a connection-oriented transport a duplicate request arises from reconnection, not from within a live connection. A fresh request on a live TCP or RDMA connection therefore means the client is not retransmitting an earlier one. UDP offers no such signal.
The signal is not conclusive. An entry can be evicted while its reply is still in flight, and a reconnect then executes the request again. Memory pressure and RC_EXPIRE already open the same window.
Add an XPT_ORDERED flag marking transports that offer the signal. During bucket pruning, evict an RC_DONE entry when the current request's transport is XPT_ORDERED, no delivery report is pending, and its c_timestamp predates xpt_last_recv. Only that transport is consulted, because it is the one the pruning thread holds a reference to.
On svcrdma, handle_connect_req() zeroes the remote port in the DRC key so a cached reply survives a reconnect, and implied-ACK eviction can retire such an entry while the original connection is still up. TCP gives up the same thing when a client re-binds its reserved port.
--- fs/nfsd/nfscache.c | 45 +++++++++++++++++++++++++++++--- fs/nfsd/trace.h | 1 + include/linux/sunrpc/svc_xprt.h | 1 + include/trace/events/sunrpc.h | 1 + net/sunrpc/svcsock.c | 1 + net/sunrpc/xprtrdma/svc_rdma_transport.c | 1 + 6 files changed, 47 insertions(+), 3 deletions(-)
fs/nfsd/nfscache.c +42 -3
@@ -90,6 +90,36 @@ nfsd_hashsize(unsigned int limit)return roundup_pow_of_two (limit /TARGET_BUCKET_SIZE ); } +/* +* A later request on @xprt is taken as evidence that the client +* received @rp's reply, because a client does not retransmit within +* a live connection. Only XPT_ORDERED transports qualify: a UDP +* client retransmits on timeout, and a datagram from any of the +* peers sharing the UDP svc_xprt would evict another client's reply. +* +* A pipelined client sends its next request before @rp's reply +* arrives, so while the transport is still going to report on that +* reply, wait for the report instead. +* +* c_timestamp is set after xpt_last_recv was recorded for @rp's own +* request, so a newer xpt_last_recv means a later request arrived. +*/ +static bool nfsd_cacherep_implied_ack (struct svc_xprt *xprt, +struct nfsd_cacherep *rp) +{ +unsigned long last_req; + +if (!xprt || rp->c_xprt != xprt->xpt_id) +return false; +if (!test_bit (XPT_ORDERED , &xprt->xpt_flags)) +return false; +if (rp->c_ack_pending) +return false; + + last_req =READ_ONCE (xprt->xpt_last_recv); +return time_after (last_req, rp->c_timestamp); +} +static struct nfsd_cacherep *nfsd_cacherep_alloc (struct svc_rqst *rqstp,__wsum csum,struct nfsd_net *nn)
@@ -337,10 +367,14 @@ static void nfsd_reply_ack(void *data, const svc_ack_cookie_t *cookie,/* * Remove and return no more than @max evictable entries in bucket @b, * visiting at most 4 * @max entries. @max must not be zero. +* +* @xprt is the transport the current request arrived on, or NULL when the +* caller has none. */ static void nfsd_prune_bucket_locked (struct nfsd_net *nn,struct nfsd_drc_bucket *b, -unsigned int max,struct list_head *dispose) +unsigned int max,struct list_head *dispose, +struct svc_xprt *xprt) {unsigned long expiry = jiffies -RC_EXPIRE ;struct nfsd_cacherep *rp, *tmp;
@@ -362,6 +396,11 @@ nfsd_prune_bucket_locked(struct nfsd_net *nn, struct nfsd_drc_bucket *b,trace_nfsd_drc_evict_expired (nn, rp); goto evict; } +if (rp->c_state ==RC_DONE && +nfsd_cacherep_implied_ack (xprt, rp)) { +trace_nfsd_drc_evict_implied_ack (nn, rp); + goto evict; + } goto next; evict:
@@ -426,7 +465,7 @@ nfsd_reply_cache_scan(struct shrinker *shrink, struct shrink_control *sc)spin_lock (&b->cache_lock);nfsd_prune_bucket_locked (nn, b, sc->nr_to_scan - freed, - &dispose); + &dispose,NULL );spin_unlock (&b->cache_lock); freed +=nfsd_cacherep_dispose (&dispose);
@@ -599,7 +638,7 @@ int nfsd_cache_lookup(struct svc_rqst *rqstp, unsigned int start, goto found_entry; *cacherep = rp; rp->c_state =RC_INPROG ; -nfsd_prune_bucket_locked (nn, b,3 , &dispose); +nfsd_prune_bucket_locked (nn, b,3 , &dispose, rqstp->rq_xprt);spin_unlock (&b->cache_lock);nfsd_cacherep_dispose (&dispose);
fs/nfsd/trace.h +1 -0
@@ -1597,6 +1597,7 @@ DEFINE_EVENT(nfsd_drc_entry_class, nfsd_drc_##name, \DEFINE_NFSD_DRC_ENTRY_EVENT (evict_acked);DEFINE_NFSD_DRC_ENTRY_EVENT (evict_pressure);DEFINE_NFSD_DRC_ENTRY_EVENT (evict_expired); +DEFINE_NFSD_DRC_ENTRY_EVENT (evict_implied_ack); TRACE_EVENT(nfsd_drc_reply_acked, TP_PROTO(
include/linux/sunrpc/svc_xprt.h +1 -0
@@ -103,6 +103,7 @@ enum {XPT_KILL_TEMP ,/* call xpo_kill_temp_xprt before closing */ XPT_CONG_CTRL ,/* has congestion control */ XPT_REPLY_ACK ,/* reports reply delivery via svc_reply_acked */ +XPT_ORDERED ,/* connection-oriented, in-order delivery */ XPT_HANDSHAKE ,/* xprt requests a handshake */ XPT_TLS_SESSION ,/* transport-layer security established */ XPT_PEER_AUTH, /* peer has been authenticated */
include/trace/events/sunrpc.h +1 -0
@@ -1932,6 +1932,7 @@ TRACE_EVENT(svc_stats_latency, svc_xprt_flag(KILL_TEMP) \ svc_xprt_flag(CONG_CTRL) \ svc_xprt_flag(REPLY_ACK) \ + svc_xprt_flag(ORDERED) \ svc_xprt_flag(HANDSHAKE) \ svc_xprt_flag(TLS_SESSION) \ svc_xprt_flag(PEER_AUTH) \
net/sunrpc/svcsock.c +1 -0
@@ -1567,6 +1567,7 @@ static bool svc_tcp_init(struct svc_sock *svsk, struct svc_serv *serv)set_bit (XPT_CACHE_AUTH , &svsk->sk_xprt.xpt_flags);set_bit (XPT_CONG_CTRL , &svsk->sk_xprt.xpt_flags);set_bit (XPT_REPLY_ACK , &svsk->sk_xprt.xpt_flags); +set_bit (XPT_ORDERED , &svsk->sk_xprt.xpt_flags);if (sk->sk_state ==TCP_LISTEN ) {strcpy (svsk->sk_xprt.xpt_remotebuf,"listener" );set_bit (XPT_LISTENER , &svsk->sk_xprt.xpt_flags);
net/sunrpc/xprtrdma/svc_rdma_transport.c +1 -0
@@ -219,6 +219,7 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,*/ set_bit (XPT_CONG_CTRL , &cma_xprt->sc_xprt.xpt_flags);set_bit (XPT_REPLY_ACK , &cma_xprt->sc_xprt.xpt_flags); +set_bit (XPT_ORDERED , &cma_xprt->sc_xprt.xpt_flags);return cma_xprt; }
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 11/12 NFSD: Remove DRC checksum and payload_misses stat
In reply to Chuck Levernfsd_cache_csum() costs CPU on every non-idempotent request. It also prevents the use of zero-copy RDMA receives for WRITE and SYMLINK because the payload must be in the server's memory before the DRC lookup can run.
Commit 01a7decf7593 ("nfsd: keep a checksum of the first 256 bytes of request") added the checksum when a growing cache made XID collisions easier to hit. It is the only guard against a reused
XID: two calls with the same procedure and equal-length arguments match on every other key field.
ACK-driven eviction retires a TCP or RDMA entry once the transport confirms delivery, so a fresh XID rarely finds a resident entry to collide with (~100/2^32 per request). A UDP entry can wait out RC_EXPIRE, but the Linux NFS client seeds its XIDs from get_random_u32(), so even a rebooted client does not replay its previous sequence. A client that repeats a live XID cannot reliably match replies to its own calls anyway.
An acknowledged entry stays in its bucket until the next prune visits it, and a lookup matches it in the meantime. The client already holds that reply, so treat a call carrying its XID as a
miss: evict the entry and insert the new one in its place.
Remove nfsd_cache_csum(), RC_CSUMLEN, and k_csum, along with the nfsd_drc_mismatch tracepoint and the payload_misses stat that counted checksum-detected collisions. Drop the start and len parameters from nfsd_cache_lookup(), so nfsd_dispatch() no longer snapshots the argument stream before decoding.
The "payload misses" line disappears from /proc/fs/nfsd/reply_cache_stats. No known userspace tool parses it.
--- .../ABI/testing/procfs-nfsd-reply_cache_stats | 11 +-- fs/nfsd/cache.h | 9 +- fs/nfsd/netns.h | 2 - fs/nfsd/nfscache.c | 103 +++++---------------- fs/nfsd/nfssvc.c | 10 +- fs/nfsd/stats.h | 5 - fs/nfsd/trace.h | 24 ----- 7 files changed, 28 insertions(+), 136 deletions(-)
Documentation/ABI/testing/procfs-nfsd-reply_cache_stats +4 -7
@@ -19,19 +19,16 @@ Description: cache misses s64 Requests not found in cache not cached s64 Idempotent requests that bypass the cache - payload misses s64 XID matched but request - checksum did not longest chain len u32 Longest hash chain observed cachesize at longest u32 Cache size when longest chain was recorded ======================= ====== ========================== Counter fields (cache hits, cache misses, not cached, - payload misses, mem usage) are maintained with per-cpu - counters and may briefly show stale values under - concurrent load. There is no way to reset these - counters; consumers should compute rates by sampling - over time. + mem usage) are maintained with per-cpu counters and + may briefly show stale values under concurrent load. + There is no way to reset these counters; consumers + should compute rates by sampling over time. New fields may be appended in future kernels. Parsers should match on field name, not line position.
fs/nfsd/cache.h +2 -7
@@ -22,9 +22,7 @@ struct nfsd_net;*/ struct nfsd_cacherep {struct { -/* Keep often-read xid, csum in the same cache line: */ __be32 k_xid; -__wsum k_csum;u32 k_proc;u32 k_prot;u32 k_vers;
@@ -80,16 +78,13 @@ enum {/* Cache entries expire after this time period */ #define RC_EXPIRE (120 * HZ) -/* Checksum this amount of the request */ -#define RC_CSUMLEN (256U) -svc_ack_cookie_t nfsd_cache_ack_cookie (const struct nfsd_cacherep *rp);int nfsd_drc_slab_create (void );void nfsd_drc_slab_free (void );int nfsd_reply_cache_init (struct nfsd_net *,struct svc_serv *);void nfsd_reply_cache_shutdown (struct nfsd_net *,struct svc_serv *); -int nfsd_cache_lookup (struct svc_rqst *rqstp,unsigned int start, -unsigned int len,struct nfsd_cacherep **cacherep); +int nfsd_cache_lookup (struct svc_rqst *rqstp, +struct nfsd_cacherep **cacherep);void nfsd_cache_update (struct svc_rqst *rqstp,struct nfsd_cacherep *rp,int cachetype,__be32 *statp);int nfsd_reply_cache_stats_show (struct seq_file *m,void *v);
fs/nfsd/netns.h +0 -2
@@ -39,8 +39,6 @@ enum nfsd_net_flag { };enum { -/* cache misses due only to checksum comparison failures */ -NFSD_STATS_PAYLOAD_MISSES ,/* amount of memory (in bytes) currently consumed by the DRC */ NFSD_STATS_DRC_MEM_USAGE ,NFSD_STATS_RC_HITS ,/* repcache hits */
fs/nfsd/nfscache.c +21 -82
178 lines not on this page · raw message
fs/nfsd/nfssvc.c +1 -9
@@ -1004,7 +1004,6 @@ int nfsd_dispatch(struct svc_rqst *rqstp)const struct svc_procedure *proc = rqstp->rq_procinfo;__be32 *statp = rqstp->rq_accept_statp;struct nfsd_cacherep *rp; -unsigned int start, len;__be32 *nfs_reply;/*
@@ -1013,13 +1012,6 @@ int nfsd_dispatch(struct svc_rqst *rqstp)*/ ntli->ntli_cachetype = proc->pc_cachetype; -/* -* ->pc_decode advances the argument stream past the NFS -* Call header, so grab the header's starting location and -* size now for the call to nfsd_cache_lookup(). -*/ - start =xdr_stream_pos (&rqstp->rq_arg_stream); - len =xdr_stream_remaining (&rqstp->rq_arg_stream);if (!proc->pc_decode (rqstp, &rqstp->rq_arg_stream)) goto out_decode_err;
@@ -1033,7 +1025,7 @@ int nfsd_dispatch(struct svc_rqst *rqstp)smp_store_release (&rqstp->rq_status_counter, rqstp->rq_status_counter |1 ); rp =NULL ; -switch (nfsd_cache_lookup (rqstp, start, len, &rp)) { +switch (nfsd_cache_lookup (rqstp, &rp)) {case RC_DOIT :break ;case RC_REPLY :
fs/nfsd/stats.h +0 -5
@@ -97,11 +97,6 @@ static inline void nfsd_stats_io_write_add(struct nfsd_net *nn, amount); } -static inline void nfsd_stats_payload_misses_inc (struct nfsd_net *nn) -{ -percpu_counter_inc (&nn->counter[NFSD_STATS_PAYLOAD_MISSES ]); -} -/** * nfsd_stats_drc_mem_usage_add - Add memory used by a cache item * @nn: target network namespace
fs/nfsd/trace.h +0 -24
@@ -1536,30 +1536,6 @@ TRACE_EVENT(nfsd_drc_found, ); -TRACE_EVENT (nfsd_drc_mismatch, -TP_PROTO ( - const struct nfsd_net *nn, - const struct nfsd_cacherep *key, - const struct nfsd_cacherep *rp - ), -TP_ARGS (nn, key, rp), -TP_STRUCT__entry ( - __field(unsigned long long, boot_time) - __field(u32, xid) - __field(u32, cached) -__field (u32, ingress) - ), -TP_fast_assign ( - __entry->boot_time = nn->boot_time; - __entry->xid = be32_to_cpu(key->c_key.k_xid); - __entry->cached = (__force u32)key->c_key.k_csum; - __entry->ingress = (__force u32)rp->c_key.k_csum; - ), -TP_printk ("boot_time=%16llx xid=0x%08x cached-csum=0x%08x ingress-csum=0x%08x" , - __entry->boot_time, __entry->xid, __entry->cached, - __entry->ingress) -); - DECLARE_EVENT_CLASS(nfsd_drc_entry_class, TP_PROTO( const struct nfsd_net *nn,
Signature · 1 line
2.55.0
Chuck Lever
maintainerPATCH v3 12/12 NFSD: Remove hard cap on duplicate reply cache size
In reply to Chuck LeverThe DRC matters most during a network partition, when clients cannot receive replies and the server must hold them for retransmission after reconnect. Explicit and implied ACK are inactive then, leaving only RC_EXPIRE, and a 256k entry ceiling can be too small for a large client cohort through a long partition. During normal operation the ACK paths keep the cache small regardless of the maximum, so the cap constrains only the failure case, where headroom matters most.
Remove the cap and let the square-root formula govern sizing. It scales sub-linearly with memory, and the hash table already uses kvzalloc, so larger sizes need no physical contiguity. The limit is unchanged at 16 GB and below. At 1 TB the formula yields 1048576 entries, four times the old cap. Even at 1 KB per entry, the worst case the old comment assumed, a full cache is 1 GB, 0.1% of that host's memory.
--- fs/nfsd/nfscache.c | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-)
fs/nfsd/nfscache.c +16 -15
@@ -48,34 +48,35 @@ static void nfsd_reply_ack(void *data, const svc_ack_cookie_t *cookie, bool delivered);/* -* Put a cap on the size of the DRC based on the amount of available -* low memory in the machine. +* Set the size of the DRC based on the amount of available low +* memory in the machine. The sizing formula scales with the +* square root of available pages, so growth is sub-linear: * * 64MB: 8192 -* 128MB: 11585 +* 128MB: 11584 * 256MB: 16384 -* 512MB: 23170 +* 512MB: 23168 * 1GB: 32768 -* 2GB: 46340 +* 2GB: 46336 * 4GB: 65536 -* 8GB: 92681 +* 8GB: 92672 * 16GB: 131072 +* 32GB: 185344 +* 64GB: 262144 +* 128GB: 370688 +* 256GB: 524288 +* 512GB: 741440 +* 1TB: 1048576 * -* ...with a hard cap of 256k entries. In the worst case, each entry will be -* ~1k, so the above numbers should give a rough max of the amount of memory -* used in k. -* -* XXX: these limits are per-container, so memory used will increase -* linearly with number of containers. Maybe that's OK. +* These limits are per-net-namespace, but the per-namespace +* shrinker reclaims entries under memory pressure. */ static unsigned int nfsd_cache_size_limit (void ) { -unsigned int limit;unsigned long low_pages =totalram_pages () -totalhigh_pages (); - limit = (16 *int_sqrt (low_pages)) << (PAGE_SHIFT -10 ); - return min_t(unsigned int, limit, 256*1024); +return (16 *int_sqrt (low_pages)) << (PAGE_SHIFT -10 ); }/*
Signature · 1 line
2.55.0
Quoted 96 lines · Chuck Lever
A completed DRC entry currently stays in its bucket for 120 seconds whether or not the client already holds the reply. On a busy server those entries lengthen every bucket walk, and under pressure they crowd out entries a retransmit could still hit.
This series modifies the DRC to retire an entry once there is reason to believe its reply was delivered.
1. The transport reports delivery outright where it can. TCP now reports when
snd_unapasses the reply's sequence number, and RDMA reports on each Send completion.
2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).UDP continues to use a traditional time-based eviction mechanism.
The first mechanism is the more reliable of the two, and the backup mechanism is more of an informed guess. A pipelined client sends its next request before the previous reply lands, and an entry evicted in that window is lost if the connection then drops and the client retransmits. In the current NFSD code, memory pressure and
RC_EXPIREalready open this same window. For instance, on a server with fast networking and storage, pressure eviction already evicts entries far younger than 120 seconds.Two DRC capacity guards that were sized for a cache full of stale replies can be removed, now that reply delivery reports keep the DRC small. The commit messages for those patches explain the rationale in detail.
v2 of this series (implied ACK only) was profiled with "perf record -e cycles -e cpu-clock -e LLC-load-misses -e branch-misses" during an NFSv3/RDMA 4KB random-write workload. v3 has not been re-profiled.
nfsd_cache_lookupoverhead dropped from 1.53% to 0.76% of CPU cycles during this test. The rb-tree operations (rb_erase,rb_insert_color) that dominated LLC cache misses fell from a combined 13.2% to 1.1% of all LLC-load-misses, because shorter-lived entries keep the per-bucket trees small.---
Changes in v3:
- Add the reply-acknowledged callback and its TCP and RDMA reporters
ahead of implied ACK, which now defers to a pending report.
- Split the prune-loop restructure into its own patch.
- Move the eviction tracepoints ahead of the ack patches; the
implied-ACK event now lands with implied ACK.
- Keep err local to thepmap_registerblock insvc_setup_socket()
(per Jeff's review).
- Never movexpt_last_recvbackwards when nfsd threads race.
- Fold theXPT_ORDEREDpatch into its consumer (per Jeff's review).
- State the re-execution exposure of implied-ACK eviction instead of
calling the DRC advisory (per Jeff's review).
- New patch: remove the DRC request checksum andpayload_missesstat.
- New patch: remove the 256k-entry cap on the DRC size.
- Link to v2: https://patch.msgid.link/[email protected]Changes in v2:
- Print the DRC eviction tracepoints' age field as unsigned.
- Link to v1: https://patch.msgid.link/[email protected]---
Chuck Lever (12):
SUNRPC: Assign a unique identifier to eachsvc_xprt
NFSD: Track transport in DRC entries
NFSD: Prepare bucket pruning for out-of-order eviction
NFSD: Add tracepoints for DRC entry eviction
NFSD: Record DRC population in lookup tracepoints
NFSD: Add reply-acknowledged callback infrastructure
SUNRPC: Add TCP sequence-number ACK tracking for reply delivery
svcrdma: Fire reply-acknowledged callback on Send completion
SUNRPC: Record last-request timestamp onsvc_xprt
NFSD: Evict unacknowledged DRC entries via implied ACK
NFSD: Remove DRC checksum andpayload_missesstat
NFSD: Remove hard cap on duplicate reply cache size.../ABI/testing/procfs-nfsd-reply_cache_stats | 11 +- fs/nfsd/cache.h | 20 +- fs/nfsd/netns.h | 2 - fs/nfsd/nfscache.c | 310 +++++++++++++-------- fs/nfsd/nfssvc.c | 20 +- fs/nfsd/stats.h | 5 - fs/nfsd/trace.h | 81 +++++- include/linux/sunrpc/svc.h | 51 ++++ include/linux/sunrpc/svc_rdma.h | 1 + include/linux/sunrpc/svc_xprt.h | 6 +- include/linux/sunrpc/svcsock.h | 10 + include/trace/events/sunrpc.h | 10 +- net/sunrpc/netns.h | 4 + net/sunrpc/sunrpc_syms.c | 2 + net/sunrpc/svc.c | 1 + net/sunrpc/svc_xprt.c | 53 +++- net/sunrpc/svcsock.c | 119 +++++++- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 14 + net/sunrpc/xprtrdma/svc_rdma_transport.c | 7 +- 19 files changed, 541 insertions(+), 186 deletions(-)
Quoted 7 lines
---
base-commit:abf2077ee32058e60d3f49ecab265d3c4bd953d9
change-id: 20260325-duplicate-reply-cache-f0fe7c7b740cBest regards,
Signature · 1 line
Chuck Lever <[email protected]>
I like the idea overall. I'm still a bit nervous that might see a non- idempotent operations fail occasionally, since this is aggressively expiring them, but let's see how it does.
Quoted 11 lines · Chuck Lever
A completed DRC entry currently stays in its bucket for 120 seconds whether or not the client already holds the reply. On a busy server those entries lengthen every bucket walk, and under pressure they crowd out entries a retransmit could still hit.
This series modifies the DRC to retire an entry once there is reason to believe its reply was delivered.
1. The transport reports delivery outright where it can. TCP now reports when
snd_unapasses the reply's sequence number, and RDMA reports on each Send completion.
As you note in the relevant patch, the TCP ack can arrive before the NFS client has seen the reply. But I don't think you drop the cached reply immediately, so that probably don't matter.
The risk is that the second non-idempotent request might fail (mkdir, but dir already exists), not that it will produce a different positive result (technically possible, but vanishingly so). Once we see the TCP ACK we can be confident the client *will* see that reply (network fault won't cause it not to) so it will mostly likely act on the first reply and ignore the second.
Quoted 3 lines
2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).
I don't follow this. Why would no report come? TCP will always send and ACK.
And I don't read RFC 1813 4.5 as saying a client does NOT retransmit on a live connection, only that it DOES on a re-opened connection. I believe that when an NFS client gets a timeout (6 minutes on TCP?) it will resend without re-opening the TCP connection.
And as you say it is possible (I would say "likely") that a client will have multiple requests outstanding. Even multiple non-idempotent requests. So I cannot see that a later request tells us anything about receiving an earlier reply.
A non-idempotent request often returns a new filehandle, and using the presentation of that filehandle as evidence that the earlier reply was seen would be sound, but I cannot see much other opportunity to draw a connection from later requests to earlier replies.
So I'm perplexed as to why point 2 is needed, or how it is justified.
Thanks,
NeilBrown
Quoted 91 lines
UDP continues to use a traditional time-based eviction mechanism.
The first mechanism is the more reliable of the two, and the backup mechanism is more of an informed guess. A pipelined client sends its next request before the previous reply lands, and an entry evicted in that window is lost if the connection then drops and the client retransmits. In the current NFSD code, memory pressure and
RC_EXPIREalready open this same window. For instance, on a server with fast networking and storage, pressure eviction already evicts entries far younger than 120 seconds.Two DRC capacity guards that were sized for a cache full of stale replies can be removed, now that reply delivery reports keep the DRC small. The commit messages for those patches explain the rationale in detail.
v2 of this series (implied ACK only) was profiled with "perf record -e cycles -e cpu-clock -e LLC-load-misses -e branch-misses" during an NFSv3/RDMA 4KB random-write workload. v3 has not been re-profiled.
nfsd_cache_lookupoverhead dropped from 1.53% to 0.76% of CPU cycles during this test. The rb-tree operations (rb_erase,rb_insert_color) that dominated LLC cache misses fell from a combined 13.2% to 1.1% of all LLC-load-misses, because shorter-lived entries keep the per-bucket trees small.---
Changes in v3:
- Add the reply-acknowledged callback and its TCP and RDMA reporters
ahead of implied ACK, which now defers to a pending report.
- Split the prune-loop restructure into its own patch.
- Move the eviction tracepoints ahead of the ack patches; the
implied-ACK event now lands with implied ACK.
- Keep err local to thepmap_registerblock insvc_setup_socket()
(per Jeff's review).
- Never movexpt_last_recvbackwards when nfsd threads race.
- Fold theXPT_ORDEREDpatch into its consumer (per Jeff's review).
- State the re-execution exposure of implied-ACK eviction instead of
calling the DRC advisory (per Jeff's review).
- New patch: remove the DRC request checksum andpayload_missesstat.
- New patch: remove the 256k-entry cap on the DRC size.
- Link to v2: https://patch.msgid.link/[email protected]Changes in v2:
- Print the DRC eviction tracepoints' age field as unsigned.
- Link to v1: https://patch.msgid.link/[email protected]---
Chuck Lever (12):
SUNRPC: Assign a unique identifier to eachsvc_xprt
NFSD: Track transport in DRC entries
NFSD: Prepare bucket pruning for out-of-order eviction
NFSD: Add tracepoints for DRC entry eviction
NFSD: Record DRC population in lookup tracepoints
NFSD: Add reply-acknowledged callback infrastructure
SUNRPC: Add TCP sequence-number ACK tracking for reply delivery
svcrdma: Fire reply-acknowledged callback on Send completion
SUNRPC: Record last-request timestamp onsvc_xprt
NFSD: Evict unacknowledged DRC entries via implied ACK
NFSD: Remove DRC checksum andpayload_missesstat
NFSD: Remove hard cap on duplicate reply cache size.../ABI/testing/procfs-nfsd-reply_cache_stats | 11 +- fs/nfsd/cache.h | 20 +- fs/nfsd/netns.h | 2 - fs/nfsd/nfscache.c | 310 +++++++++++++-------- fs/nfsd/nfssvc.c | 20 +- fs/nfsd/stats.h | 5 - fs/nfsd/trace.h | 81 +++++- include/linux/sunrpc/svc.h | 51 ++++ include/linux/sunrpc/svc_rdma.h | 1 + include/linux/sunrpc/svc_xprt.h | 6 +- include/linux/sunrpc/svcsock.h | 10 + include/trace/events/sunrpc.h | 10 +- net/sunrpc/netns.h | 4 + net/sunrpc/sunrpc_syms.c | 2 + net/sunrpc/svc.c | 1 + net/sunrpc/svc_xprt.c | 53 +++- net/sunrpc/svcsock.c | 119 +++++++- net/sunrpc/xprtrdma/svc_rdma_sendto.c | 14 + net/sunrpc/xprtrdma/svc_rdma_transport.c | 7 +- 19 files changed, 541 insertions(+), 186 deletions(-)---
base-commit:abf2077ee32058e60d3f49ecab265d3c4bd953d9
change-id: 20260325-duplicate-reply-cache-f0fe7c7b740cBest regards,
Signature · 1 line
Chuck Lever <[email protected]>
Chuck Lever
maintainerQuoted 8 lines · NeilBrown
Quoted 3 lines · Chuck Lever
1. The transport reports delivery outright where it can. TCP now reports when
snd_unapasses the reply's sequence number, and RDMA reports on each Send completion.As you note in the relevant patch, the TCP ack can arrive before the NFS client has seen the reply. But I don't think you drop the cached reply immediately, so that probably don't matter.
Right. The ACK marks the entry, and the next prune of that bucket evicts it.
Quoted 6 lines
Quoted 3 lines
2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).
I don't follow this. Why would no report come? TCP will always send and ACK.
The cover letter was unclear. TCP always ACKs, but svc_tcp_sendto() does not always track the reply. The per-socket ring is fixed-size, so a burst overflows it. A kTLS send that leaves part of the record unsent is not recorded, and neither is a failed send. Each of those is reported as untracked right away, so the DRC entry has no report to wait for.
Implied ACK was meant to retire those entries early. Without it, an untracked entry is handled the way every entry is handled today. The untracked entries just sit in their buckets for the full two minutes, lengthening each lookup that hashes there, which is the cost this series set out to remove.
Quoted 4 lines
And I don't read RFC 1813 4.5 as saying a client does NOT retransmit on a live connection, only that it DOES on a re-opened connection. I believe that when an NFS client gets a timeout (6 minutes on TCP?) it will resend without re-opening the TCP connection.
Section 4.5 says that a connection break with automatic reestablishment requires duplicate request processing. I misread that as naming the only way a duplicate reaches a server over a connection-oriented transport, and then reasoned that a request arriving on a live connection could not be one. Re-reading, I see that text supports no such exclusion. It only gives one case that a DRC must handle, and says nothing about what a client does while the connection remains healthy.
Quoted 4 lines
And as you say it is possible (I would say "likely") that a client will have multiple requests outstanding. Even multiple non-idempotent requests. So I cannot see that a later request tells us anything about receiving an earlier reply.
Agreed.
Quoted 1 line
So I'm perplexed as to why point 2 is needed, or how it is justified.
With the explicit reports in place I expect implied ACK to fire
rarely: svcrdma reports every reply, and on TCP only a ring overflow leaves a reply untracked. Shall I drop implied ACK from v4 of the series, measure the untracked rate under load, and size the ring from that instead?
Signature · 1 line
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)
Quoted 21 lines · Chuck Lever
Quoted 8 lines · NeilBrown
Quoted 3 lines · Chuck Lever
1. The transport reports delivery outright where it can. TCP now reports when
snd_unapasses the reply's sequence number, and RDMA reports on each Send completion.As you note in the relevant patch, the TCP ack can arrive before the NFS client has seen the reply. But I don't think you drop the cached reply immediately, so that probably don't matter.
Right. The ACK marks the entry, and the next prune of that bucket evicts it.
Quoted 6 lines
Quoted 3 lines
2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).
I don't follow this. Why would no report come? TCP will always send and ACK.
Quoted 6 lines
The cover letter was unclear. TCP always ACKs, but
svc_tcp_sendto()does not always track the reply. The per-socket ring is fixed-size, so a burst overflows it. A kTLS send that leaves part of the record unsent is not recorded, and neither is a failed send. Each of those is reported as untracked right away, so the DRC entry has no report to wait for.
I don't see the need for the per-socket ring.
When a reply is sent - record the TCP sequence number in the DRC. When pruning the DRC, get the TCP ack number first and compare it against sequence numbers.
Do this often enough that that entries are pruned before the sequence wraps past them.
kTLS is clearly more subtle but there must be some way record an approximate seq number for an incomplete send .. maybe current seq + header-size + queued size ??
Unsent replies will never get an ack, but presumably the client will resend, be answered using the cached reply, and we can then get an ack on the new connection. I think it would be wrong to drop an unsent reply before the timeout, unless it gets re-sent and transport-acked before then.
I think the opaque cookie aspect of the design is a mistake. Everything is fully sequenced over a network connection so a sequence number (clearly paired with xpt_id) makes more sense...
Or does RDMA not sequence replies? If so then it can only ack a single reply at a time, not a range of replies. But that needn't be a barrier.
When a message is transmitted the transport hands a seq-number (and xpt_id) to the cache. When it gets an ack it hands a range of seq-numbers to the cache. For a transport that had out-of-order acks, this range would always be of length one (that transport might need a private atomic_t to generate the sequence numbers). For a transport that batches acks (like TCP) the range could be much longer.
Also, I wonder if the xpt_id should be 64bit assigned sequentially with no xa keeping track of used one. 64bits never wraps.
I think that with this design the implied ACK would add no value.
Thanks,
NeilBrown
Chuck Lever
maintainerQuoted 59 lines · NeilBrown
Quoted 21 lines · Chuck Lever
Quoted 8 lines · NeilBrown
Quoted 3 lines · Chuck Lever
1. The transport reports delivery outright where it can. TCP now reports when
snd_unapasses the reply's sequence number, and RDMA reports on each Send completion.As you note in the relevant patch, the TCP ack can arrive before the NFS client has seen the reply. But I don't think you drop the cached reply immediately, so that probably don't matter.
Right. The ACK marks the entry, and the next prune of that bucket evicts it.
Quoted 6 lines
Quoted 3 lines
2. Where no report will come, a later request on the same connection stands in as an implied ACK, since a client on a live TCP or RDMA connection does not retransmit within it (RFC 1813 Section 4.5).
I don't follow this. Why would no report come? TCP will always send and ACK.
Quoted 6 lines
The cover letter was unclear. TCP always ACKs, but
svc_tcp_sendto()does not always track the reply. The per-socket ring is fixed-size, so a burst overflows it. A kTLS send that leaves part of the record unsent is not recorded, and neither is a failed send. Each of those is reported as untracked right away, so the DRC entry has no report to wait for.I don't see the need for the per-socket ring.
When a reply is sent - record the TCP sequence number in the DRC. When pruning the DRC, get the TCP ack number first and compare it against sequence numbers.
Do this often enough that that entries are pruned before the sequence wraps past them.kTLS is clearly more subtle but there must be some way record an approximate seq number for an incomplete send .. maybe current seq + header-size + queued size ??
Unsent replies will never get an ack, but presumably the client will resend, be answered using the cached reply, and we can then get an ack on the new connection. I think it would be wrong to drop an unsent reply before the timeout, unless it gets re-sent and transport-acked before then.
I think the opaque cookie aspect of the design is a mistake. Everything is fully sequenced over a network connection so a sequence number (clearly paired with
xpt_id) makes more sense...
Or does RDMA not sequence replies? If so then it can only ack a single reply at a time, not a range of replies. But that needn't be a barrier.When a message is transmitted the transport hands a seq-number (and
xpt_id) to the cache. When it gets an ack it hands a range of seq-numbers to the cache. For a transport that had out-of-order acks, this range would always be of length one (that transport might need a privateatomic_tto generate the sequence numbers). For a transport that batches acks (like TCP) the range could be much longer.
Quoted 4 lines
Also, I wonder if the
xpt_idshould be 64bit assigned sequentially with no xa keeping track of used one. 64bits never wraps.I think that with this design the implied ACK would add no value.
Thanks, all makes sense to me and reduces code complexity.
Signature · 1 line
Chuck Lever (Come to NFS bake-a-thon! https://nfsv4bat.org)