SUNRPC: a latency floor for interactive clients via sparse-flow dispatch
Subsystems (3)
cover SUNRPC: a latency floor for interactive clients via sparse-flow dispatch
This RFC follows the per-client fair-scheduling discussion from May/June [1], and Chuck's slot-growth clamp that grew out of it [2]. It takes the direction that thread settled on: rather than enforce proportional fairness between clients, give an interactive client a latency floor that a busy neighbour cannot push it below.
Problem
-------
nfsd dispatches ready transports per-pool in FIFO order, with no notion of how much work each transport already has outstanding. A client that keeps many requests in flight -- nconnect, a deep v4.1 slot table, or simply several connections -- sits in that queue on equal terms with a client that has been idle. So the request a user is waiting on (a stat, an open, a directory listing) waits behind a backup job's backlog, even though servicing it costs a single round trip. The user-visible symptom is an interactive session that stutters whenever a bulk client is busy.
Approach
--------
This is starvation avoidance, not proportional fairness (per Chuck's reframe on [1]). The protected unit is the interactive cycle: one user command fans out into a burst of correlated RPCs, so it is the whole burst, not a single RPC, that must be allowed to jump ahead.
The signal is already on every transport: xpt_nr_rqsts, the count of requests in flight. A transport with nothing in flight when new work arrives is, by definition, not the one hogging threads -- so it goes on a second, high-priority per-pool queue that is drained ahead of the normal one. To cover the whole burst rather than only its leading edge, an idle->active transport is granted a budget of 64 high-priority dispatches, spent down as the burst is serviced and refilled only when the transport next idles. A transport that never idles spends its budget once and then shares the normal queue like everyone else.
There is no client identity anywhere in this -- it keys only on a transport's own in-flight depth -- so it covers v3, v4.0 and v4.1 uniformly, needs no lookup or lifecycle, adds no lock (the existing lwq barriers carry the sleep/wake), and is always on with nothing to tune. (LOCALIO is out of scope: a local client's reads and writes bypass the RPC dispatch path entirely, so they neither contend here nor are reordered by this change.)
patch 1 add the second per-pool ready queue (no functional change) patch 2 dispatch idle transports ahead of backlogged ones patch 3 grant the 64-RPC burst budget
Relationship to Chuck's slot-growth RFC
---------------------------------------
This series is based on Chuck's "Stop NFSv4.1 slot-growth heuristic from rewarding busy clients" [2], and both A and B above include it -- so the A/B delta isolates this series and the clamp does not account for the improvement.
The two are complementary, not overlapping. [2] stops nfsd from growing a busy session's slot table past the thread ceiling: nfsd slot accounting. This series changes the order in which ready transports are dispatched:
sunrpc dispatch. Basing on [2] changes neither the design nor the result here -- a slot-capped session still fills the pool, so the dispatch-layer floor is still wanted on top of it.
Results
-------
Because the goal is a latency floor and not a share, the measurement differs from the earlier bucket RFC and the numbers are not comparable:
that series measured a greedy client's share of throughput; this one measures how long an interactive client's command takes to complete while a neighbour is busy.
Workload: a backlogged aggressor (K connections, deep offered load, saturating the pool) shares the server with one interactive client that issues a burst of N requests, waits for all replies, idles 50ms, and repeats. A 10ms service time is injected per op (a test-only debug hook, not part of this series) so the measurement isolates the dispatch path from filesystem work. A is stock nfsd-testing; B is the same plus this series. Figures are NFSv3 burst-completion p50 in ms; NFSv4.1 tracks within a few percent (the classifier uses no source identity, so v3 runs on plain loopback and behaves the same).
Interactive burst completion vs aggressor connection count K (N=32; unobstructed floor 45ms):
K 4 8 16 32
A 81.6 242.0 432.9 826.6
B 60.9 64.5 65.4 64.4
1.3x 3.8x 6.6x 12.8xBaseline climbs ~linearly with the aggressor's backlog; patched holds the floor no matter how deep that backlog is -- the property that matters.
vs burst size N (K=16), showing the 64-credit knee (floor for reference):
N 1 8 32 64 96 128
floor 14.9 31.9 44.8 71.5 96.8 122.4
A 28.6 122.0 434.2 862.4 1272.6 1696.3
B 16.7 36.0 65.1 124.5 546.4 961.6For N <= 64 the whole burst is covered and tracks the floor (B within 1.7x); beyond 64 the uncovered tail -- the N-64 RPCs that spill to the normal queue -- degrades toward baseline (B/floor jumps 1.7x -> 5.6x across the 64->96 boundary). The knee at exactly 64 is the budget working as designed, not a regression.
Aggregate throughput (aggressor + interactive) is within 1% A vs B at saturation (~1290 ops/s) -- the dispatcher reorders, it does not throttle. With no aggressor the interactive floor is identical A vs B; the kernels differ only under contention.
Figures are p50; longer runs are available for firmer tails.
What this deliberately does not do
----------------------------------
- It is not proportional fairness. Two backlogged clients still split the pool by connection count; this protects only flows that idle.
- The normal queue can wait under sustained high-priority load. v1 drains high-priority strictly first; bounding that starvation -- an inter-tier deficit round-robin, the way fq_codel actually interleaves its sparse and bulk tiers -- is left for a follow-up, pending the question below.
- The 64-dispatch budget defeats a pipelined aggressor (it lands wholly in the normal queue), but a client that shapes its traffic to look sparse -- send <=64, drain, repeat -- can still scale its high-priority share with connection count. Whether to harden against that is the open
question: if this is cooperative scheduling (Neil's framing) a bulk floor suffices; if it is an attack surface (Chuck's framing) the high-priority tier needs intra-tier fairness, which reintroduces identity.
Open questions for the list: is the budget (64 here) the right shape, and should it be fixed, derived from the thread count, or hinted by the v4.1 slot table? And does the cooperative framing hold, or must we resist deliberate sparse-shaping?
[1] https://lore.kernel.org/linux-nfs/[email protected]/
[2] https://lore.kernel.org/linux-nfs/[email protected]/
Benjamin Coddington (3):
SUNRPC: add a second per-pool ready queue for high-priority transports
SUNRPC: dispatch idle transports ahead of backlogged ones
SUNRPC: grant an idle flow a burst allowance on the high-priority
queue
include/linux/sunrpc/svc.h | 1 + include/linux/sunrpc/svc_xprt.h | 1 + net/sunrpc/svc.c | 1 + net/sunrpc/svc_xprt.c | 52 +++++++++++++++++++++++---------- 4 files changed, 39 insertions(+), 16 deletions(-)
Signature · 1 line
2.53.0
RFC PATCH 1/3 SUNRPC: add a second per-pool ready queue for high-priority transports
A pool's ready transports are tracked in a single lockless queue, sp_xprts, and dispatched in FIFO order. A transport's position in that queue is independent of how much work it already has in flight, so a connection that keeps many requests pending is serviced on equal terms with one that has been idle -- the idle flow's next request waits behind the busy flow's backlog.
Add a second queue, sp_xprts_hi, alongside sp_xprts and initialise it with each pool. Nothing is enqueued to it yet, so this is no functional change; a later patch routes transports with no requests in flight onto it and drains it ahead of sp_xprts, giving an idle flow's request a latency floor that a backlogged flow cannot push it below.
--- include/linux/sunrpc/svc.h | 1 + net/sunrpc/svc.c | 1 + 2 files changed, 2 insertions(+)
include/linux/sunrpc/svc.h +1 -0
@@ -39,6 +39,7 @@ struct svc_pool {unsigned int sp_nrthrmin;/* Min number of threads to run per pool */ unsigned int sp_nrthrmax;/* Max requested number of threads in pool */ struct lwq sp_xprts;/* pending transports */ +struct lwq sp_xprts_hi;/* high-priority pending transports */ struct list_head sp_all_threads;/* all server threads */ struct llist_head sp_idle_threads;/* idle server threads */
net/sunrpc/svc.c +1 -0
@@ -566,6 +566,7 @@ __svc_create(struct svc_program *prog, int nprogs, struct svc_stat *stats, pool->sp_id = i;lwq_init (&pool->sp_xprts); +lwq_init (&pool->sp_xprts_hi);INIT_LIST_HEAD (&pool->sp_all_threads);init_llist_head (&pool->sp_idle_threads);
Signature · 1 line
2.53.0
RFC PATCH 2/3 SUNRPC: dispatch idle transports ahead of backlogged ones
In reply to Benjamin CoddingtonA pool dispatches ready transports in FIFO order, so a connection that already has requests in flight sits in the same queue, on equal terms, as one that has been idle. A client driving many concurrent requests therefore delays the next request of an interactive client whose previous reply has already completed: the interactive request waits behind the busy client's backlog even though servicing it costs a single round trip.
Route a transport with no requests in flight onto the new high-priority queue, sp_xprts_hi, and drain that queue ahead of sp_xprts. xpt_nr_rqsts is incremented when a thread reserves a request slot and decremented when the reply completes, so at enqueue time it counts the transport's prior in-flight work, not the request that just arrived: a flow whose last reply has completed reads zero and jumps the queue, while a flow with requests still in flight stays in the bulk queue. The classification needs no client identity and no lock -- each queue is an independent lwq with its own ordering, so svc_thread_should_sleep() simply tests both.
This gives an idle flow's request a latency floor that a backlogged flow cannot push it below. It is starvation avoidance, not proportional
fairness: under sustained load across many idle flows the bulk queue can wait, and bounding that is left to later work.
--- net/sunrpc/svc_xprt.c | 44 +++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-)
net/sunrpc/svc_xprt.c +28 -16
@@ -523,7 +523,10 @@ void svc_xprt_enqueue(struct svc_xprt *xprt)percpu_counter_inc (&pool->sp_sockets_queued); xprt->xpt_qtime =ktime_get (); -lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts); +if (atomic_read (&xprt->xpt_nr_rqsts)) +lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts); +else +lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts_hi);svc_pool_wake_idle_thread (pool); }
@@ -536,7 +539,9 @@ static struct svc_xprt *svc_xprt_dequeue(struct svc_pool *pool) {struct svc_xprt *xprt =NULL ; - xprt = lwq_dequeue(&pool->sp_xprts, struct svc_xprt, xpt_ready); + xprt = lwq_dequeue(&pool->sp_xprts_hi, struct svc_xprt, xpt_ready); +if (!xprt) + xprt = lwq_dequeue(&pool->sp_xprts, struct svc_xprt, xpt_ready);if (xprt)svc_xprt_get (xprt);return xprt;
@@ -759,7 +764,7 @@ svc_thread_should_sleep(struct svc_rqst *rqstp)return false;/* was a socket queued? */ -if (!lwq_empty (&pool->sp_xprts)) +if (!lwq_empty (&pool->sp_xprts_hi) || !lwq_empty (&pool->sp_xprts))return false;/* are we shutting down? */
@@ -1183,26 +1188,33 @@ static int svc_close_list(struct svc_serv *serv, struct list_head *xprt_list, streturn ret; } -static void svc_clean_up_xprts (struct svc_serv *serv,struct net *net) +static void svc_clean_up_queue (struct lwq *queue,struct net *net) {struct svc_xprt *xprt; +struct llist_node *q, **t1, *t2; + + q =lwq_dequeue_all (queue); +lwq_for_each_safe (xprt, t1, t2, &q, xpt_ready) { +if (xprt->xpt_net == net) { +set_bit (XPT_CLOSE , &xprt->xpt_flags); +svc_delete_xprt (xprt); + xprt =NULL ; + } + } + +if (q) +lwq_enqueue_batch (q, queue); +} + +static void svc_clean_up_xprts (struct svc_serv *serv,struct net *net) +{int i;for (i =0 ; i < serv->sv_nrpools; i++) {struct svc_pool *pool = &serv->sv_pools[i]; -struct llist_node *q, **t1, *t2; - - q =lwq_dequeue_all (&pool->sp_xprts); -lwq_for_each_safe (xprt, t1, t2, &q, xpt_ready) { -if (xprt->xpt_net == net) { -set_bit (XPT_CLOSE , &xprt->xpt_flags); -svc_delete_xprt (xprt); - xprt =NULL ; - } - } -if (q) -lwq_enqueue_batch (q, &pool->sp_xprts); +svc_clean_up_queue (&pool->sp_xprts_hi, net); +svc_clean_up_queue (&pool->sp_xprts, net); } }
Signature · 1 line
2.53.0
Chuck Lever
maintainerRe: [PATCH RFC 2/3] SUNRPC: dispatch idle transports ahead of backlogged ones
Quoted 14 lines · Benjamin Coddington
A pool dispatches ready transports in FIFO order, so a connection that already has requests in flight sits in the same queue, on equal terms, as one that has been idle. A client driving many concurrent requests therefore delays the next request of an interactive client whose previous reply has already completed: the interactive request waits behind the busy client's backlog even though servicing it costs a single round trip.
net/sunrpc/svc_xprt.c +0
-0@@ -523,7 +523,10 @@ void svc_xprt_enqueue(struct svc_xprt *xprt)
net/sunrpc/svc_xprt.c void svc_xprt_enqueue(struct svc_xprt *xprt) · line 532
Quoted 10 lines
net/sunrpc/svc_xprt.c +4
-1percpu_counter_inc (&pool->sp_sockets_queued); xprt->xpt_qtime =ktime_get (); -lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts); +if (atomic_read (&xprt->xpt_nr_rqsts)) +lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts); +else +lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts_hi);svc_pool_wake_idle_thread (pool); }
The classification is keyed on in-flight request count, but a deferred close is enqueued the same way. svc_xprt_deferred_close() sets XPT_CLOSE and calls svc_xprt_enqueue() while the slot is still held; svc_rdma_sendto() on a send error is one such path, and svc_check_conn_limits() is another. svc_handle_xprt() reserves the slot before calling recvfrom and releases it only after svc_process() returns; recvfrom clears XPT_BUSY in between (svc_rdma_recvfrom -> svc_xprt_received). So the deferred close is enqueued with XPT_BUSY clear and xpt_nr_rqsts != 0, and the close lands on sp_xprts. Since svc_xprt_dequeue() drains sp_xprts_hi first, the teardown waits behind idle-flow traffic precisely when the server is busy and wants the resources back.
Control work isn't request work, and svc_xprt_ready() already treats XPT_CONN|XPT_CLOSE|XPT_HANDSHAKE as a class distinct from XPT_DATA. Routing that class to the high-priority queue regardless of the request count keeps the two consistent:
if (xprt->xpt_flags & (BIT(XPT_CONN) | BIT(XPT_CLOSE) |
BIT(XPT_HANDSHAKE)) ||
!atomic_read(&xprt->xpt_nr_rqsts))
lwq_enqueue(&xprt->xpt_ready, &pool->sp_xprts_hi);
else
lwq_enqueue(&xprt->xpt_ready, &pool->sp_xprts);The burst-allowance patch narrows this -- a close with credit left rides sp_xprts_hi -- but a flow that has spent its budget still routes its close to the bulk queue, so the gap remains for exactly the backlogged connections most worth closing promptly.
You should also have a look at the pre-existing issue that sashiko
identified: it might be amplified by this patch, so we should consider addressing that issue as a pre-requisite to this series.
https://sashiko.dev/#/patchset/[email protected]?part=2
Signature · 1 line
Chuck Lever
Re: [PATCH RFC 2/3] SUNRPC: dispatch idle transports ahead of backlogged ones
Quoted 51 lines · Chuck Lever
Quoted 14 lines · Benjamin Coddington
A pool dispatches ready transports in FIFO order, so a connection that already has requests in flight sits in the same queue, on equal terms, as one that has been idle. A client driving many concurrent requests therefore delays the next request of an interactive client whose previous reply has already completed: the interactive request waits behind the busy client's backlog even though servicing it costs a single round trip.
net/sunrpc/svc_xprt.c +0
-0@@ -523,7 +523,10 @@ void svc_xprt_enqueue(struct svc_xprt *xprt)net/sunrpc/svc_xprt.c void svc_xprt_enqueue(struct svc_xprt *xprt) · line 532
Quoted 10 lines
net/sunrpc/svc_xprt.c +4
-1percpu_counter_inc (&pool->sp_sockets_queued); xprt->xpt_qtime =ktime_get (); -lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts); +if (atomic_read (&xprt->xpt_nr_rqsts)) +lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts); +else +lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts_hi);svc_pool_wake_idle_thread (pool); }The classification is keyed on in-flight request count, but a deferred close is enqueued the same way.
svc_xprt_deferred_close()setsXPT_CLOSEand callssvc_xprt_enqueue()while the slot is still held;svc_rdma_sendto()on a send error is one such path, andsvc_check_conn_limits()is another.svc_handle_xprt()reserves the slot before calling recvfrom and releases it only aftersvc_process()returns; recvfrom clearsXPT_BUSYin between (svc_rdma_recvfrom->svc_xprt_received). So the deferred close is enqueued withXPT_BUSYclear andxpt_nr_rqsts!= 0, and the close lands onsp_xprts. Sincesvc_xprt_dequeue()drainssp_xprts_hifirst, the teardown waits behind idle-flow traffic precisely when the server is busy and wants the resources back.Control work isn't request work, and
svc_xprt_ready()already treats XPT_CONN|XPT_CLOSE|XPT_HANDSHAKE as a class distinct fromXPT_DATA. Routing that class to the high-priority queue regardless of the request count keeps the two consistent:if (xprt->xpt_flags & (BIT(XPT_CONN) | BIT(XPT_CLOSE) | BIT(XPT_HANDSHAKE)) || !atomic_read(&xprt->xpt_nr_rqsts)) lwq_enqueue(&xprt->xpt_ready, &pool->sp_xprts_hi); else lwq_enqueue(&xprt->xpt_ready, &pool->sp_xprts);
Quoted 4 lines
The burst-allowance patch narrows this -- a close with credit left rides
sp_xprts_hi-- but a flow that has spent its budget still routes its close to the bulk queue, so the gap remains for exactly the backlogged connections most worth closing promptly.
Excellent - if we v1, I'll add this fix.
Quoted 5 lines
You should also have a look at the pre-existing issue that sashiko
identified: it might be amplified by this patch, so we should consider addressing that issue as a pre-requisite to this series.
I'll head over there (cool - sashiko looks smart) and give that issue attention for any v1.
Ben
Re: [PATCH RFC 2/3] SUNRPC: dispatch idle transports ahead of backlogged ones
In reply to Benjamin CoddingtonQuoted 18 lines · Benjamin Coddington
A pool dispatches ready transports in FIFO order, so a connection that already has requests in flight sits in the same queue, on equal terms, as one that has been idle. A client driving many concurrent requests therefore delays the next request of an interactive client whose previous reply has already completed: the interactive request waits behind the busy client's backlog even though servicing it costs a single round trip.
Route a transport with no requests in flight onto the new high-priority queue,
sp_xprts_hi, and drain that queue ahead ofsp_xprts.xpt_nr_rqstsis incremented when a thread reserves a request slot and decremented when the reply completes, so at enqueue time it counts the transport's prior in-flight work, not the request that just arrived: a flow whose last reply has completed reads zero and jumps the queue, while a flow with requests still in flight stays in the bulk queue. The classification needs no client identity and no lock -- each queue is an independent lwq with its own ordering, sosvc_thread_should_sleep()simply tests both.
Quoted 4 lines
This gives an idle flow's request a latency floor that a backlogged flow cannot push it below. It is starvation avoidance, not proportional
fairness: under sustained load across many idle flows the bulk queue can wait, and bounding that is left to later work.
I don't feel comfortable completely leaving this for later work. I think some minimal solution should be present from the start. Possibly we could simple alternate between interactive and batch. This would effectively defeat the unfairness created by using nconnect but otherwise share resources fairly well.
NeilBrown
Quoted 93 lines
Signed-off-by Benjamin Coddington <[email protected]>--- net/sunrpc/svc_xprt.c | 44 +++++++++++++++++++++++++++---------------- 1 file changed, 28 insertions(+), 16 deletions(-)net/sunrpc/svc_xprt.c +28
-16@@ -523,7 +523,10 @@ void svc_xprt_enqueue(struct svc_xprt *xprt)percpu_counter_inc (&pool->sp_sockets_queued); xprt->xpt_qtime =ktime_get (); -lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts); +if (atomic_read (&xprt->xpt_nr_rqsts)) +lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts); +else +lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts_hi);svc_pool_wake_idle_thread (pool); }@@ -536,7 +539,9 @@ static struct svc_xprt *svc_xprt_dequeue(struct svc_pool *pool) {struct svc_xprt *xprt =NULL ; - xprt = lwq_dequeue(&pool->sp_xprts, struct svc_xprt, xpt_ready); + xprt = lwq_dequeue(&pool->sp_xprts_hi, struct svc_xprt, xpt_ready); +if (!xprt) + xprt = lwq_dequeue(&pool->sp_xprts, struct svc_xprt, xpt_ready);if (xprt)svc_xprt_get (xprt);return xprt;@@ -759,7 +764,7 @@ svc_thread_should_sleep(struct svc_rqst *rqstp)return false;/* was a socket queued? */ -if (!lwq_empty (&pool->sp_xprts)) +if (!lwq_empty (&pool->sp_xprts_hi) || !lwq_empty (&pool->sp_xprts))return false;/* are we shutting down? */ @@ -1183,26 +1188,33 @@ static int svc_close_list(struct svc_serv *serv, struct list_head *xprt_list, streturn ret; } -static void svc_clean_up_xprts (struct svc_serv *serv,struct net *net) +static void svc_clean_up_queue (struct lwq *queue,struct net *net) {struct svc_xprt *xprt; +struct llist_node *q, **t1, *t2; + + q =lwq_dequeue_all (queue); +lwq_for_each_safe (xprt, t1, t2, &q, xpt_ready) { +if (xprt->xpt_net == net) { +set_bit (XPT_CLOSE , &xprt->xpt_flags); +svc_delete_xprt (xprt); + xprt =NULL ; + } + } + +if (q) +lwq_enqueue_batch (q, queue); +} + +static void svc_clean_up_xprts (struct svc_serv *serv,struct net *net) +{int i;for (i =0 ; i < serv->sv_nrpools; i++) {struct svc_pool *pool = &serv->sv_pools[i]; -struct llist_node *q, **t1, *t2; - - q =lwq_dequeue_all (&pool->sp_xprts); -lwq_for_each_safe (xprt, t1, t2, &q, xpt_ready) { -if (xprt->xpt_net == net) { -set_bit (XPT_CLOSE , &xprt->xpt_flags); -svc_delete_xprt (xprt); - xprt =NULL ; - } - } -if (q) -lwq_enqueue_batch (q, &pool->sp_xprts); +svc_clean_up_queue (&pool->sp_xprts_hi, net); +svc_clean_up_queue (&pool->sp_xprts, net); } }Signature · 1 line
2.53.0
RFC PATCH 3/3 SUNRPC: grant an idle flow a burst allowance on the high-priority queue
In reply to Benjamin CoddingtonRouting only the leading request of a flow to the high-priority queue protects single, isolated round trips but not interactive work: one client request -- a syscall, a page fault against an NFS-backed file -- fans out into many correlated RPCs. With a budget of one, the first jumps the queue and the rest fall to the bulk queue and finish at the aggressor's rate, so the command completes no sooner than before.
Grant a transport a budget of SVC_XPRT_HI_BURST high-priority dispatches when it becomes active after idling (xpt_nr_rqsts == 0 at enqueue), spending one credit per dispatch and falling back to the bulk queue once the budget is gone. The whole burst a request fans out into is now serviced ahead of backlogged flows, not just its leading edge.
The budget refills only on an idle-to-active transition, so a flow that stays continuously backlogged spends its budget once and then lives in the bulk queue: a client cannot hold the high-priority queue by keeping its connections busy or by opening more of them. xpt_hi_credit is touched only under the XPT_BUSY serialisation in svc_xprt_enqueue, so it needs no atomic or lock.
SVC_XPRT_HI_BURST is a fixed 64 for now -- generous enough to cover a typical request's RPC fan-out, and cheap to over-provision since a backlogged flow rides the high-priority queue for at most that many dispatches per idle period.
--- include/linux/sunrpc/svc_xprt.h | 1 + net/sunrpc/svc_xprt.c | 14 +++++++++++--- 2 files changed, 12 insertions(+), 3 deletions(-)
include/linux/sunrpc/svc_xprt.h +1 -0
@@ -61,6 +61,7 @@ struct svc_xprt {struct svc_serv *xpt_server;/* service for transport */ atomic_t xpt_reserved;/* space on outq that is rsvd */ atomic_t xpt_nr_rqsts;/* Number of requests */ +int xpt_hi_credit;/* high-priority dispatch budget */ struct mutex xpt_mutex;/* to serialize sending data */ spinlock_t xpt_lock;/* protects sk_deferred * and xpt_auth_cache */
net/sunrpc/svc_xprt.c +11 -3
@@ -499,6 +499,10 @@ static bool svc_xprt_ready(struct svc_xprt *xprt)return false; } +/* High-priority dispatch budget granted to a flow when it becomes active +* after idling -- sized to cover an interactive request's burst of RPCs. */ +#define SVC_XPRT_HI_BURST 64 +/** * svc_xprt_enqueue - Queue a transport on an idle nfsd thread * @xprt: transport with data pending
@@ -523,10 +527,14 @@ void svc_xprt_enqueue(struct svc_xprt *xprt)percpu_counter_inc (&pool->sp_sockets_queued); xprt->xpt_qtime =ktime_get (); -if (atomic_read (&xprt->xpt_nr_rqsts)) -lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts); -else +if (atomic_read (&xprt->xpt_nr_rqsts) ==0 ) + xprt->xpt_hi_credit =SVC_XPRT_HI_BURST ; +if (xprt->xpt_hi_credit >0 ) { + xprt->xpt_hi_credit--;lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts_hi); + }else { +lwq_enqueue (&xprt->xpt_ready, &pool->sp_xprts); + }svc_pool_wake_idle_thread (pool); }
Signature · 1 line
2.53.0
Chuck Lever
maintainerQuoted 3 lines
an
idle->activetransport is granted a budget of 64 high-priority dispatches [...] refilled only when the transport next idles.
[...]
Quoted 2 lines
Open questions for the list: is the budget (64 here) the right shape [...] And does the cooperative framing hold[?]
Before the budget shape, I wanted to pin down the re-arm trigger itself, since the whole floor rests on it. Traced through, the grant is gated on a true zero-crossing: svc_xprt_enqueue() refills the credit only when atomic_read(&xprt->xpt_nr_rqsts) == 0 at the instant new data arrives -- i.e. every earlier request on that transport has already passed svc_xprt_release(). Not a low-watermark; exact zero.
My worry was that a sustained interactive flow (a tree walk holding a couple of RPCs in flight) never reaches zero, spends its 64 once, and then rides the normal queue. The measurements push back on that. I put a kprobe on the in-flight depth at each committed enqueue and ran a cold v4.2 loopback walk of a 7500-file tree (~15.5k server RPCs):
depth 0 : 25.6% <- re-arms the 64-credit budget
depth 1 : 62.3%
depth 2 : 12.2%
depth 3 : ~0So zero recurs about every fourth enqueue. ~3 credits spent between refills against a budget of 64 -- the credit never approaches zero and the flow stays high-priority for the whole walk. Uncontended, the "== 0" trigger is far from one-shot.
The catch: this can't reach the case the series exists for. Loopback nfsd keeps up, so depth collapses on its own. The demotion risk is under pool contention -- aggressor saturating the threads, the interactive flow's replies delayed, depth staying elevated, the zero-crossings that refill credits becoming rare. I couldn't build that window without your per-op service-time hook.
Could you run the same probe under your aggressor + 10ms hook? Keyed by transport it separates the interactive flow from the aggressor in one histogram:
T=/sys/kernel/tracing
echo 'p:svcenq svc_xprt_enqueue depth=+76($arg1):s32 \
flags=+56($arg1):x64 xprt=$arg1:x64' >> $T/dynamic_events
echo '!(flags & 1)' > $T/events/kprobes/svcenq/filter
echo 'hist:keys=xprt,depth:sort=xprt,depth' \
> $T/events/kprobes/svcenq/trigger
echo 1 > $T/events/kprobes/svcenq/enable(+76 = offsetof(svc_xprt, xpt_nr_rqsts), +56 = xpt_flags, x86_64; unchanged by the series since xpt_hi_credit lands after it. depth=$arg1->xpt_nr_rqsts.counter:s32 might be a cleaner approach.
filter !(flags & 1) keeps only committed enqueues -- XPT_BUSY
clear at entry.)
The question that settles it: for the interactive xprt under load, does the depth==0 fraction hold up, or do you see depth>=1 runs longer than 64? If it holds, the trigger is fine and the uncovered tail is just the N>64 knee you already document. If it collapses, that's the argument for a low-watermark refill rather than exact-zero -- and it belongs in the cover letter, which currently asserts "idle" without showing an interactive flow reaches it under contention.
Signature · 1 line
Chuck Lever
Quoted 10 lines · Chuck Lever
My worry was that a sustained interactive flow (a tree walk holding a couple of RPCs in flight) never reaches zero, spends its 64 once, and then rides the normal queue.
...
Could you run the same probe under your aggressor + 10ms hook?
...
The question that settles it: for the interactive xprt under load, does the depth==0 fraction hold up, or do you see depth>=1 runs longer than 64? If it holds, the trigger is fine [...]. If it collapses, that's the argument for a low-watermark refill rather than exact-zero.
Hey Chuck - I didn't need to run any tests to be conviced that your re-arm problem exists, it seems obvious to me. That said - I did send a clanker out to run it, here's the response:
"
I ran it, and it collapses -- but as a sharp step, not a gradual fade, which I think is the useful part of the answer.
Setup: your probe (depth = xpt_nr_rqsts, filtered to committed enqueues, keyed by xprt) plus xpt_hi_credit, offsets from BTF. The victim is a sustained flow holding a fixed window W of RPCs continuously in flight on one connection -- a deliberately conservative stand-in for the tree walk:
a real walk has dependency gaps that return it to depth 0, mine refills instantly and so idles as rarely as possible. Aggressor is the backlogged worker set from the series; 10ms service hook throughout. (This is a different workload from the burst victim in the cover letter, which drains fully and idles 50ms between bursts -- that one re-arms every cycle and is not what's at issue here.)
Re-arm rate, committed-enqueue depth==0 fraction, by victim window:
W=1 (strictly serial) depth0 = 100% credit stays 63
W=2 depth0 = 0.4% credit pinned at 0
W=4 depth0 = 0.1% credit pinned at 0
W=8 depth0 = 0.2% credit pinned at 0So the whole thing hinges on the transport going *completely* empty. A strictly serial flow re-arms on every single enqueue and rides the high queue forever. The moment a flow keeps even two RPCs continuously in flight, the exact-zero condition essentially stops occurring: it spends its 64 within the first ~64 enqueues and then lives in the bulk queue -- depth>=1 runs are not "longer than 64", they're unbounded.
What that costs depends on how deep the bulk queue is. Victim per-op p50 (ms), vs window W and aggressor connection count K (floor = victim alone; aggressor p50 ~ the bulk rate: 17 / 27 / 52 ms for K = 4 / 8 / 16):
W floor K=4 K=8 K=16
1 14 14 16 17 <- serial: protected at every K
2 26 25 37 26 <- marginal, right on the edge
3 20 19 24 39
4 17 17 28 52 <- == aggressor: fully demoted
6 16 17 42 79
8 16 19 57 106 <- worse than the aggressor
Under light contention (K=4) bulk is shallow, so even a non-re-arming flow stays near its floor and the demotion is invisible. It only bites once the bulk queue is deep, and then the threshold window shrinks as K grows -- by K=16 anything holding >=3 in flight is at or past the aggressor's latency. (W=8 lands *worse* than the aggressor because the demoted victim's own workers then serialize behind each other in the bulk queue.)
Reconciling with your uncontended histogram (depth 0: 25.6%, 1: 62.3%):
that cold walk idles to zero a quarter of the time, which is plenty -- it sits up at the W=1 end and would stay protected. So the exact-zero trigger is fine for genuinely sparse or gappy flows. Where it's brittle is the continuously-pipelined flow with no gaps -- aggressive readahead, an app doing async I/O, a v4.1 client that keeps its slot table full -- which demotes wholesale the first time it sustains a couple RPCs under load.
So I read this as the argument for the low-watermark refill you floated:
re-arm when depth drops below a small threshold rather than at exactly 0. A watermark of even 2-4 would move the W=1 column's behavior out to cover the modestly-pipelined flows, while still excluding a real bulk client (which sits far above any sane watermark). It doesn't reintroduce identity and it's a one-line change to the refill condition.
I could spin that as a 4th patch for v1 and re-run this sweep against it -- I'd expect the protected band to widen from "W<=1" to "W<=watermark". Does a fixed small watermark match your intuition, or were you thinking of deriving it (thread count, or the slot-table hint you mentioned)?
The probe harness (kprobe sweep + the load generator) is standalone if you want to reproduce any of this.
"
Ben
Chuck Lever
maintainerQuoted 11 lines · Benjamin Coddington
So I read this as the argument for the low-watermark refill you floated:
re-arm when depth drops below a small threshold rather than at exactly 0. A watermark of even 2-4 would move the W=1 column's behavior out to cover the modestly-pipelined flows, while still excluding a real bulk client (which sits far above any sane watermark). It doesn't reintroduce identity and it's a one-line change to the refill condition.I could spin that as a 4th patch for v1 and re-run this sweep against it -- I'd expect the protected band to widen from "W<=1" to "W<=watermark". Does a fixed small watermark match your intuition, or were you thinking of deriving it (thread count, or the slot-table hint you mentioned)?
Hey Ben,
Thanks for running this, and thanks to the clanker too. Two things fell out when I put the numbers next to the refill code, and together they push me past the watermark model and toward letting the credit decay more gracefully.
The first is that the budget resets to a full SVC_XPRT_HI_BURST every time the transport idles; it doesn't accumulate. So a flow doesn't need a high depth==0 fraction to stay protected, but rather it only needs to hit zero once per 64 high-priority dispatches. That reframes the W=2 collapse:
0.4% is below 1/64 (~1.6%), so the flow runs dry between idle moments and falls to the bulk queue. Your uncontended histogram sat at 25.6%, which is 16x above that line. So the question we care about isn't whether the depth==0 fraction "holds up", it's whether a real interactive workload under load stays north of ~1.6%.
That's the metric we still don't have. The synthetic victim refills instantly and idles as rarely as it can, so it answers a question we already knew the answer to: a flow that never goes idle never re-arms on idle. Before changing anything I'd want the same depth==0 probe under load against a real workload: a cold tree walk, git status, a build's header reads, instead of the fixed-window generator. If those stay above 1.6%, exact-zero is fine and we'd be polishing a non-problem.
The second thing is about the watermark itself. Because the credit resets to full whenever the trigger fires, the trigger is binary per flow: a flow either re-arms often enough to live on the high queue forever, or it spends its 64 once and settles in the bulk queue. Exact-zero puts that split between W=1 and W=2. A watermark of N doesn't soften the split, it slides it to between W=N and W=N+1. Anything under the watermark gets near- permanent residency, anything over it demotes wholesale, same as today. So fixed vs derived is really just picking where to stand the cliff, not /whether/ there's a cliff.
If what we actually want is for protection to fade as a flow gets busier rather than snap off at a line, the budget has to decay instead of reset.
Roughly: spend a credit per high-priority dispatch as it does now, but on the idle transition add back a fraction of the budget instead of slamming it to 64. A flow with frequent gaps tops up faster than it spends and keeps its place. A flow that idles rarely bleeds down and slides into the bulk queue gradually, in proportion to how backlogged it really is.
I haven't pinned the refill fraction, and that wants the same sweep your harness already does. If you're up for it, I'd love three lines on your
axes: exact-zero (the RFC), a fixed watermark, and a decaying credit. My bet is the first two are cliffs in different spots and the third is a ramp. If the ramp lands where we want the protected band to sit, it answers the fixed-vs-derived question by making it moot.
Let's get the real-workload depth==0 numbers first, because if they stay high under load, none of the rest matters.
Signature · 1 line
Chuck Lever
Quoted 133 lines · Benjamin Coddington
This RFC follows the per-client fair-scheduling discussion from May/June [1], and Chuck's slot-growth clamp that grew out of it [2]. It takes the direction that thread settled on: rather than enforce proportional fairness between clients, give an interactive client a latency floor that a busy neighbour cannot push it below.
Problem
-------nfsd dispatches ready transports per-pool in FIFO order, with no notion of how much work each transport already has outstanding. A client that keeps many requests in flight -- nconnect, a deep v4.1 slot table, or simply several connections -- sits in that queue on equal terms with a client that has been idle. So the request a user is waiting on (a stat, an open, a directory listing) waits behind a backup job's backlog, even though servicing it costs a single round trip. The user-visible symptom is an interactive session that stutters whenever a bulk client is busy.
Approach
--------This is starvation avoidance, not proportional fairness (per Chuck's reframe on [1]). The protected unit is the interactive cycle: one user command fans out into a burst of correlated RPCs, so it is the whole burst, not a single RPC, that must be allowed to jump ahead.
The signal is already on every transport:
xpt_nr_rqsts, the count of requests in flight. A transport with nothing in flight when new work arrives is, by definition, not the one hogging threads -- so it goes on a second, high-priority per-pool queue that is drained ahead of the normal one. To cover the whole burst rather than only its leading edge, anidle->activetransport is granted a budget of 64 high-priority dispatches, spent down as the burst is serviced and refilled only when the transport next idles. A transport that never idles spends its budget once and then shares the normal queue like everyone else.There is no client identity anywhere in this -- it keys only on a transport's own in-flight depth -- so it covers v3, v4.0 and v4.1 uniformly, needs no lookup or lifecycle, adds no lock (the existing lwq barriers carry the sleep/wake), and is always on with nothing to tune. (LOCALIO is out of scope: a local client's reads and writes bypass the RPC dispatch path entirely, so they neither contend here nor are reordered by this change.)
patch 1 add the second per-pool ready queue (no functional change) patch 2 dispatch idle transports ahead of backlogged ones patch 3 grant the 64-RPC burst budgetRelationship to Chuck's slot-growth RFC
---------------------------------------This series is based on Chuck's "Stop NFSv4.1 slot-growth heuristic from rewarding busy clients" [2], and both A and B above include it -- so the A/B delta isolates this series and the clamp does not account for the improvement.
The two are complementary, not overlapping. [2] stops nfsd from growing a busy session's slot table past the thread ceiling: nfsd slot accounting. This series changes the order in which ready transports are dispatched:
sunrpc dispatch. Basing on [2] changes neither the design nor the result here -- a slot-capped session still fills the pool, so the dispatch-layer floor is still wanted on top of it.Results
-------Because the goal is a latency floor and not a share, the measurement differs from the earlier bucket RFC and the numbers are not comparable:
that series measured a greedy client's share of throughput; this one measures how long an interactive client's command takes to complete while a neighbour is busy.Workload: a backlogged aggressor (K connections, deep offered load, saturating the pool) shares the server with one interactive client that issues a burst of N requests, waits for all replies, idles 50ms, and repeats. A 10ms service time is injected per op (a test-only debug hook, not part of this series) so the measurement isolates the dispatch path from filesystem work. A is stock nfsd-testing; B is the same plus this series. Figures are NFSv3 burst-completion p50 in ms; NFSv4.1 tracks within a few percent (the classifier uses no source identity, so v3 runs on plain loopback and behaves the same).
Interactive burst completion vs aggressor connection count K (N=32; unobstructed floor 45ms):
K 4 8 16 32 A 81.6 242.0 432.9 826.6 B 60.9 64.5 65.4 64.4 1.3x 3.8x 6.6x 12.8xBaseline climbs ~linearly with the aggressor's backlog; patched holds the floor no matter how deep that backlog is -- the property that matters.
vs burst size N (K=16), showing the 64-credit knee (floor for reference):
N 1 8 32 64 96 128
floor 14.9 31.9 44.8 71.5 96.8 122.4 A 28.6 122.0 434.2 862.4 1272.6 1696.3 B 16.7 36.0 65.1 124.5 546.4 961.6For N <= 64 the whole burst is covered and tracks the floor (B within 1.7x); beyond 64 the uncovered tail -- the N-64 RPCs that spill to the normal queue -- degrades toward baseline (B/floor jumps 1.7x -> 5.6x across the 64->96 boundary). The knee at exactly 64 is the budget working as designed, not a regression.
Aggregate throughput (aggressor + interactive) is within 1% A vs B at saturation (~1290 ops/s) -- the dispatcher reorders, it does not throttle. With no aggressor the interactive floor is identical A vs B; the kernels differ only under contention.
Figures are p50; longer runs are available for firmer tails.
What this deliberately does not do
----------------------------------- It is not proportional fairness. Two backlogged clients still split the pool by connection count; this protects only flows that idle.
- The normal queue can wait under sustained high-priority load. v1 drains high-priority strictly first; bounding that starvation -- an inter-tier deficit round-robin, the way
fq_codelactually interleaves its sparse and bulk tiers -- is left for a follow-up, pending the question below.- The 64-dispatch budget defeats a pipelined aggressor (it lands wholly in the normal queue), but a client that shapes its traffic to look sparse -- send <=64, drain, repeat -- can still scale its high-priority share with connection count. Whether to harden against that is the open
question: if this is cooperative scheduling (Neil's framing) a bulk floor suffices; if it is an attack surface (Chuck's framing) the high-priority tier needs intra-tier fairness, which reintroduces identity.
Quoted 4 lines
Open questions for the list: is the budget (64 here) the right shape, and should it be fixed, derived from the thread count, or hinted by the v4.1 slot table? And does the cooperative framing hold, or must we resist deliberate sparse-shaping?
I wonder if we should make "batch" the special case, rather than "interactive".
I don't think this makes a big difference to the code (there are still two queues and everyone starts out interactive) but it might change how we think about it.
I would wait a lot longer than 64 requests before considering an xprt to be "batch", and I would probably want to measure it in seconds rather than requests. Maybe 10-15 seconds.
I might also only consider an xprt batch if xpt_nr_rqsts remains above some small number - maybe 5.
Maybe we could keep track of the number of "batch" xprts and compare it to the thread limit. Batch xprts might only be allowed some share of the thread limit...
If there are a bunch of READ requests on the same file, the first could trigger a read-ahead, and the next several might wait for that read-ahead, so they would all be blocked on the same thing, which can be pointless. nfsd doesn;'t have any direct visibility into this, but limiting requests-per-xprt differently for batch and interactive might be useful. non-sync WRITE requests probably behave differently...
While this approach does seem pleasingly simple, I wonder how easy it would be for a client to accidentally start appearing to be "batch". Multiple interactive sessions on the one client could incorrectly trigger the batch detection.
Maybe it would be useful to collect some statistics of what a "batch" stream typically looks like (OP mix, concurrency,...) as there may be a better signal to look for than xpt_nr_rqsts.
Thanks,
NeilBrown
Quoted 20 lines
[1] https://lore.kernel.org/linux-nfs/[email protected]/
[2] https://lore.kernel.org/linux-nfs/[email protected]/Benjamin Coddington (3):
SUNRPC: add a second per-pool ready queue for high-priority transports
SUNRPC: dispatch idle transports ahead of backlogged ones
SUNRPC: grant an idle flow a burst allowance on the high-priority
queueinclude/linux/sunrpc/svc.h | 1 + include/linux/sunrpc/svc_xprt.h | 1 + net/sunrpc/svc.c | 1 + net/sunrpc/svc_xprt.c | 52 +++++++++++++++++++++++---------- 4 files changed, 39 insertions(+), 16 deletions(-)Signature · 1 line
2.53.0
Quoted 23 lines · NeilBrown
I wonder if we should make "batch" the special case, rather than "interactive".
I don't think this makes a big difference to the code (there are still two queues and everyone starts out interactive) but it might change how we think about it.I would wait a lot longer than 64 requests before considering an xprt to be "batch", and I would probably want to measure it in seconds rather than requests. Maybe 10-15 seconds.
I might also only consider an xprt batch ifxpt_nr_rqstsremains above some small number - maybe 5.Maybe we could keep track of the number of "batch" xprts and compare it to the thread limit. Batch xprts might only be allowed some share of the thread limit...
If there are a bunch of READ requests on the same file, the first could trigger a read-ahead, and the next several might wait for that read-ahead, so they would all be blocked on the same thing, which can be pointless. nfsd doesn;'t have any direct visibility into this, but limiting requests-per-xprt differently for batch and interactive might be useful. non-sync WRITE requests probably behave differently...
Quoted 8 lines
While this approach does seem pleasingly simple, I wonder how easy it would be for a client to accidentally start appearing to be "batch". Multiple interactive sessions on the one client could incorrectly trigger the batch detection.
Maybe it would be useful to collect some statistics of what a "batch" stream typically looks like (OP mix, concurrency,...) as there may be a better signal to look for than
xpt_nr_rqsts.
I like your thinking here - but while I've been putting some numbers together for the re-arm problem Chuck was pointing out, I've found the behavior of the server (even with this simple mechanism) to be surprisingly complex. The re-arm floor/ceiling is highly dependent on timing from multiple actors.
I'm not sure I can appropriately characterize what batch looks like. I'm afraid I'll end up creating a heuristic monster that will hunt us down in the future.
I'm starting to think that per-client round-robin is suddenly looking simpler to understand and will do the right thing in most cases, but I haven't really given up on this dual-queue setup yet.
Here's another crazy idea: instead of letting me break this for everyone, let's leave the current behavior untouched and create a bpf interface so folks like me can inject their desired scheduler from userspace.
Ben