sched/fair, KVM: Semantics-aware directed yield for oversubscribed KVM
v3 cover sched/fair, KVM: Semantics-aware directed yield for oversubscribed KVM
From: Wanpeng Li <[email protected]>
On overcommitted hosts, a spinning vCPU often calls yield_to() to let a lock holder or IPI receiver run. The hint can be ineffective for two independent reasons: the scheduler may fail to select the nominated task, and KVM may nominate a task that is not the one the spinning vCPU is waiting for.
This series addresses both sides. The scheduler side credits bounded EEVDF lag to the nominated next-buddy so the buddy hint is honored across the relevant cgroup hierarchy, and forces a local reschedule so the credited buddy can be selected immediately. The KVM side tracks recent unicast fixed IPI sender/receiver pairs and prefers the confirmed receiver when selecting a directed-yield target.
Problem Statement
-----------------
In overcommitted virtualization scenarios, vCPUs frequently spin on locks held by other vCPUs that are not currently running, or on IPI responses from vCPUs that are runnable but not scheduled. Paravirtual spinlock support and PLE detect these situations and call yield_to() to let the other vCPU make progress.
The current implementation has two limitations:
1. Scheduler-side limitation:
yield_to_task_fair() relies on set_next_buddy() to express a preference for the target. set_next_buddy() nominates the target at every level of its cgroup ancestor chain, but pick_eevdf()'s PICK_BUDDY branch only returns cfs_rq->next when that entity is already eligible (entity_eligible()). A target that is behind avg_vruntime at any level of the chain is skipped, and the hint is dropped at the first ineligible group entity.
Even when the target is eligible, yield_to() does not by itself force the caller off the CPU. An active RUN_TO_PARITY protect_slice() on the local yielder can therefore keep pick_eevdf() returning the yielder instead of the target.
The recent forfeit-on-yield work (commits 79104becf42b "sched/fair:
Forfeit vruntime on yield" and 127b90315ca0 "sched/proxy: Yield the donor task") makes the yielder ineligible, but it does not make the nominated target eligible when that target is behind avg_vruntime, keep PICK_BUDDY from being dropped at the first ineligible group entity, or cancel an active RUN_TO_PARITY slice on the yielder. This series builds on that behavior by crediting the target and cancelling slice protection, so the nominated entity is the one pick_eevdf() returns.
2. KVM-side limitation:
kvm_vcpu_on_spin() selects a directed-yield target from coarse preempted / preempted-in-kernel state. It cannot distinguish a vCPU spinning on an IPI response from a vCPU spinning on a lock. When a vCPU sends an IPI and spins waiting for the response, the heuristic can boost an unrelated vCPU and miss the actual IPI receiver.
These effects lengthen lock hold times and increase spin time, context-switch overhead and cache pressure in overcommitted environments, especially for workloads with fine-grained synchronization.
Solution Overview
-----------------
Part 1: Scheduler EEVDF lag credit (patches 1-5)
Rather than penalizing the yielding vCPU, credit the nominated target so pick_eevdf() honors the buddy hint.
The mechanism is EEVDF-native and cgroup-hierarchy-aware:
- Credit bounded EEVDF lag to the nominated next-buddy so pick_eevdf()'s PICK_BUDDY branch returns it. Walk the same ancestor chain that set_next_buddy() nominated and credit each not-yet-eligible level, so the hint is not dropped at the first ineligible group entity.
- Credit to a small positive-vlag margin, not merely the vlag = 0 eligibility boundary, so the target stays eligible across several scheduling decisions rather than a single pick. The margin scales with runqueue depth and is clamped to entity_lag()'s legal positive-lag bound, preserving EEVDF fairness.
- Handle both the off-tree current entity (shifted in place, carrying any vprot window) and a queued (on-tree) entity (repositioned via the canonical place_entity()-paired requeue used by requeue_delayed_entity(), keeping sum_w_vruntime consistent with entity_key()).
- Force a local reschedule at the end of the credit path: cancel RUN_TO_PARITY slice protection along the yielder's sched_entity chain and resched_curr() the local CPU. Only this forced preemption is rate limited (once per 6ms per rq) to avoid excessive forced preemption on PLE-heavy guests; the lag credit itself runs on every directed yield.
The mechanism is gated by SCHED_FEAT(YIELD_TO_LAG_CREDIT) (default on). With the feature off, yield_to_task_fair() keeps the existing forfeit-only behavior.
Part 2: KVM IPI-aware directed yield (patches 6-10)
KVM tracks recent unicast fixed IPI sender/receiver relationships and uses them to prioritize directed-yield targets.
- Record unicast fixed IPIs from both LAPIC delivery paths, the APIC-map fast path and the slow fallback, when exactly one destination vCPU accepts the interrupt.
- Use READ_ONCE()/WRITE_ONCE() accessors. The per-vCPU ipi_context state is only a best-effort scheduling hint.
- Age out stale relationships with a recency window (50ms default), and clear state on a matching-vector EOI without dropping unrelated pending IPI state.
Directed-yield candidate selection uses the following priority order:
1. A confirmed recent IPI receiver of the spinning vCPU.
2. The arch-specific pending-interrupt hint
(kvm_arch_dy_has_pending_interrupt()).
3. The existing preempted / preempted-in-kernel heuristic.
If the strict IPI-aware pass finds no eligible candidate, an optional second pass falls back to a relaxed preempted-only search. The fallback is controlled by the enable_relaxed_boost module parameter (default on).
Runtime controls:
* /sys/kernel/debug/sched/features (YIELD_TO_LAG_CREDIT)
* /sys/module/kvm/parameters/ipi_tracking_enabled
* /sys/module/kvm/parameters/ipi_window_ns
* /sys/module/kvm/parameters/enable_relaxed_boost
Host-side deployment model
--------------------------
The series is host-side by design. It requires no guest ABI, paravirtual driver, negotiated feature bit, or guest kernel change, so existing guests benefit without coordination between host and guest software.
That deployment model gives the mechanisms broad coverage. The scheduler lag credit applies to every yield_to() the host already receives, including PLE and paravirtual spinlock paths. The KVM side observes the actual unicast-IPI sender/receiver relationship at software LAPIC delivery time, so it covers spin and IPI waits from spinlocks, RCU, smp_call_function() and IPI-based wakeups rather than a single paravirtualized operation such as TLB shootdown.
The host-side approach also composes with existing paravirtualization. If a guest provides PV TLB shootdown or PV spinlocks, those interfaces reduce the amount of spinning that reaches the host; this series handles the residual yield_to() and IPI waits that remain. It is runtime gated as described above and can be enabled or disabled per host.
The scheduler side is independent of APICv, IPI virtualization and the LAPIC delivery path. The KVM side depends on software LAPIC delivery: when IPI/EOI virtualization handles the guest's ICR and EOI writes in hardware, no sender/receiver relationship is recorded, and candidate selection falls back to the pending-interrupt and preempted heuristics, plus the relaxed preempted-only pass added in patch 10. In that configuration the tracking state stays empty while the scheduler side remains fully active.
The design separates the consumer of the hint from its source. Software IPI tracking supplies the confirmed receiver on hosts where software LAPIC delivery is observable today; a future guest-cooperative scheduling hint could populate the same slot without changing the priority-ordered candidate selection.
Performance Results
-------------------
Test environment: a 16-core x86-64 host, 16 vCPUs per guest. Host CPU overcommit is varied by co-locating 2, 3 and 4 guests (120 runs per point), with APICv disabled so the KVM side observes IPI delivery in software. Dbench reports throughput and reflects the scheduler-side lag credit; the PARSEC workloads report end-to-end latency reduction under the full series.
Dbench (filesystem metadata operations), throughput improvement:
2 VMs: +6.65% 3 VMs: +4.80% 4 VMs: +7.59%
PARSEC Dedup, simlarge input (IPI-heavy synchronization), latency reduction:
2 VMs: +8.87%
3 VMs: +10.29%
4 VMs: +15.60%
PARSEC VIPS, simlarge input (balanced sync and compute), latency reduction:
2 VMs: +10.23%
3 VMs: +6.63%
4 VMs: +4.50%
Analysis:
- Dedup's gains grow with the VM count: as more runnable vCPUs compete for each physical CPU, a directed yield is more likely to land on a vCPU that is genuinely preempted while an IPI sender spins, so honoring the confirmed receiver matters more.
- Dedup, with its IPI-heavy synchronization, benefits most from the IPI-aware directed yield. Preferring the confirmed IPI receiver over the generic preempted-lock-holder heuristic shortens IPI response latency.
- VIPS mixes synchronization and compute, so its gains shrink as the VM count rises: at higher overcommit more of each run is spent in compute that a directed yield cannot accelerate, leaving less spin time to recover.
- Dbench benefits primarily from the scheduler-side lag credit; its lock patterns involve more direct lock-holder boosting than IPI spinning.
- No configuration regressed; the mechanisms degrade gracefully as contention rises.
The gains stem from three factors:
1. Lock holders receive sustained CPU time to complete critical sections, reducing lock hold duration and cascading contention.
2. IPI receivers are scheduled promptly when senders spin, reducing IPI response latency and wasted spin cycles.
3. Reduced context switching between lock waiters and holders improves cache utilization.
Scope of the scheduler-side benefit
-----------------------------------
The lag credit takes effect only when the yielding vCPU and its target share a runqueue, i.e. when more runnable vCPUs than pCPUs contend for a CPU:
- Under CPU overcommit - co-located guests, or a VM whose vCPUs are pooled onto fewer pCPUs than it has vCPUs - the waiter and the lock-holder or IPI-receiver land on the same rq, and the buddy hint applies. The results here are from this regime, with guests co-located so their vCPUs contend for shared pCPUs.
- Without such contention - 1:1 vCPU:pCPU pinning, or a matched vCPU:pCPU count with no intra-VM overcommit - there is no eligible buddy to credit, so the path is inert and adds no overhead or regression.
Independent s390 testing (directed yield there uses the diag9c hypercall) shows the same pattern: under intra-VM vCPU pooling the yield-to hypercall rate falls by more than half with a few percent throughput gain, while 1:1 pinning and matched vCPU:pCPU configurations show no change either way.
Directed yield is a same-runqueue mechanism and cannot help a waiter whose target is on a different rq; extending it to cross-runqueue cases is left as future work.
Patch Organization
------------------
Patches 1-5: Scheduler EEVDF lag credit
Patch 1: Add the eevdf_credit_entity_vlag() primitive and the
YIELD_TO_LAG_CREDIT feature. Handles the off-tree current entity and has no functional effect on its own.
Patch 2: Credit to a persistent, queue-depth-scaled positive-vlag
margin, clamped to entity_lag()'s legal bound.
Patch 3: Extend the primitive to a queued (on-tree) entity via the
canonical place_entity()-paired requeue.
Patch 4: Wire the credit walk into yield_to_task_fair(), crediting each
level of the nominated ancestor chain.
Patch 5: Force a local reschedule (cancel RUN_TO_PARITY slice protection
and resched_curr()) so the credited buddy can be selected. Activation patch; rate-limits only the forced preemption.
Patches 6-10: KVM IPI-aware directed yield
Patch 6: Add per-vCPU IPI tracking infrastructure, module parameters and
helper functions. Candidate selection is unchanged.
Patch 7: Track unicast fixed IPI delivery from both LAPIC paths.
Patch 8: Clear IPI tracking on a matching-vector EOI.
Patch 9: Implement IPI-aware directed-yield candidate selection with the
priority order above.
Patch 10: Add the relaxed preempted-only fallback as a safety net.
Testing
-------
Workloads tested:
- Dbench (filesystem metadata stress)
- PARSEC benchmarks (Dedup, VIPS)
- Kernel compilation (make -j16 in each VM)
No regressions observed on any configuration. The mechanisms show neutral to positive impact across diverse workloads.
Rate-limit policy
-----------------
The scheduler-side forced reschedule is rate-limited to bound the cost of frequent VM exits. Under the kvm-full profile, PLE-heavy workloads such as PARSEC VIPS and Dedup take many PAUSE-loop exits; each exit can drive a yield_to(), and thus a potential forced preemption. Forcing a reschedule on every yield_to() would add needless preemption pressure and cache churn.
The series limits only the forced preemption path (cancel_protect_slice() plus resched_curr()) to once per 6ms per rq. The lag credit itself remains unthrottled, so each directed yield refreshes the buddy hint. The fixed 6ms interval is intentionally conservative; an adaptive limit based on the per-rq yield_to()/PLE-exit rate can be explored separately.
Changelog:
v2 -> v3:
- Redesign the scheduler side. v2 applied a bounded vruntime penalty to the yielding vCPU (a "debooster"); v3 instead credits bounded EEVDF lag to the nominated next-buddy so pick_eevdf()'s PICK_BUDDY branch returns it. Crediting the target is EEVDF-native, composes cleanly with RUN_TO_PARITY, and avoids the fairness reasoning required when shifting the yielder's vruntime in a cgroup hierarchy. The redesign also removes the bulk of the v2 machinery:
* Drop the cgroup LCA finder, reverse-pair debouncing, the per-rq penalty tracking and the dedicated debugfs sysctl. The mechanism is now gated by SCHED_FEAT(YIELD_TO_LAG_CREDIT).
* Credit to a queue-depth-scaled positive-vlag margin clamped to entity_lag()'s legal bound, keeping the target eligible across several picks while preserving EEVDF fairness.
* Handle the off-tree current entity (in-place shift) and a queued on-tree entity (canonical place_entity()-paired requeue) separately, so sum_w_vruntime stays consistent with entity_key().
* Add an explicit forced local reschedule that cancels RUN_TO_PARITY slice protection so the credited buddy can be selected; only the forced preemption is rate limited (6ms/rq), the lag credit runs on every yield.
- KVM side keeps the v2 design; rebased and reorganized into five patches (infrastructure, track delivery, clear-on-EOI, candidate selection, relaxed fallback). Tracking now hooks both the APIC-map fast path and the slow fallback, and the EOI clear is vector-matched.
- Rebase onto v7.1-rc7.
v1 -> v2:
- Rebase onto v6.19-rc1 (v1 was based on v6.18-rc4).
- Drop the "KVM: Fix last_boosted_vcpu index assignment bug" patch, as v6.19-rc1 already contains the fix.
- Scheduler side (the v2 vruntime debooster, since replaced in v3):
* Apply the deboost before yield_task_fair() to adapt to v6.19's EEVDF forfeit behavior (se->vruntime = se->deadline), which would otherwise inflate the yielder's vruntime before the penalty was computed.
* Use rq->donor instead of rq->curr for correct EEVDF donor tracking.
* Use h_nr_queued instead of nr_queued for accurate hierarchical task counting in the penalty cap.
* Drop the vlag assignment (recalculated on dequeue/enqueue) and the update_min_vruntime() call (the yielder is cfs_rq->curr, off-tree), and remove the unnecessary gran_floor safeguard.
* Rename the debugfs knob to vcpu_debooster_enabled.
- KVM IPI tracking: improve module-parameter documentation and add the kvm_vcpu_is_ipi_receiver() declaration to x86.h.
Wanpeng Li (10):
sched/fair: Add EEVDF lag credit primitive for nominated next-buddy sched/fair: Credit a persistent, queue-depth-scaled vlag margin sched/fair: Credit queued next-buddy via canonical requeue sched/fair: Credit nominated next-buddy in yield_to_task_fair() sched/fair: Force a local resched on yield_to() so the buddy is picked
KVM: x86: Add IPI tracking infrastructure for directed yield
KVM: x86/lapic: Track unicast fixed IPI delivery
KVM: x86/lapic: Clear IPI tracking on matching-vector EOI
KVM: Add IPI-aware directed-yield candidate selection
KVM: Add relaxed preempted-only fallback for directed yield
arch/x86/include/asm/kvm_host.h | 19 +++ arch/x86/kvm/lapic.c | 234 +++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 3 + arch/x86/kvm/x86.h | 8 ++ include/linux/kvm_host.h | 8 ++ kernel/sched/fair.c | 224 +++++++++++++++++++++++++++++- kernel/sched/features.h | 9 ++ kernel/sched/sched.h | 10 ++ virt/kvm/kvm_main.c | 95 +++++++++++-- 9 files changed, 594 insertions(+), 16 deletions(-)
base-commit: 4549871118cf616eecdd2d939f78e3b9e1dddc48
Signature · 1 line
2.43.0
PATCH v3 1/10 sched/fair: Add EEVDF lag credit primitive for nominated next-buddy
From: Wanpeng Li <[email protected]>
pick_eevdf()'s PICK_BUDDY path only returns cfs_rq->next when the entity is eligible. A yield_to() target that is behind avg_vruntime at any level of its sched_entity hierarchy is skipped, and the set_next_buddy() hint is lost.
Add eevdf_credit_entity_vlag(), which can credit a nominated entity up to the eligibility boundary so that pick_eevdf() can honor the buddy hint. The helper handles cfs_rq->curr, which is off-tree and can be shifted in place while carrying any active vprot window.
Gate the helper behind SCHED_FEAT(YIELD_TO_LAG_CREDIT). The helper has no caller in this change, so mark it __maybe_unused; there is no functional change.
--- kernel/sched/fair.c | 48 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/features.h | 9 ++++++++ 2 files changed, 57 insertions(+)
kernel/sched/fair.c +48 -0
@@ -9341,6 +9341,54 @@ static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t } } +/* +* eevdf_credit_entity_vlag - credit a nominated next-buddy to eligibility +* +* Advance @se (already nominated by set_next_buddy(), so cfs_rq->next == se) +* just enough negative vlag to reach the eligibility boundary (vlag = 0) so +* pick_eevdf()'s PICK_BUDDY branch returns it. cfs_rq->curr is shifted in +* place (off-tree, carrying any vprot window). Queued entities are left +* unchanged. +* +* Idempotent: a no-op once @se is already eligible. Caller must hold +* rq_of(cfs_rq)->lock with rq_clock up to date. +*/ +static void __maybe_unused +eevdf_credit_entity_vlag (struct cfs_rq *cfs_rq,struct sched_entity *se) +{ +u64 avruntime, credit; +s64 vlag; + +/* Callers gate this helper with YIELD_TO_LAG_CREDIT. */ +if (cfs_rq->nr_queued <2 ) +return ; +if (throttled_hierarchy (cfs_rq)) +return ; +if (WARN_ON_ONCE (!se->on_rq) || se->sched_delayed) +return ; + +update_curr (cfs_rq); + avruntime =avg_vruntime (cfs_rq); + vlag =entity_lag (cfs_rq, se, avruntime); + +/* Already eligible: nothing to do. */ +if (vlag >=0 ) +return ; + + credit = (u64)(-vlag); + +if (cfs_rq->curr == se) { +/* curr is off-tree: in-place shift, carrying any vprot window. */ +if (protect_slice (se)) + se->vprot -= credit; + se->vruntime -= credit; + se->deadline -= credit; +return ; + } + +/* Queued entities are left unchanged by this helper path. */ +} +/* * sched_yield() is very simple */
kernel/sched/features.h +9 -0
@@ -40,6 +40,15 @@ SCHED_FEAT(NEXT_BUDDY, false)*/ SCHED_FEAT (PICK_BUDDY , true) +/* +* Let yield_to_task_fair() credit bounded EEVDF lag to the nominated +* next-buddy so pick_eevdf() honors the hint even when the target has +* negative vlag at some level of its ancestor chain. The credit is bounded +* by a queue-depth-scaled margin within entity_lag()'s legal range, so +* fairness is preserved. + */ +SCHED_FEAT (YIELD_TO_LAG_CREDIT , true) +/* * Consider buddies to be cache hot, decreases the likeliness of a * cache buddy being migrated away, increases cache locality.
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v3 01/10] sched/fair: Add EEVDF lag credit primitive for nominated next-buddy
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] eevdf_credit_entity_vlag() inadvertently revives expired slice protection for the current task…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] eevdf_credit_entity_vlag() inadvertently revives expired slice protection for the current task because it only shifts vprot when protect_slice is true. --
commit d8a18d46db47fd7df4dc661550f9788e6c195ed7
Author: Wanpeng Li <[email protected]>
sched/fair: Add EEVDF lag credit primitive for nominated next-buddy
This commit adds eevdf_credit_entity_vlag() to allow yield_to() targets to receive bounded lag credit, enabling pick_eevdf() to honor buddy hints even for targets with a negative vlag. The helper updates cfs_rq->curr in place by shifting its vruntime, deadline, and optionally vprot backwards.
Quoted 4 lines
kernel/sched/fair.c +0
-0
[ ... ]
kernel/sched/fair.c static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t · line 9347
Quoted 8 lines
kernel/sched/fair.c +7
-0@@ -9341,6 +9341,54 @@ static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t + +if (cfs_rq->curr == se) { +/* curr is off-tree: in-place shift, carrying any vprot window. */ +if (protect_slice (se)) + se->vprot -= credit; + se->vruntime -= credit; + se->deadline -= credit;
[Severity: High]
Could conditionally updating se->vprot unintentionally revive expired slice protection?
If protect_slice(se) is currently false (meaning the protected slice has already expired and se->vprot <= se->vruntime), se->vprot is not shifted.
Since se->vruntime is unconditionally shifted backward by credit, couldn't se->vruntime fall behind the unmodified se->vprot?
This would cause protect_slice(se) to suddenly evaluate to true again, inappropriately shielding the task from preemption in update_curr().
Quoted 5 lines
kernel/sched/fair.c +5
-0+return ; + } + +/* Queued entities are left unchanged by this helper path. */ +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
K Prateek Nayak
reviewerRe: [PATCH v3 01/10] sched/fair: Add EEVDF lag credit primitive for nominated next-buddy
In reply to Wanpeng LiHello Wanpeng,
Quoted 1 line · Wanpeng Li
+static void
__maybe_unused
I would suggest adding this with the first caller to avoid having to jump between this and patch 3.
Quoted 12 lines
kernel/sched/fair.c +12
-0+eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se) +{ + u64 avruntime, credit; +s64 vlag; + +/* Callers gate this helper with YIELD_TO_LAG_CREDIT. */ +if (cfs_rq->nr_queued <2 ) +return ; +if (throttled_hierarchy (cfs_rq)) +return ; +if (WARN_ON_ONCE (!se->on_rq) || se->sched_delayed) +return ;
kernel/sched/fair.c static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t · line 9378
Quoted 10 lines
kernel/sched/fair.c +10
-0+ +update_curr (cfs_rq); + avruntime =avg_vruntime (cfs_rq); + vlag =entity_lag (cfs_rq, se, avruntime); + +/* Already eligible: nothing to do. */ +if (vlag >=0 ) +return ; + + credit = (u64)(-vlag);
What is this random credit and how is it fair? My suggestion on the previous version was simple:
Task0 -> yield_to -> Task1
- Task0 is eligible and has +ve lag
- Task0 transfers lag to Task1 and makes itself ineligible
- Task1 picks the proportional lag; It may become eligible and get to run but there is no guarantee because that is fair.
There is nothing fair about simply making a task eligible just because someone else decided to call yield_to() on it.
Review
Quoted 2 lines
(quoted diff) +2
-0+ + if (cfs_rq->curr == se) {
As mentioned in my reply on the cover, I'm having a hard time understanding how this can arise when we make sure "p" is a TASK_RUNNING that is also "!p->on_cpu && p->se.on_rq" while holding p->pi_lock and the rq_lock.
Quoted 10 lines
kernel/sched/fair.c +10
-0+/* curr is off-tree: in-place shift, carrying any vprot window. */ +if (protect_slice (se)) + se->vprot -= credit; + se->vruntime -= credit; + se->deadline -= credit; +return ; + } + +/* Queued entities are left unchanged by this helper path. */ +}
Signature · 2 lines
Thanks and Regards, Prateek
Vincent Guittot
maintainerRe: [PATCH v3 01/10] sched/fair: Add EEVDF lag credit primitive for nominated next-buddy
In reply to Wanpeng LiQuoted 56 lines · Wanpeng Li
From: Wanpeng Li <[email protected]>
pick_eevdf()'s
PICK_BUDDYpath only returnscfs_rq->nextwhen the entity is eligible. Ayield_to()target that is behindavg_vruntimeat any level of itssched_entityhierarchy is skipped, and theset_next_buddy()hint is lost.Add
eevdf_credit_entity_vlag(), which can credit a nominated entity up to the eligibility boundary so thatpick_eevdf()can honor the buddy hint. The helper handlescfs_rq->curr, which is off-tree and can be shifted in place while carrying any active vprot window.Gate the helper behind
SCHED_FEAT(YIELD_TO_LAG_CREDIT). The helper has no caller in this change, so mark it__maybe_unused; there is no functional change.Signed-off-by Wanpeng Li <[email protected]>--- kernel/sched/fair.c | 48 +++++++++++++++++++++++++++++++++++++++++ kernel/sched/features.h | 9 ++++++++ 2 files changed, 57 insertions(+)kernel/sched/fair.c +25
-0@@ -9341,6 +9341,54 @@ static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t } } +/* +* eevdf_credit_entity_vlag - credit a nominated next-buddy to eligibility +* +* Advance @se (already nominated by set_next_buddy(), so cfs_rq->next == se) +* just enough negative vlag to reach the eligibility boundary (vlag = 0) so +* pick_eevdf()'s PICK_BUDDY branch returns it. cfs_rq->curr is shifted in +* place (off-tree, carrying any vprot window). Queued entities are left +* unchanged. +* +* Idempotent: a no-op once @se is already eligible. Caller must hold +* rq_of(cfs_rq)->lock with rq_clock up to date. +*/ +static void __maybe_unused +eevdf_credit_entity_vlag (struct cfs_rq *cfs_rq,struct sched_entity *se) +{ +u64 avruntime, credit; +s64 vlag; + +/* Callers gate this helper with YIELD_TO_LAG_CREDIT. */ +if (cfs_rq->nr_queued <2 ) +return ; +if (throttled_hierarchy (cfs_rq)) +return ; +if (WARN_ON_ONCE (!se->on_rq) || se->sched_delayed) +return ;
kernel/sched/fair.c static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t · line 9378
Quoted 10 lines
kernel/sched/fair.c +10
-0+ +update_curr (cfs_rq); + avruntime =avg_vruntime (cfs_rq); + vlag =entity_lag (cfs_rq, se, avruntime); + +/* Already eligible: nothing to do. */ +if (vlag >=0 ) +return ; + + credit = (u64)(-vlag);
Please merge patch 1 and 2 because the current function is incomplete on its own.
Quoted 39 lines
kernel/sched/fair.c +13
-0+ +if (cfs_rq->curr == se) { +/* curr is off-tree: in-place shift, carrying any vprot window. */ +if (protect_slice (se)) + se->vprot -= credit; + se->vruntime -= credit; + se->deadline -= credit; +return ; + } + +/* Queued entities are left unchanged by this helper path. */ +} +/* * sched_yield() is very simple */ kernel/sched/features.h +9
-1@@ -40,6 +40,15 @@ SCHED_FEAT(NEXT_BUDDY, false)*/ SCHED_FEAT (PICK_BUDDY , true) +/* +* Let yield_to_task_fair() credit bounded EEVDF lag to the nominated +* next-buddy so pick_eevdf() honors the hint even when the target has +* negative vlag at some level of its ancestor chain. The credit is bounded +* by a queue-depth-scaled margin within entity_lag()'s legal range, so +* fairness is preserved. + */ +SCHED_FEAT (YIELD_TO_LAG_CREDIT , true) +/* * Consider buddies to be cache hot, decreases the likeliness of a * cache buddy being migrated away, increases cache locality. -- 2.43.0
PATCH v3 2/10 sched/fair: Credit a persistent, queue-depth-scaled vlag margin
In reply to Wanpeng LiFrom: Wanpeng Li <[email protected]>
Crediting only up to vlag = 0 makes the buddy eligible for a single pick_eevdf() pass. The next update_curr() can push it below zero again before PICK_BUDDY consumes the hint.
Credit to a bounded positive-vlag margin instead, so the buddy stays eligible across several scheduling decisions. Scale the margin with runqueue depth, because deeper runqueues dilute eligibility faster, and clamp it to entity_lag()'s legal positive-lag bound to preserve EEVDF fairness.
The helper is not called in this change; there is no functional change.
--- kernel/sched/fair.c | 58 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 47 insertions(+), 11 deletions(-)
kernel/sched/fair.c +47 -11
@@ -9342,21 +9342,45 @@ static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t }/* -* eevdf_credit_entity_vlag - credit a nominated next-buddy to eligibility +* Positive-vlag target margin for a credited buddy, scaled by runqueue +* depth so it stays eligible across several picks. The caller clamps it to +* entity_lag()'s legal bound, so EEVDF fairness is preserved. +*/ +static u64 __maybe_unused +eevdf_persistent_margin (struct cfs_rq *cfs_rq,struct sched_entity *se) +{ +u64 base = sysctl_sched_base_slice; +unsigned int n = cfs_rq->h_nr_queued; +u64 raw; + +if (n <=4 ) + raw = base *4 ; +else if (n <=8 ) + raw = base *6 ; +else if (n <=16 ) + raw = base *8 ; +else + raw = base *12 ; + +return calc_delta_fair (raw, se); +} + +/* +* eevdf_credit_entity_vlag - credit bounded vlag to a nominated next-buddy * * Advance @se (already nominated by set_next_buddy(), so cfs_rq->next == se) -* just enough negative vlag to reach the eligibility boundary (vlag = 0) so -* pick_eevdf()'s PICK_BUDDY branch returns it. cfs_rq->curr is shifted in -* place (off-tree, carrying any vprot window). Queued entities are left -* unchanged. +* to a bounded positive-vlag margin so pick_eevdf()'s PICK_BUDDY branch +* keeps returning it across several picks, without exceeding entity_lag()'s +* legal bound. cfs_rq->curr is shifted in place (off-tree, carrying any +* vprot window). Queued entities are left unchanged. * -* Idempotent: a no-op once @se is already eligible. Caller must hold +* Idempotent once @se holds the margin. Caller must hold * rq_of(cfs_rq)->lock with rq_clock up to date. */ static void __maybe_unused eevdf_credit_entity_vlag (struct cfs_rq *cfs_rq,struct sched_entity *se) { -u64 avruntime, credit; +u64 avruntime, credit, want, margin, max_slice, lag_limit;s64 vlag;/* Callers gate this helper with YIELD_TO_LAG_CREDIT. */
@@ -9371,11 +9395,23 @@ eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se) avruntime =avg_vruntime (cfs_rq); vlag =entity_lag (cfs_rq, se, avruntime); -/* Already eligible: nothing to do. */ -if (vlag >=0 ) -return ; +/* Clamp the margin to entity_lag()'s bound so place_entity() keeps it. */ + max_slice =cfs_rq_max_slice (cfs_rq) +TICK_NSEC ; + lag_limit =calc_delta_fair (max_slice, se); + margin =eevdf_persistent_margin (cfs_rq, se); +if (lag_limit && margin > lag_limit) + margin = lag_limit; +if (vlag >=0 ) { +if ((u64 )vlag >= margin) +return ; + want = margin - (u64 )vlag; + }else { + want = margin + (u64)(-vlag); + } - credit = (u64)(-vlag); + credit = want; +if (!credit) +return ;if (cfs_rq->curr == se) {/* curr is off-tree: in-place shift, carrying any vprot window. */
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v3 02/10] sched/fair: Credit a persistent, queue-depth-scaled vlag margin
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] __enqueue_entity() fails to initialize se->max_slice before adding the entity to the augmented RB-tree, poisoning the tree's max_slice. --
commit c9a9a8b18145d7a6a58148e991409677da519193
Author: Wanpeng Li <[email protected]>
sched/fair: Credit a persistent, queue-depth-scaled vlag margin
This commit replaces the credit-to-zero buddy system with a bounded positive-vlag margin. The margin scales with runqueue depth to preserve the buddy's eligibility across multiple scheduling decisions without exceeding EEVDF fairness bounds.
Quoted 4 lines
kernel/sched/fair.c +0
-0
[ ... ]
kernel/sched/fair.c eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 9399
Quoted 9 lines
kernel/sched/fair.c +2
-3@@ -9371,11 +9395,23 @@ eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se) avruntime =avg_vruntime (cfs_rq); vlag =entity_lag (cfs_rq, se, avruntime); -/* Already eligible: nothing to do. */ -if (vlag >=0 ) -return ; +/* Clamp the margin to entity_lag()'s bound so place_entity() keeps it. */ + max_slice =cfs_rq_max_slice (cfs_rq) +TICK_NSEC ;
[Severity: High]
This isn't a bug introduced by this patch, but does the tree's max_slice contain poisoned data here?
When an entity is dequeued, rb_erase_augmented_cached() removes the node but does not clear se->max_slice. When enqueued again, __enqueue_entity() initializes se->min_vruntime and se->min_slice, but misses se->max_slice:
kernel/sched/fair.c:__enqueue_entity() {
sum_w_vruntime_add(cfs_rq, se);
se->min_vruntime = se->vruntime;
se->min_slice = se->slice;
// Missing se->max_slice = se->slice; here?
rb_add_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline,
__entity_less, &min_vruntime_cb);
}Won't the new leaf node propagate a potentially large, stale max_slice from its previous subtree up to the root? This would artificially inflate cfs_rq_max_slice() and corrupt the lag_limit calculated here, potentially allowing tasks to accumulate massive lag while sleeping and breaking EEVDF fairness bounds.
Quoted 4 lines
kernel/sched/fair.c +4
-0+ lag_limit =calc_delta_fair (max_slice, se); + margin =eevdf_persistent_margin (cfs_rq, se); +if (lag_limit && margin > lag_limit) + margin = lag_limit;
[ ... ]
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
K Prateek Nayak
reviewerRe: [PATCH v3 02/10] sched/fair: Credit a persistent, queue-depth-scaled vlag margin
In reply to Wanpeng LiHello Wanpeng,
kernel/sched/fair.c eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 9407
Quoted 13 lines · Wanpeng Li
kernel/sched/fair.c +10
-3-/* Already eligible: nothing to do. */ -if (vlag >=0 ) -return ; +/* Clamp the margin to entity_lag()'s bound so place_entity() keeps it. */ + max_slice =cfs_rq_max_slice (cfs_rq) +TICK_NSEC ; + lag_limit =calc_delta_fair (max_slice, se); + margin =eevdf_persistent_margin (cfs_rq, se); +if (lag_limit && margin > lag_limit) + margin = lag_limit; +if (vlag >=0 ) { +if ((u64 )vlag >= margin) +return ; + want = margin - (u64 )vlag;
And we are making an eligible task more eligible but *where* is this credit coming from? Someone should be paying the price right?
Just adding an arbitrary credit and then moving the task doing the yield_to() 0-lag point is simply not fair IMO.
Quoted 3 lines
kernel/sched/fair.c +3
-0+ }else { + want = margin + (u64)(-vlag); + }
Signature · 2 lines
Thanks and Regards, Prateek
Vincent Guittot
maintainerRe: [PATCH v3 02/10] sched/fair: Credit a persistent, queue-depth-scaled vlag margin
Quoted 18 lines · K Prateek Nayak
Hello Wanpeng,
kernel/sched/fair.c eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 9407
Quoted 13 lines · Wanpeng Li
kernel/sched/fair.c +10
-3-/* Already eligible: nothing to do. */ -if (vlag >=0 ) -return ; +/* Clamp the margin to entity_lag()'s bound so place_entity() keeps it. */ + max_slice =cfs_rq_max_slice (cfs_rq) +TICK_NSEC ; + lag_limit =calc_delta_fair (max_slice, se); + margin =eevdf_persistent_margin (cfs_rq, se); +if (lag_limit && margin > lag_limit) + margin = lag_limit; +if (vlag >=0 ) { +if ((u64 )vlag >= margin) +return ; + want = margin - (u64 )vlag;
Quoted 5 lines
And we are making an eligible task more eligible but *where* is this credit coming from? Someone should be paying the price right?
Just adding an arbitrary credit and then moving the task doing the
yield_to()0-lag point is simply not fair IMO.
I agree, you can't arbitrarily add some lag to an entity just because you want it to run.
Quoted 9 lines
kernel/sched/fair.c eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 9410
Quoted 4 lines
kernel/sched/fair.c +3
-0+ }else { + want = margin + (u64)(-vlag); + }--
Thanks and Regards,
Prateek
PATCH v3 3/10 sched/fair: Credit queued next-buddy via canonical requeue
In reply to Wanpeng LiFrom: Wanpeng Li <[email protected]>
eevdf_credit_entity_vlag() handles cfs_rq->curr, which is off-tree. Extend it to a queued entity, which is the common case when yield_to() targets a task that is runnable but not currently executing.
A queued entity's key contributes to cfs_rq->sum_w_vruntime, so its vruntime cannot be shifted in place. Use the canonical place_entity()- paired requeue pattern: snapshot vlag, make the deadline relative to se->vruntime, dequeue the entity, apply the credit via place_entity(), and enqueue it again. This keeps sum_w_vruntime consistent with entity_key(), preserves the slice, and shifts the deadline by the same amount as the off-tree current-entity path.
The helper is not called in this change; there is no functional change.
--- kernel/sched/fair.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-)
kernel/sched/fair.c +19 -2
@@ -9372,7 +9372,9 @@ eevdf_persistent_margin(struct cfs_rq *cfs_rq, struct sched_entity *se)* to a bounded positive-vlag margin so pick_eevdf()'s PICK_BUDDY branch * keeps returning it across several picks, without exceeding entity_lag()'s * legal bound. cfs_rq->curr is shifted in place (off-tree, carrying any -* vprot window). Queued entities are left unchanged. +* vprot window); a queued entity uses the canonical +* place_entity()-paired requeue, keeping sum_w_vruntime consistent with +* entity_key(). * * Idempotent once @se holds the margin. Caller must hold * rq_of(cfs_rq)->lock with rq_clock up to date.
@@ -9422,7 +9424,22 @@ eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se)return ; } -/* Queued entities are left unchanged by this helper path. */ +/* +* Canonical place_entity()-paired requeue: see dequeue_entity() and +* requeue_delayed_entity(). place_entity() restores the deadline via +* "se->deadline += se->vruntime", so make the deadline relative to +* se->vruntime here (not avg_vruntime). This preserves the slice and +* shifts the deadline by exactly -credit, mirroring the curr branch +* and keeping deadline > vruntime. +*/ + se->vlag = vlag + (s64 )credit; + se->deadline -= se->vruntime; + se->rel_deadline =1 ; + cfs_rq->nr_queued--; +__dequeue_entity (cfs_rq, se); +place_entity (cfs_rq, se,0 ); +__enqueue_entity (cfs_rq, se); + cfs_rq->nr_queued++; }/*
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v3 03/10] sched/fair: Credit queued next-buddy via canonical requeue
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
New issues:
- [High] Unconditional setting of se->rel_deadline = 1 ignores sched_feat(PLACE_REL_DEADLINE), leaking relative deadline flags and corrupting absolute deadlines on reweighting.
Pre-existing issues:
- [High] __enqueue_entity fails to reset se->max_slice = se->slice, causing stale subtree maximums to incorrectly inflate the augmented rbtree. --
commit 61176c863b2b0bee3f465f1148e81251b1cf5bda
Author: Wanpeng Li <[email protected]>
sched/fair: Credit queued next-buddy via canonical requeue
This commit extends eevdf_credit_entity_vlag() to handle a queued entity. It uses the canonical requeue pattern to keep sum_w_vruntime consistent with entity_key(), preserving the slice and shifting the deadline.
Quoted 3 lines
kernel/sched/fair.c +0
-0
[ ... ]
Quoted 6 lines
kernel/sched/fair.c +1
-1@@ -9422,7 +9424,22 @@ eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se)return ; } -/* Queued entities are left unchanged by this helper path. */ +/*
kernel/sched/fair.c eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 9437
Quoted 10 lines
kernel/sched/fair.c +10
-0+* Canonical place_entity()-paired requeue: see dequeue_entity() and +* requeue_delayed_entity(). place_entity() restores the deadline via +* "se->deadline += se->vruntime", so make the deadline relative to +* se->vruntime here (not avg_vruntime). This preserves the slice and +* shifts the deadline by exactly -credit, mirroring the curr branch +* and keeping deadline > vruntime. +*/ + se->vlag = vlag + (s64 )credit; + se->deadline -= se->vruntime; + se->rel_deadline =1 ;
[Severity: High]
Does setting the relative deadline flag unconditionally here cause an issue if the PLACE_REL_DEADLINE feature is disabled?
Looking at place_entity(), the flag is only cleared if the feature is enabled:
place_entity() {
...
if (sched_feat(PLACE_REL_DEADLINE) && se->rel_deadline) {
se->deadline += se->vruntime;
se->rel_deadline = 0;
return;
}
...
se->deadline = se->vruntime + vslice;
}If the feature is disabled, place_entity() computes an absolute deadline but leaves se->rel_deadline set to 1. If the task then sleeps and its nice value changes, rescale_entity() checks se->rel_deadline and will incorrectly apply relative scaling to the absolute deadline:
rescale_entity() {
...
if (se->rel_deadline)
se->deadline = div64_long(se->deadline * old_weight, weight);
...
}Could this arithmetic overflow and corrupt the deadline, potentially breaking the rbtree priority ordering?
kernel/sched/fair.c eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 9441
Quoted 4 lines
kernel/sched/fair.c +4
-0+ cfs_rq->nr_queued--; +__dequeue_entity (cfs_rq, se); +place_entity (cfs_rq, se,0 ); +__enqueue_entity (cfs_rq, se);
[Severity: High]
This is a pre-existing issue, but does the call to __enqueue_entity() expose a stale max_slice issue in the augmented rbtree?
When an entity is inserted as a leaf node, __enqueue_entity() resets the min_vruntime and min_slice, but appears to leave max_slice untouched:
__enqueue_entity() {
sum_w_vruntime_add(cfs_rq, se);
se->min_vruntime = se->vruntime;
se->min_slice = se->slice;
rb_add_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline,
__entity_less, &min_vruntime_cb);
}If the task previously had a large max_slice (for instance, when it was a parent of tasks with large slices), does that stale max_slice propagate up the tree during rb_add_augmented_cached()?
Could this artificially inflate cfs_rq_max_slice(cfs_rq) and circumvent the lag clamping bounds in entity_lag(), allowing tasks to hoard excessive lag?
Quoted 2 lines
kernel/sched/fair.c +1
-0+ cfs_rq->nr_queued++; }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Vincent Guittot
maintainerRe: [PATCH v3 03/10] sched/fair: Credit queued next-buddy via canonical requeue
In reply to Wanpeng LiQuoted 15 lines · Wanpeng Li
From: Wanpeng Li <[email protected]>
eevdf_credit_entity_vlag()handlescfs_rq->curr, which is off-tree. Extend it to a queued entity, which is the common case whenyield_to()targets a task that is runnable but not currently executing.A queued entity's key contributes to
cfs_rq->sum_w_vruntime, so its vruntime cannot be shifted in place. Use the canonical place_entity()- paired requeue pattern: snapshot vlag, make the deadline relative tose->vruntime, dequeue the entity, apply the credit viaplace_entity(), and enqueue it again. This keepssum_w_vruntimeconsistent withentity_key(), preserves the slice, and shifts the deadline by the same amount as the off-tree current-entity path.
Quoted 1 line
The helper is not called in this change; there is no functional change.
Merge this patch with the 2 previous ones. I don't see the benefit of split it in 3 different patches
Quoted 48 lines
Signed-off-by Wanpeng Li <[email protected]>--- kernel/sched/fair.c | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-)kernel/sched/fair.c +19
-3@@ -9372,7 +9372,9 @@ eevdf_persistent_margin(struct cfs_rq *cfs_rq, struct sched_entity *se)* to a bounded positive-vlag margin so pick_eevdf()'s PICK_BUDDY branch * keeps returning it across several picks, without exceeding entity_lag()'s * legal bound. cfs_rq->curr is shifted in place (off-tree, carrying any -* vprot window). Queued entities are left unchanged. +* vprot window); a queued entity uses the canonical +* place_entity()-paired requeue, keeping sum_w_vruntime consistent with +* entity_key(). * * Idempotent once @se holds the margin. Caller must hold * rq_of(cfs_rq)->lock with rq_clock up to date. @@ -9422,7 +9424,22 @@ eevdf_credit_entity_vlag(struct cfs_rq *cfs_rq, struct sched_entity *se)return ; } -/* Queued entities are left unchanged by this helper path. */ +/* +* Canonical place_entity()-paired requeue: see dequeue_entity() and +* requeue_delayed_entity(). place_entity() restores the deadline via +* "se->deadline += se->vruntime", so make the deadline relative to +* se->vruntime here (not avg_vruntime). This preserves the slice and +* shifts the deadline by exactly -credit, mirroring the curr branch +* and keeping deadline > vruntime. +*/ + se->vlag = vlag + (s64 )credit; + se->deadline -= se->vruntime; + se->rel_deadline =1 ; + cfs_rq->nr_queued--; +__dequeue_entity (cfs_rq, se); +place_entity (cfs_rq, se,0 ); +__enqueue_entity (cfs_rq, se); + cfs_rq->nr_queued++; }/* -- 2.43.0
PATCH v3 4/10 sched/fair: Credit nominated next-buddy in yield_to_task_fair()
In reply to Wanpeng LiFrom: Wanpeng Li <[email protected]>
After set_next_buddy() nominates the yield_to() target at every level of its sched_entity hierarchy, walk that same hierarchy and credit bounded EEVDF lag to each not-yet-eligible entity. This allows pick_eevdf()'s PICK_BUDDY path to select the nominated target instead of dropping the hint at the first ineligible group entity.
Gate the walk with YIELD_TO_LAG_CREDIT. With the feature disabled, yield_to_task_fair() keeps the existing forfeit-based behavior. yield_to() holds both rq locks via double_rq_lock(), so touching the target task's cfs_rqs, including remote cfs_rqs, is safe. Stop the walk where set_next_buddy() stopped, and skip delayed or throttled entities.
Refresh the target rq clock when it differs from the local rq so the per-level update_curr() calls observe current rq_clock values. The local rq still uses the existing yield_task_fair() path in this change.
--- kernel/sched/fair.c | 44 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 41 insertions(+), 3 deletions(-)
kernel/sched/fair.c +41 -3
@@ -9346,8 +9346,8 @@ static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct t* depth so it stays eligible across several picks. The caller clamps it to * entity_lag()'s legal bound, so EEVDF fairness is preserved. */ -static u64 __maybe_unused -eevdf_persistent_margin (struct cfs_rq *cfs_rq,struct sched_entity *se) +static u64 eevdf_persistent_margin (struct cfs_rq *cfs_rq, +struct sched_entity *se) {u64 base = sysctl_sched_base_slice;unsigned int n = cfs_rq->h_nr_queued;
@@ -9379,7 +9379,7 @@ eevdf_persistent_margin(struct cfs_rq *cfs_rq, struct sched_entity *se)* Idempotent once @se holds the margin. Caller must hold * rq_of(cfs_rq)->lock with rq_clock up to date. */ -static void __maybe_unused +static void eevdf_credit_entity_vlag (struct cfs_rq *cfs_rq,struct sched_entity *se) {u64 avruntime, credit, want, margin, max_slice, lag_limit;
@@ -9488,6 +9488,7 @@ static void yield_task_fair(struct rq *rq)static bool yield_to_task_fair (struct rq *rq,struct task_struct *p) {struct sched_entity *se = &p->se; +struct rq *p_rq =task_rq (p);/* !se->on_rq also covers throttled task */ if (!se->on_rq)
@@ -9496,6 +9497,43 @@ static bool yield_to_task_fair(struct rq *rq, struct task_struct *p)/* Tell the scheduler that we'd really like se to run next. */ set_next_buddy (se); +/* Without lag credit, keep the existing forfeit-based yield. */ +if (!sched_feat (YIELD_TO_LAG_CREDIT )) { +yield_task_fair (rq); +return true; + } + +/* +* Walk the ancestor chain set_next_buddy() just nominated and credit +* bounded lag to each not-yet-eligible level so pick_eevdf() returns +* it. yield_to() holds both rq locks via double_rq_lock(), so touching +* p's cfs_rqs (possibly on another CPU) is safe; the primitive is +* idempotent, so no rate limiting is needed. +* +* Only refresh p_rq's clock when it differs from the local rq. A +* remote p_rq must be refreshed so the per-level update_curr() is +* accurate. In the same-rq case we skip it: the credit is a +* best-effort hint and the rq clock is recent enough, while the +* trailing yield_task_fair() would otherwise make this a second +* update_rq_clock() on the same rq and trip +* SCHED_WARN_ON(WARN_DOUBLE_CLOCK). +*/ +if (rq != p_rq) +update_rq_clock (p_rq); + +for_each_sched_entity (se) { +struct cfs_rq *cfs_rq =cfs_rq_of (se); + +if (cfs_rq->next != se) +break ; +if (se->sched_delayed) +break ; +if (throttled_hierarchy (cfs_rq)) +break ; + +eevdf_credit_entity_vlag (cfs_rq, se); + } +yield_task_fair (rq);return true;
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v3 04/10] sched/fair: Credit nominated next-buddy in yield_to_task_fair()
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Skipping `update_rq_clock()` for the same-runqueue case in `yield_to_task_fair()` unconditionally…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Skipping update_rq_clock() for the same-runqueue case in yield_to_task_fair() unconditionally triggers a WARN_ON_ONCE in assert_clock_updated() via update_curr(). --
commit 18fe3f24f01f55a4b16680e6b460846c6d5e6260
Author: Wanpeng Li <[email protected]>
sched/fair: Credit nominated next-buddy in yield_to_task_fair()
This patch credits bounded EEVDF lag to nominated targets during yield_to(). It walks the sched_entity hierarchy to provide the credit, avoiding forfeiture of the yield hint when a group entity isn't yet eligible.
Quoted 4 lines
kernel/sched/fair.c +0
-0
[ ... ]
Quoted 29 lines
kernel/sched/fair.c +25
-0@@ -9496,6 +9497,43 @@ static bool yield_to_task_fair(struct rq *rq, struct task_struct *p)/* Tell the scheduler that we'd really like se to run next. */ set_next_buddy (se); +/* Without lag credit, keep the existing forfeit-based yield. */ +if (!sched_feat (YIELD_TO_LAG_CREDIT )) { +yield_task_fair (rq); +return true; + } + +/* +* Walk the ancestor chain set_next_buddy() just nominated and credit +* bounded lag to each not-yet-eligible level so pick_eevdf() returns +* it. yield_to() holds both rq locks via double_rq_lock(), so touching +* p's cfs_rqs (possibly on another CPU) is safe; the primitive is +* idempotent, so no rate limiting is needed. +* +* Only refresh p_rq's clock when it differs from the local rq. A +* remote p_rq must be refreshed so the per-level update_curr() is +* accurate. In the same-rq case we skip it: the credit is a +* best-effort hint and the rq clock is recent enough, while the +* trailing yield_task_fair() would otherwise make this a second +* update_rq_clock() on the same rq and trip +* SCHED_WARN_ON(WARN_DOUBLE_CLOCK). +*/ +if (rq != p_rq) +update_rq_clock (p_rq); + +for_each_sched_entity (se) {
kernel/sched/fair.c static bool yield_to_task_fair(struct rq *rq, struct task_struct *p) · line 9534
Quoted 10 lines
kernel/sched/fair.c +10
-0+struct cfs_rq *cfs_rq =cfs_rq_of (se); + +if (cfs_rq->next != se) +break ; +if (se->sched_delayed) +break ; +if (throttled_hierarchy (cfs_rq)) +break ; + +eevdf_credit_entity_vlag (cfs_rq, se);
[Severity: High]
Does skipping update_rq_clock() here trigger a WARN_ON_ONCE() when the target task is on the same runqueue?
Since yield_to() acquires locks via double_rq_lock(), the clock update flag is cleared. By skipping update_rq_clock(p_rq) for local tasks, the call to eevdf_credit_entity_vlag() will invoke update_curr() with an un-updated clock:
eevdf_credit_entity_vlag()
update_curr()
update_se()
rq_clock_task()
assert_clock_updated() -> WARN_ON_ONCE(rq->clock_update_flags < RQCF_ACT_SKIP)Can this reliably trigger a warning when yield_to() targets a task on the same CPU (for instance, via KVM vCPU yielding)?
Quoted 5 lines
kernel/sched/fair.c +1
-0+ }yield_task_fair (rq);return true;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
PATCH v3 5/10 sched/fair: Force a local resched on yield_to() so the buddy is picked
In reply to Wanpeng LiFrom: Wanpeng Li <[email protected]>
Lag credit makes the target eligible for PICK_BUDDY, but yield_to() does not by itself force the caller off the CPU. An active RUN_TO_PARITY protect_slice() on the local yielder can therefore keep pick_eevdf() returning the yielder instead of the credited buddy.
Add yield_to_local_force_resched() for the lag-credit path. It applies the existing leaf forfeit, cancels slice protection along the yielder's sched_entity hierarchy, and calls resched_curr() on the local rq. cancel_protect_slice() is already used by PREEMPT_WAKEUP_SHORT and does not modify vruntime.
Rate-limit only the forced preemption (cancel_protect_slice() plus resched_curr()) to once per 6ms per rq. The lag credit itself remains unthrottled so each directed yield refreshes the scheduling hint, while compute-bound guests avoid excessive forced preemption on PLE-heavy spin loops.
Dbench (filesystem metadata operations) on 16-vCPU guests under host CPU overcommit, throughput improvement from the scheduler side alone:
2 VMs: +6.65% 3 VMs: +4.80% 4 VMs: +7.59%
--- kernel/sched/fair.c | 113 +++++++++++++++++++++++++++++++++++++------ kernel/sched/sched.h | 10 ++++ 2 files changed, 108 insertions(+), 15 deletions(-)
kernel/sched/fair.c +98 -15
@@ -9485,6 +9485,83 @@ static void yield_task_fair(struct rq *rq) } } +/* +* Rate-limit the forced local reschedule on the yield_to() lag-credit path +* to at most once per 6ms per rq. +* +* Lag credit is intentionally not rate-limited: a contended lock holder +* should be credited on every directed yield to keep the scheduling hint +* effective. Only the forced preemption needs bounding, as cancelling +* RUN_TO_PARITY protection and calling resched_curr() on every PLE-driven +* yield_to() can cause excessive preemption on compute-bound guests. +* +* Returns true if the caller should skip forcing a reschedule because a +* recent one already happened on this rq; the credit just applied still +* persists, so the buddy can be selected at the next scheduling point. +* +* Called under rq->lock with rq_clock up to date. yield_to_task_fair() +* updates the clock before walking the hierarchy because yield_to() takes +* the rq locks without updating them. +*/ +static bool yield_to_force_resched_rate_limit (struct rq *rq) +{ +u64 now =rq_clock (rq); +u64 last = rq->yield_to_force_resched_last_ns; + +if (last && (now - last) <=6 *NSEC_PER_MSEC ) +return true; + + rq->yield_to_force_resched_last_ns = now; +return false; +} + +/* +* Forfeit the local yielder, cancel its RUN_TO_PARITY slice protection +* along the whole sched_entity chain, and force a reschedule. +* +* yield_to() does not reschedule the caller, and an active protect_slice() +* at any level can keep pick_eevdf() returning the yielder instead of the +* credited buddy. cancel_protect_slice() is EEVDF-native (also used by +* PREEMPT_WAKEUP_SHORT) and does not touch vruntime. Caller holds the +* local rq lock via yield_to()'s double_rq_lock(). +* +* Only the forced preemption here is rate-limited (to once per 6ms per rq); +* the lag credit applied by the caller runs on every yield_to(). When +* throttled, the credited buddy can still be selected at the next natural +* scheduling point without tearing down slice protection and forcing an +* immediate switch. +*/ +static void yield_to_local_force_resched (struct rq *rq) +{ +struct sched_entity *yse = &rq->donor->se; + +yield_task_fair (rq); + +/* +* If the yielder is the only runnable task on this rq there is nothing +* for resched_curr() to switch to: any credited buddy is on a remote rq +* in this cross-rq case, where yield_to() already issued resched_curr() +* on the target's rq. Skip the forced reschedule: it would be a no-op +* and an unnecessary preemption of an unrelated local task. +* yield_task_fair() also returns early here without updating rq_clock. +*/ +if (rq->nr_running <=1 ) +return ; + +/* +* Rate-limit the forced preemption (cancel_protect_slice + resched_curr) +* to once per 6ms per rq. rq's clock was refreshed by the caller before +* the credit walk, so rq_clock(rq) read here is current. +*/ +if (yield_to_force_resched_rate_limit (rq)) +return ; + +for_each_sched_entity (yse) +cancel_protect_slice (yse); + +resched_curr (rq); +} +static bool yield_to_task_fair (struct rq *rq,struct task_struct *p) {struct sched_entity *se = &p->se;
@@ -9504,21 +9581,22 @@ static bool yield_to_task_fair(struct rq *rq, struct task_struct *p) }/* -* Walk the ancestor chain set_next_buddy() just nominated and credit -* bounded lag to each not-yet-eligible level so pick_eevdf() returns -* it. yield_to() holds both rq locks via double_rq_lock(), so touching -* p's cfs_rqs (possibly on another CPU) is safe; the primitive is -* idempotent, so no rate limiting is needed. +* Walk the ancestor chain nominated by set_next_buddy() and credit +* bounded lag to each not-yet-eligible level, so pick_eevdf() can +* honor the buddy hint. Lag credit runs on every directed yield; only +* the forced preemption in yield_to_local_force_resched() is +* rate-limited. yield_to() holds both rq locks via double_rq_lock(), +* so touching p's cfs_rqs (possibly on another CPU) is safe. * -* Only refresh p_rq's clock when it differs from the local rq. A -* remote p_rq must be refreshed so the per-level update_curr() is -* accurate. In the same-rq case we skip it: the credit is a -* best-effort hint and the rq clock is recent enough, while the -* trailing yield_task_fair() would otherwise make this a second -* update_rq_clock() on the same rq and trip -* SCHED_WARN_ON(WARN_DOUBLE_CLOCK). -*/ -if (rq != p_rq) +* Refresh the local rq clock first: yield_to() took the locks without +* updating any clock and the per-level update_curr() below reads +* rq_clock; assert_clock_updated() (default-on, no sched_feat gate) +* fires otherwise. For a remote p_rq refresh it too; in the same-rq +* case the refresh above already covers it (a redundant update is only +* warned about under the default-off WARN_DOUBLE_CLOCK). +*/ +update_rq_clock (rq); +if (p_rq != rq)update_rq_clock (p_rq);for_each_sched_entity (se) {
@@ -9534,7 +9612,12 @@ static bool yield_to_task_fair(struct rq *rq, struct task_struct *p)eevdf_credit_entity_vlag (cfs_rq, se); } -yield_task_fair (rq); +/* +* Force the local CPU to reschedule so the credited buddy can be +* selected instead of the protected yielder; +* yield_to_local_force_resched() also does the leaf forfeit. +*/ +yield_to_local_force_resched (rq);return true; }
kernel/sched/sched.h +10 -0
@@ -1316,6 +1316,16 @@ struct rq {unsigned int ttwu_local;#endif +/* +* Last rq_clock at which the yield_to() lag-credit path forced a local +* reschedule on this rq. Used to rate-limit only the forced preemption +* (cancel_protect_slice + resched_curr) to at most once per 6ms per rq, +* preventing excessive forced preemption on PLE-heavy guests. The lag +* credit itself is not rate-limited. Functional state, not a statistic, +* so kept outside CONFIG_SCHEDSTATS. +*/ +u64 yield_to_force_resched_last_ns; +#ifdef CONFIG_CPU_IDLE /* Must be inspected within a RCU lock section */ struct cpuidle_state *idle_state;
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v3 05/10] sched/fair: Force a local resched on yield_to() so the buddy is picked
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Double clock update triggers WARN_DOUBLE_CLOCK on the local runqueue in yield_to_task_fair(). --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Double clock update triggers WARN_DOUBLE_CLOCK on the local runqueue in yield_to_task_fair(). --
commit fb0067f31069bbf43450f383db09dc5489957b42
Author: Wanpeng Li <[email protected]>
sched/fair: Force a local resched on yield_to() so the buddy is picked
This commit modifies yield_to_task_fair() to force a local reschedule when applying lag credit, cancelling slice protection and calling resched_curr() on the local runqueue. The forced preemption is rate-limited to at most once per 6ms per rq.
Quoted 4 lines
kernel/sched/fair.c +0
-0
[ ... ]
Quoted 1 line
kernel/sched/fair.c +0
-0@@ -9485,6 +9485,83 @@ static void yield_task_fair(struct rq *rq)
[ ... ]
Quoted 5 lines
kernel/sched/fair.c +5
-0+static void yield_to_local_force_resched (struct rq *rq) +{ +struct sched_entity *yse = &rq->donor->se; + +yield_task_fair (rq);
[ ... ]
Quoted 1 line
kernel/sched/fair.c +0
-0@@ -9504,21 +9581,22 @@ static bool yield_to_task_fair(struct rq *rq, struct task_struct *p)
[ ... ]
Quoted 6 lines
kernel/sched/fair.c +2
-1-if (rq != p_rq) +update_rq_clock (rq); +if (p_rq != rq)update_rq_clock (p_rq);for_each_sched_entity (se) {
[ ... ]
kernel/sched/fair.c static bool yield_to_task_fair(struct rq *rq, struct task_struct *p) · line 9620
Quoted 7 lines
kernel/sched/fair.c +6
-1-yield_task_fair (rq); +/* +* Force the local CPU to reschedule so the credited buddy can be +* selected instead of the protected yielder; +* yield_to_local_force_resched() also does the leaf forfeit. +*/ +yield_to_local_force_resched (rq);
[Severity: Medium]
Will this sequence trigger WARN_DOUBLE_CLOCK on the local runqueue?
The patch comments note this redundant update is only warned about under the default-off WARN_DOUBLE_CLOCK.
However, update_rq_clock(rq) is called in yield_to_task_fair() setting RQCF_UPDATED. Then yield_to_local_force_resched(rq) calls yield_task_fair() which calls update_rq_clock(rq) a second time.
Since the flag isn't cleared between these calls, this will trigger:WARN_ON_ONCE(rq->clock_update_flags & RQCF_UPDATED)
Is it okay to knowingly leave this double clock update, or could it cause spurious stack traces for developers actively debugging scheduler clock issues?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
PATCH v3 6/10 KVM: x86: Add IPI tracking infrastructure for directed yield
In reply to Wanpeng LiFrom: Wanpeng Li <[email protected]>
On overcommitted hosts, a vCPU spinning on an IPI response is difficult to distinguish from a vCPU spinning on a lock. kvm_vcpu_on_spin() can therefore yield to an unrelated vCPU based only on coarse preemption state.
Add per-vCPU IPI tracking for directed yield. struct kvm_vcpu_arch now records the last sender and receiver vCPU indexes, the vector, a pending flag, and a monotonic timestamp. Add helpers to record a send, query whether a vCPU is the recent IPI receiver of another vCPU, and clear or reset the context. Accesses use READ_ONCE() and WRITE_ONCE() because the state is only a best-effort scheduling hint.
Add module parameters to enable tracking and to control the recency window. Provide a weak generic kvm_vcpu_is_ipi_receiver() stub so non-x86 builds keep the existing behavior. The state is reset on vCPU create and destroy, and cleared on INIT.
This adds only state and helpers; directed-yield candidate selection is unchanged.
--- arch/x86/include/asm/kvm_host.h | 19 ++++++ arch/x86/kvm/lapic.c | 102 ++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 3 + arch/x86/kvm/x86.h | 8 +++ include/linux/kvm_host.h | 8 +++ virt/kvm/kvm_main.c | 6 ++ 6 files changed, 146 insertions(+)
arch/x86/include/asm/kvm_host.h +19 -0
@@ -1065,6 +1065,25 @@ struct kvm_vcpu_arch {int pending_external_vector;int highest_stale_pending_ioapic_eoi; +/* +* IPI tracking for directed-yield optimization. +* +* Populated by kvm_track_ipi_communication() when a unicast fixed +* IPI is delivered, and queried by kvm_vcpu_is_ipi_receiver() from +* kvm_vcpu_on_spin() to prefer the confirmed IPI target before +* generic preempted-lock-holder heuristics. +* +* All accesses are lockless READ_ONCE/WRITE_ONCE; best-effort by +* design (see comment on kvm_vcpu_is_good_yield_candidate()). +*/ +struct { +int last_ipi_sender;/* vCPU idx of last IPI sender */ +int last_ipi_receiver;/* vCPU idx of last IPI target */ +u8 vector;/* vector of the pending IPI */ +bool pending_ipi;/* awaiting IPI response */ +u64 ipi_time_ns;/* mono timestamp of IPI send */ + } ipi_context; +/* be preempted when it's in kernel-mode(cpl=0) */ bool preempted_in_kernel;
arch/x86/kvm/lapic.c +102 -0
114 lines not on this page · raw message
arch/x86/kvm/x86.c +3 -0
@@ -12907,6 +12907,7 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) goto free_guest_fpu;kvm_xen_init_vcpu (vcpu); +kvm_vcpu_reset_ipi_context (vcpu);vcpu_load (vcpu);kvm_vcpu_after_set_cpuid (vcpu);kvm_set_tsc_khz (vcpu, vcpu->kvm->arch.default_tsc_khz);
@@ -12974,6 +12975,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)kvm_mmu_destroy (vcpu);srcu_read_unlock (&vcpu->kvm->srcu, idx);free_page ((unsigned long )vcpu->arch.pio_data); +kvm_vcpu_reset_ipi_context (vcpu);kvfree (vcpu->arch.cpuid_entries); }
@@ -13050,6 +13052,7 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)kvm_leave_nested (vcpu);kvm_lapic_reset (vcpu, init_event); +kvm_vcpu_clear_ipi_context (vcpu);WARN_ON_ONCE (is_guest_mode (vcpu) ||is_smm (vcpu)); vcpu->arch.hflags =0 ;
arch/x86/kvm/x86.h +8 -0
@@ -475,6 +475,14 @@ int handle_ud(struct kvm_vcpu *vcpu);void kvm_deliver_exception_payload (struct kvm_vcpu *vcpu,struct kvm_queued_exception *ex); +/* IPI tracking helpers for directed-yield optimization (see lapic.c). */ +void kvm_track_ipi_communication (struct kvm_vcpu *sender, +struct kvm_vcpu *receiver,u8 vector); +bool kvm_vcpu_is_ipi_receiver (struct kvm_vcpu *sender, +struct kvm_vcpu *receiver); +void kvm_vcpu_clear_ipi_context (struct kvm_vcpu *vcpu); +void kvm_vcpu_reset_ipi_context (struct kvm_vcpu *vcpu); +int kvm_mtrr_set_msr (struct kvm_vcpu *vcpu,u32 msr,u64 data);int kvm_mtrr_get_msr (struct kvm_vcpu *vcpu,u32 msr,u64 *pdata);void kvm_fixup_and_inject_pf_error (struct kvm_vcpu *vcpu,gva_t gva,u16 error_code);
include/linux/kvm_host.h +8 -0
@@ -1549,6 +1549,14 @@ static inline void kvm_vcpu_kick(struct kvm_vcpu *vcpu)int kvm_vcpu_yield_to (struct kvm_vcpu *target);void kvm_vcpu_on_spin (struct kvm_vcpu *vcpu,bool yield_to_kernel_mode); +/* +* IPI-aware directed-yield hook. Architectures that support IPI +* tracking (currently x86 via arch/x86/kvm/lapic.c) override this; +* the generic __weak stub in virt/kvm/kvm_main.c returns false. +*/ +bool kvm_vcpu_is_ipi_receiver (struct kvm_vcpu *sender, +struct kvm_vcpu *receiver); +void kvm_flush_remote_tlbs (struct kvm *kvm);void kvm_flush_remote_tlbs_range (struct kvm *kvm,gfn_t gfn,u64 nr_pages); void kvm_flush_remote_tlbs_memslot(struct kvm *kvm,
virt/kvm/kvm_main.c +6 -0
@@ -3957,6 +3957,12 @@ bool __weak kvm_arch_dy_has_pending_interrupt(struct kvm_vcpu *vcpu)return false; } +bool __weak kvm_vcpu_is_ipi_receiver (struct kvm_vcpu *sender, +struct kvm_vcpu *receiver) +{ +return false; +} +void kvm_vcpu_on_spin (struct kvm_vcpu *me,bool yield_to_kernel_mode) {int nr_vcpus, start, i, idx, yielded;
Signature · 1 line
2.43.0
PATCH v3 7/10 KVM: x86/lapic: Track unicast fixed IPI delivery
In reply to Wanpeng LiFrom: Wanpeng Li <[email protected]>
Record IPI sender/receiver relationships from both LAPIC delivery paths:
the APIC-map fast path and the slow fallback.
Only record a directed sender->receiver hint when the IPI originates from a vCPU, uses APIC_DM_FIXED delivery, has no destination shorthand, is accepted by exactly one destination vCPU, and is not a self-IPI. The tracking helper also filters the disabled case.
This change only records the relationship; directed-yield candidate selection is unchanged.
--- arch/x86/kvm/lapic.c | 71 ++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 66 insertions(+), 5 deletions(-)
arch/x86/kvm/lapic.c +66 -5
@@ -1326,7 +1326,8 @@ static bool __kvm_irq_delivery_to_apic_fast(struct kvm *kvm, struct kvm_lapic *sstruct kvm_apic_map *map;unsigned long bitmap;struct kvm_lapic **dst =NULL ; -int i; +struct kvm_vcpu *ipi_unique =NULL ; +int i, ipi_targets =0 ;bool ret; *r =-1 ;
@@ -1347,10 +1348,37 @@ static bool __kvm_irq_delivery_to_apic_fast(struct kvm *kvm, struct kvm_lapic *sif (ret) { *r =0 ;for_each_set_bit (i, &bitmap,16 ) { +int delivered; +if (!dst[i])continue ; - *r +=kvm_apic_set_irq (dst[i]->vcpu, irq, rtc_status); + delivered =kvm_apic_set_irq (dst[i]->vcpu, irq, rtc_status); + *r += delivered; +if (delivered >0 ) { + ipi_targets++; + ipi_unique = dst[i]->vcpu; + } } + +/* +* Track unicast fixed IPIs for directed-yield optimization. +* +* Only record when: +* - the IPI originated from a vCPU (LAPIC write, not kernel +* injection): src != NULL; +* - delivery mode is plain fixed: synchronization +* primitives such as spinlocks, TLB flushes and +* smp_call_function() use APIC_DM_FIXED; +* - no shorthand: shorthand encodes broadcasts and self +* which we explicitly do not track; +* - exactly one recipient accepted the interrupt, giving a +* directed sender->receiver relationship. +*/ +if (src && irq->delivery_mode ==APIC_DM_FIXED && + irq->shorthand ==APIC_DEST_NOSHORT && + ipi_targets ==1 && ipi_unique && ipi_unique != src->vcpu) +kvm_track_ipi_communication (src->vcpu, ipi_unique, + irq->vector); }rcu_read_unlock ();
@@ -1443,6 +1471,13 @@ int __kvm_irq_delivery_to_apic(struct kvm *kvm, struct kvm_lapic *src,struct kvm_vcpu *vcpu, *lowest =NULL ;unsigned long i, dest_vcpu_bitmap[BITS_TO_LONGS (KVM_MAX_VCPUS )];unsigned int dest_vcpus =0 ; +/* +* Track unicast fixed IPI for directed-yield optimization in this +* slow fallback path (APIC map miss). See the fast-path equivalent in +* __kvm_irq_delivery_to_apic_fast() for the full filtering rationale. +*/ +struct kvm_vcpu *ipi_unique =NULL ; +int ipi_targets =0 ;if (__kvm_irq_delivery_to_apic_fast (kvm, src, irq, &r, rtc_status))return r;
@@ -1456,6 +1491,8 @@ int __kvm_irq_delivery_to_apic(struct kvm *kvm, struct kvm_lapic *src,memset (dest_vcpu_bitmap,0 ,sizeof (dest_vcpu_bitmap));kvm_for_each_vcpu (i, vcpu, kvm) { +int delivered; +if (!kvm_apic_present (vcpu))continue ;
@@ -1466,7 +1503,12 @@ int __kvm_irq_delivery_to_apic(struct kvm *kvm, struct kvm_lapic *src,if (!kvm_lowest_prio_delivery (irq)) {if (r <0 ) r =0 ; - r +=kvm_apic_set_irq (vcpu, irq, rtc_status); + delivered =kvm_apic_set_irq (vcpu, irq, rtc_status); + r += delivered; +if (delivered >0 ) { + ipi_targets++; + ipi_unique = vcpu; + } }else if (kvm_apic_sw_enabled (vcpu->arch.apic)) {if (!vector_hashing_enabled) {if (!lowest)
@@ -1487,8 +1529,27 @@ int __kvm_irq_delivery_to_apic(struct kvm *kvm, struct kvm_lapic *src, lowest =kvm_get_vcpu (kvm, idx); } -if (lowest) - r =kvm_apic_set_irq (lowest, irq, rtc_status); +if (lowest) { +int delivered =kvm_apic_set_irq (lowest, irq, rtc_status); + + r = delivered; +if (delivered >0 ) { + ipi_targets++; + ipi_unique = lowest; + } + } + +/* +* Record a unicast fixed IPI delivered via this slow path. The fast +* path records the APIC-map-hit case; this covers the fallback +* where kvm_apic_map_get_dest_lapic() missed but delivery still +* resolves to exactly one recipient. +*/ +if (src && irq->delivery_mode ==APIC_DM_FIXED && + irq->shorthand ==APIC_DEST_NOSHORT && + ipi_targets ==1 && ipi_unique && ipi_unique != src->vcpu) +kvm_track_ipi_communication (src->vcpu, ipi_unique, + irq->vector);return r; }
Signature · 1 line
2.43.0
PATCH v3 8/10 KVM: x86/lapic: Clear IPI tracking on matching-vector EOI
In reply to Wanpeng LiFrom: Wanpeng Li <[email protected]>
Clear IPI tracking from both EOI sites, apic_set_eoi() and kvm_apic_set_eoi_accelerated(), matched on the EOI'd vector. A receiver may handle unrelated interrupts between an IPI's arrival and its EOI; a vector-blind clear could evict the still-pending context and make directed yield miss the receiver.
When the vector matches, clear the receiver's context unconditionally, and clear the sender's pending flag only if it still points at this receiver with the same vector and within the recency window. The existing trace_kvm_eoi() tracepoint already records the vector, so no new tracepoint is added.
--- arch/x86/kvm/lapic.c | 61 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 61 insertions(+)
arch/x86/kvm/lapic.c +61 -0
@@ -1746,6 +1746,65 @@ static void kvm_ioapic_send_eoi(struct kvm_lapic *apic, int vector)#endif } +/* +* Clear IPI tracking state associated with a just-acknowledged vector. +* +* Called from both the emulated APIC_EOI write path (apic_set_eoi) +* and the accelerated exit path (kvm_apic_set_eoi_accelerated). +* +* Vector matching is deliberate: the receiver may handle an unrelated +* interrupt (timer, device IRQ) between an IPI's arrival and its EOI. +* If we cleared unconditionally, such an intermediate EOI would evict +* the still-pending IPI context and cause directed yield to miss the +* real receiver. We therefore only touch state when the EOI'd vector +* matches the one we recorded at send time. +* +* Two-stage cleanup: +* 1. Clear the receiver's context unconditionally when the vector +* matches: it has processed exactly this IPI. +* 2. Clear the sender's pending_ipi flag only if the sender still +* points at this receiver, with the same vector, and within the +* configured recency window. This avoids evicting a newer IPI +* that happens to share the vector with a stale one. +*/ +static void kvm_clear_ipi_on_eoi (struct kvm_lapic *apic,int vector) +{ +struct kvm_vcpu *receiver = apic->vcpu; +struct kvm_vcpu *sender; +int sender_idx; +u64 then, now; + +if (unlikely (!READ_ONCE (ipi_tracking_enabled))) +return ; + +if (vector <0 || vector >0xff ) +return ; + +if (READ_ONCE (receiver->arch.ipi_context.vector) != (u8 )vector) +return ; + + sender_idx =READ_ONCE (receiver->arch.ipi_context.last_ipi_sender); +kvm_vcpu_clear_ipi_context (receiver); + +if (sender_idx <0 ) +return ; + + sender =kvm_get_vcpu (receiver->kvm, sender_idx); +if (!sender) +return ; + +if (READ_ONCE (sender->arch.ipi_context.last_ipi_receiver) != + receiver->vcpu_idx) +return ; +if (READ_ONCE (sender->arch.ipi_context.vector) != (u8 )vector) +return ; + + then =READ_ONCE (sender->arch.ipi_context.ipi_time_ns); + now =ktime_get_mono_fast_ns (); +if (now - then <=READ_ONCE (ipi_window_ns)) +WRITE_ONCE (sender->arch.ipi_context.pending_ipi, false); +} +static int apic_set_eoi (struct kvm_lapic *apic) {int vector =apic_find_highest_isr (apic);
@@ -1766,6 +1825,7 @@ static int apic_set_eoi(struct kvm_lapic *apic)kvm_hv_synic_send_eoi (apic->vcpu, vector);kvm_ioapic_send_eoi (apic, vector); +kvm_clear_ipi_on_eoi (apic, vector);kvm_make_request (KVM_REQ_EVENT , apic->vcpu);return vector; }
@@ -1781,6 +1841,7 @@ void kvm_apic_set_eoi_accelerated(struct kvm_vcpu *vcpu, int vector)trace_kvm_eoi (apic, vector);kvm_ioapic_send_eoi (apic, vector); +kvm_clear_ipi_on_eoi (apic, vector);kvm_make_request (KVM_REQ_EVENT , apic->vcpu); }EXPORT_SYMBOL_FOR_KVM_INTERNAL (kvm_apic_set_eoi_accelerated);
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v3 08/10] KVM: x86/lapic: Clear IPI tracking on matching-vector EOI
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The EOI handler incorrectly clears the vCPU's outgoing IPI tracking state when acknowledging an…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The EOI handler incorrectly clears the vCPU's outgoing IPI tracking state when acknowledging an incoming IPI due to conflated incoming and outgoing tracking fields. --
commit f711eb495ad18e2339b40d0c737023fdb3459a71
Author: Wanpeng Li <[email protected]>
KVM: x86/lapic: Clear IPI tracking on matching-vector EOI
This commit updates apic_set_eoi() and kvm_apic_set_eoi_accelerated() to clear IPI tracking state only when the EOI vector matches the tracked vector, aiming to prevent eviction of pending contexts when unrelated interrupts are handled.
Quoted 3 lines
arch/x86/kvm/lapic.c +0
-0
[ ... ]
Quoted 8 lines
arch/x86/kvm/lapic.c +8
-0+static void kvm_clear_ipi_on_eoi (struct kvm_lapic *apic,int vector) +{ +struct kvm_vcpu *receiver = apic->vcpu; +struct kvm_vcpu *sender; +int sender_idx; +u64 then, now; + +if (unlikely (!READ_ONCE (ipi_tracking_enabled)))
arch/x86/kvm/lapic.c static void kvm_ioapic_send_eoi(struct kvm_lapic *apic, int vector) · line 1787
Quoted 10 lines
arch/x86/kvm/lapic.c +10
-0+return ; + +if (vector <0 || vector >0xff ) +return ; + +if (READ_ONCE (receiver->arch.ipi_context.vector) != (u8 )vector) +return ; + + sender_idx =READ_ONCE (receiver->arch.ipi_context.last_ipi_sender); +kvm_vcpu_clear_ipi_context (receiver);
[Severity: Medium]
Does calling kvm_vcpu_clear_ipi_context() here unintentionally clear the vCPU's outgoing IPI tracking state?
Looking at kvm_vcpu_clear_ipi_context(), it clears the entire ipi_context structure, including pending_ipi and last_ipi_receiver. Since a vCPU can act as both a sender and a receiver concurrently, acknowledging an incoming IPI here appears to erase the vCPU's state as a sender.
Could this destroy the directed yield hint and cause concurrent IPI sender and receiver vCPUs to miss the yield optimization?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
PATCH v3 9/10 KVM: Add IPI-aware directed-yield candidate selection
In reply to Wanpeng LiFrom: Wanpeng Li <[email protected]>
kvm_vcpu_on_spin() selects a directed-yield target from coarse preempted and preempted-in-kernel state. It cannot distinguish a vCPU spinning on an IPI response from a vCPU spinning on a lock, and can therefore yield to an unrelated vCPU.
Add kvm_vcpu_is_good_yield_candidate(), a priority-ordered filter. Prefer a confirmed recent IPI receiver of the spinning vCPU, then an arch-provided pending-interrupt hint, and finally the existing preempted heuristic with the optional in-kernel filter.
Use the helper in the strict scan before falling back to the existing directed-yield eligibility checks.
--- virt/kvm/kvm_main.c | 65 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 55 insertions(+), 10 deletions(-)
virt/kvm/kvm_main.c +55 -10
@@ -3963,12 +3963,51 @@ bool __weak kvm_vcpu_is_ipi_receiver(struct kvm_vcpu *sender,return false; } +/* +* Priority-based candidate filter for directed yield. +* +* 1) Confirmed IPI receiver of @me within the recency window. +* This is the highest-confidence signal that selecting @vcpu +* may help @me complete its spin-on-IPI-response. +* 2) Arch-provided fast pending-interrupt hint (APICv / AVIC / +* arch_dy_has_pending_interrupt()). Covers cases where IPI +* tracking was bypassed by hardware acceleration. +* 3) Legacy preempted fallback, with the existing optional +* in-kernel filter when @yield_to_kernel_mode is set. +* +* Returning false asks kvm_vcpu_on_spin() to skip @vcpu in the +* strict (first) round; the relaxed (second) round applies only a +* vcpu->preempted check. +*/ +static bool kvm_vcpu_is_good_yield_candidate (struct kvm_vcpu *me, +struct kvm_vcpu *vcpu, +bool yield_to_kernel_mode) +{ +/* Priority 1: confirmed recent IPI receiver. */ +if (kvm_vcpu_is_ipi_receiver (me, vcpu)) +return true; + +/* Priority 2: arch-specific pending-interrupt hint. */ +if (kvm_arch_dy_has_pending_interrupt (vcpu)) +return true; + +/* Priority 3: preempted, with optional in-kernel requirement. */ +if (!READ_ONCE (vcpu->preempted)) +return false; + +if (yield_to_kernel_mode && !kvm_arch_vcpu_preempted_in_kernel (vcpu)) +return false; + +return true; +} +void kvm_vcpu_on_spin (struct kvm_vcpu *me,bool yield_to_kernel_mode) {int nr_vcpus, start, i, idx, yielded;struct kvm *kvm = me->kvm;struct kvm_vcpu *vcpu;int try =3 ; +bool first_round = true; nr_vcpus =atomic_read (&kvm->online_vcpus);if (nr_vcpus <2 )
@@ -4010,16 +4049,21 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)if (kvm_vcpu_is_blocking (vcpu) && !vcpu_dy_runnable (vcpu))continue ; -/* -* Treat the target vCPU as being in-kernel if it has a pending -* interrupt, as the vCPU trying to yield may be spinning -* waiting on IPI delivery, i.e. the target vCPU is in-kernel -* for the purposes of directed yield. -*/ -if (READ_ONCE (vcpu->preempted) && yield_to_kernel_mode && - !kvm_arch_dy_has_pending_interrupt (vcpu) && - !kvm_arch_vcpu_preempted_in_kernel (vcpu)) -continue ; +if (first_round) { +/* Strict round: IPI-aware and legacy preempted filters. */ +if (!kvm_vcpu_is_good_yield_candidate (me, vcpu, + yield_to_kernel_mode)) +continue ; + }else { +/* +* Relaxed round: only require preempted. This is the +* safety net for missed IPI tracking (e.g. APICv) or +* transient runnable-set changes since the strict +* scan. +*/ +if (!READ_ONCE (vcpu->preempted)) +continue ; + }if (!kvm_vcpu_eligible_for_directed_yield (vcpu))continue ;
@@ -4032,6 +4076,7 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)break ; } } +kvm_vcpu_set_in_spin_loop (me, false);/* Ensure vcpu is not eligible during next spinloop */
Signature · 1 line
2.43.0
[email protected]
AI reviewRe: [PATCH v3 09/10] KVM: Add IPI-aware directed-yield candidate selection
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The new candidate filter incorrectly excludes woken-up, runnable vCPUs from being yield targets by…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The new candidate filter incorrectly excludes woken-up, runnable vCPUs from being yield targets by strictly requiring the preempted flag. --
commit f2a5464aa09eda51a02ed6e5f04bf8ea0072c9ba
Author: Wanpeng Li <[email protected]>
KVM: Add IPI-aware directed-yield candidate selection
This commit introduces a priority-based candidate filter for directed yields in kvm_vcpu_on_spin(). It prefers recent IPI receivers or vCPUs with pending interrupts before falling back to the legacy preempted heuristic.
Quoted 3 lines
virt/kvm/kvm_main.c +0
-0
[ ... ]
Quoted 5 lines
virt/kvm/kvm_main.c +5
-0+static bool kvm_vcpu_is_good_yield_candidate (struct kvm_vcpu *me, +struct kvm_vcpu *vcpu, +bool yield_to_kernel_mode) +{ +/* Priority 1: confirmed recent IPI receiver. */
virt/kvm/kvm_main.c bool __weak kvm_vcpu_is_ipi_receiver(struct kvm_vcpu *sender, · line 3996
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+if (kvm_vcpu_is_ipi_receiver (me, vcpu)) +return true; + +/* Priority 2: arch-specific pending-interrupt hint. */ +if (kvm_arch_dy_has_pending_interrupt (vcpu)) +return true; + +/* Priority 3: preempted, with optional in-kernel requirement. */ +if (!READ_ONCE (vcpu->preempted)) +return false;
[Severity: High]
Does this code inadvertently exclude runnable but non-preempted vCPUs from being yield targets?
In the previous implementation, the preempted check was part of a compound skip condition:
if (READ_ONCE(vcpu->preempted) && yield_to_kernel_mode && !kvm_arch_dy_has_pending_interrupt(vcpu) && !kvm_arch_vcpu_preempted_in_kernel(vcpu)) continue;
This meant that if a vCPU was not preempted (for instance, it was recently woken up from a blocking state and is ready to run), it bypassed the skip condition and was considered a valid target.
By explicitly returning false here when vcpu->preempted is false, this new filter seems to drop those valid targets.
Quoted 6 lines
virt/kvm/kvm_main.c +6
-0+ +if (yield_to_kernel_mode && !kvm_arch_vcpu_preempted_in_kernel (vcpu)) +return false; + +return true; +}
[ ... ]
Quoted 17 lines
virt/kvm/kvm_main.c +4
-10if (kvm_vcpu_is_blocking (vcpu) && !vcpu_dy_runnable (vcpu))continue ; -/* -* Treat the target vCPU as being in-kernel if it has a pending -* interrupt, as the vCPU trying to yield may be spinning -* waiting on IPI delivery, i.e. the target vCPU is in-kernel -* for the purposes of directed yield. -*/ -if (READ_ONCE (vcpu->preempted) && yield_to_kernel_mode && - !kvm_arch_dy_has_pending_interrupt (vcpu) && - !kvm_arch_vcpu_preempted_in_kernel (vcpu)) -continue ; +if (first_round) { +/* Strict round: IPI-aware and legacy preempted filters. */ +if (!kvm_vcpu_is_good_yield_candidate (me, vcpu, + yield_to_kernel_mode))
virt/kvm/kvm_main.c void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode) · line 4065
Quoted 10 lines
virt/kvm/kvm_main.c +10
-0+continue ; + }else { +/* +* Relaxed round: only require preempted. This is the +* safety net for missed IPI tracking (e.g. APICv) or +* transient runnable-set changes since the strict +* scan. +*/ +if (!READ_ONCE (vcpu->preempted)) +continue ;
[Severity: High]
Similarly, does this relaxed round unconditionally skip non-preempted vCPUs?
If the vCPU is simply runnable but not marked as preempted, this safety net will completely bypass it.
Quoted 3 lines
virt/kvm/kvm_main.c +0
-0}if (!kvm_vcpu_eligible_for_directed_yield (vcpu))
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
PATCH v3 10/10 KVM: Add relaxed preempted-only fallback for directed yield
In reply to Wanpeng LiFrom: Wanpeng Li <[email protected]>
The strict IPI-aware candidate filter can find no target if IPI tracking misses the relationship, for example for APICv-delivered IPIs, or if the runnable set changes during the scan.
If the strict pass yields nothing, run a second relaxed pass gated only by vcpu->preempted. Control the fallback with the enable_relaxed_boost module parameter (default on), so it can be disabled at runtime if it causes over-boosting.
With the full series, PARSEC simlarge on 16-vCPU guests under host CPU overcommit, latency reduction:
Dedup (IPI-heavy synchronization):
2 VMs: +8.87%
3 VMs: +10.29%
4 VMs: +15.60%VIPS (balanced sync and compute):
2 VMs: +10.23%
3 VMs: +6.63%
4 VMs: +4.50%The IPI-heavy Dedup workload benefits most, as the confirmed IPI receiver is preferred over the generic preempted lock-holder heuristic.
--- virt/kvm/kvm_main.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
virt/kvm/kvm_main.c +24 -0
@@ -101,6 +101,19 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(halt_poll_ns_shrink);static bool __ro_after_init allow_unsafe_mappings;module_param (allow_unsafe_mappings, bool,0444 ); +/* +* enable_relaxed_boost - second-round safety net for kvm_vcpu_on_spin(). +* +* When on (default), if the strict scan finds no eligible yield target, +* fall back to a relaxed scan gated only by vcpu->preempted. This +* preserves forward progress if IPI tracking is missed (e.g. +* APICv-delivered IPIs) or the runnable set changes mid-scan. +* +* Disable this at runtime if the relaxed pass causes over-boosting. +*/ +static bool enable_relaxed_boost = true; +module_param (enable_relaxed_boost, bool,0644 ); +/* * Ordering of locks: *
@@ -4037,6 +4050,8 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode)* they may all try to yield to the same vCPU(s). But as above, this * is all best effort due to KVM's lack of visibility into the guest. */ +retry: + yielded =0 ; start =READ_ONCE (kvm->last_boosted_vcpu) +1 ;for (i =0 ; i < nr_vcpus; i++) { idx = (start + i) % nr_vcpus;
@@ -4077,6 +4092,15 @@ void kvm_vcpu_on_spin(struct kvm_vcpu *me, bool yield_to_kernel_mode) } } +/* +* Second, relaxed pass if enabled, the strict pass yielded nothing, +* and we still have retry budget for -ESRCH paths. +*/ +if (enable_relaxed_boost && first_round && yielded <=0 && try >0 ) { + first_round = false; + goto retry; + } +kvm_vcpu_set_in_spin_loop (me, false);/* Ensure vcpu is not eligible during next spinloop */
Signature · 1 line
2.43.0
K Prateek Nayak
reviewerHello Wanpeng,
Quoted 11 lines · Wanpeng Li
Part 1: Scheduler EEVDF lag credit (patches 1-5)
Rather than penalizing the yielding vCPU, credit the nominated target so
pick_eevdf()honors the buddy hint.The mechanism is EEVDF-native and cgroup-hierarchy-aware:
- Credit bounded EEVDF lag to the nominated next-buddy so pick_eevdf()'s
PICK_BUDDYbranch returns it. Walk the same ancestor chain thatset_next_buddy()nominated and credit each not-yet-eligible level, so the hint is not dropped at the first ineligible group entity.
I believe Peter is planning to flatten the pick by v7.3 so I would suggest you to test the flattened pick series [1] which is available in Peter's tree in sched/flat branch [2].
That should get rid of the need to traverse the hierarchy and should solve one part of your problem of yielding to vCPUs across different cgroups.
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=sched/flat
Quoted 9 lines
- Credit to a small positive-vlag margin, not merely the vlag = 0 eligibility boundary, so the target stays eligible across several scheduling decisions rather than a single pick. The margin scales with runqueue depth and is clamped to entity_lag()'s legal positive-lag bound, preserving EEVDF fairness.
- Handle both the off-tree current entity (shifted in place, carrying any vprot window)
Is this even possible to yield to an out of tree entity? The core bits in syscalls.c already bails out for:
if (task_on_cpu(p_rq, p) || !task_is_running(p))
return;and the early bits in yield_to_task_fair() bail out for "!se->on_rq" which makes me wonder when will we ever have the p->se as cfs_rq->curr while holding both the p->pi_lock and the rq_lock?
The task must be on the rq while being preempted for yield_to to work, no?
Quoted 13 lines
and a queued (on-tree) entity (repositioned via the
canonical place_entity()-paired requeue used byrequeue_delayed_entity(), keepingsum_w_vruntimeconsistent withentity_key()).- Force a local reschedule at the end of the credit path: cancel
RUN_TO_PARITYslice protection along the yielder'ssched_entitychain andresched_curr()the local CPU. Only this forced preemption is rate limited (once per 6ms per rq) to avoid excessive forced preemption on PLE-heavy guests; the lag credit itself runs on every directed yield.The mechanism is gated by
SCHED_FEAT(YIELD_TO_LAG_CREDIT)(default on). With the feature off,yield_to_task_fair()keeps the existing forfeit-only behavior.
[...]
Quoted 7 lines
The gains stem from three factors:
1. Lock holders receive sustained CPU time to complete critical sections, reducing lock hold duration and cascading contention.
2. IPI receivers are scheduled promptly when senders spin, reducing IPI response latency and wasted spin cycles.
Looking at kvm_smp_send_call_func_ipi() in arch/x86/kernel/kvm.c, there can be multiple destination vCPUs for the IPI. Why does it make sense for the sender to yield almost all its time to the first vCPU on the mask then?
And do all IPIs have to spin? Can't they be async too?
Quoted 4 lines
3. Reduced context switching between lock waiters and holders improves cache utilization.
[...]
Quoted 16 lines
Patch Organization
------------------Patches 1-5: Scheduler EEVDF lag credit
Patch 1: Add the
eevdf_credit_entity_vlag()primitive and the
YIELD_TO_LAG_CREDITfeature. Handles the off-tree current entity and has no functional effect on its own.Patch 2: Credit to a persistent, queue-depth-scaled positive-vlag
margin, clamped to entity_lag()'s legal bound.Patch 3: Extend the primitive to a queued (on-tree) entity via the
canonical place_entity()-paired requeue.
Quoted 6 lines
Patch 4: Wire the credit walk into
yield_to_task_fair(), crediting each
level of the nominated ancestor chain.Patch 5: Force a local reschedule (cancel
RUN_TO_PARITYslice protection
andresched_curr()) so the credited buddy can be selected. Activation patch; rate-limits only the forced preemption.
I don't know if it is just me but this structure made it insanely difficult to review with unused functions and callers being only added at Patch 4 to understand how it all worked.
All of this will require rework with flattened pick but I would suggest adding the simple lag movement bits first and adding the eevdf_persistent_margin() magic later on top.
Signature · 2 lines
Thanks and Regards, Prateek
Hi Wanpeng.
Quoted 8 lines · Wanpeng Li
From: Wanpeng Li <[email protected]>
On overcommitted hosts, a spinning vCPU often calls
yield_to()to let a lock holder or IPI receiver run. The hint can be ineffective for two independent reasons: the scheduler may fail to select the nominated task, and KVM may nominate a task that is not the one the spinning vCPU is waiting for.
I was curious to try this as we have been working on paravirt scheduling which aims at reducing the vCPU preemption based on the steal time.
https://lore.kernel.org/all/[email protected]/
I gave the patch series a try. With both real life workload(daytrader) and micro-benchmarks (hackbench) I don't see any benefit. Note that it is running the same workloads in all VM's at the same time.
I had thought about this series:
- Do you observe steal time when you run your workload? If yes, if you reduce the vCPU, IPI will natually move to reduced set and due to reduction in vCPU preemption yeild also may not be necessary.
- is there vCPU pinning? workload is pinned separately to explicit vCPUs.
- Is only a few vCPUs doing the locks in the workload you have mentioned in the cover-letter?
Vincent Guittot
maintainerQuoted 29 lines · Wanpeng Li
From: Wanpeng Li <[email protected]>
On overcommitted hosts, a spinning vCPU often calls
yield_to()to let a lock holder or IPI receiver run. The hint can be ineffective for two independent reasons: the scheduler may fail to select the nominated task, and KVM may nominate a task that is not the one the spinning vCPU is waiting for.This series addresses both sides. The scheduler side credits bounded EEVDF lag to the nominated next-buddy so the buddy hint is honored across the relevant cgroup hierarchy, and forces a local reschedule so the credited buddy can be selected immediately. The KVM side tracks recent unicast fixed IPI sender/receiver pairs and prefers the confirmed receiver when selecting a directed-yield target.
Problem Statement
-----------------In overcommitted virtualization scenarios, vCPUs frequently spin on locks held by other vCPUs that are not currently running, or on IPI responses from vCPUs that are runnable but not scheduled. Paravirtual spinlock support and PLE detect these situations and call
yield_to()to let the other vCPU make progress.The current implementation has two limitations:
1. Scheduler-side limitation:
Quoted 7 lines
yield_to_task_fair()relies onset_next_buddy()to express a preference for the target.set_next_buddy()nominates the target at every level of its cgroup ancestor chain, but pick_eevdf()'sPICK_BUDDYbranch only returnscfs_rq->nextwhen that entity is already eligible (entity_eligible()). A target that is behindavg_vruntimeat any level of the chain is skipped, and the hint is dropped at the first ineligible group entity.
Because the target has already consumed more than its cpu bandwidth
Quoted 6 lines
Even when the target is eligible,
yield_to()does not by itself force the caller off the CPU. An activeRUN_TO_PARITYprotect_slice()on the local yielder can therefore keeppick_eevdf()returning the yielder instead of the target.
Quoted 8 lines
The recent forfeit-on-yield work (commits
79104becf42b"sched/fair:
Forfeit vruntime on yield" and127b90315ca0"sched/proxy: Yield the donor task") makes the yielder ineligible, but it does not make the nominated target eligible when that target is behindavg_vruntime, keepPICK_BUDDYfrom being dropped at the first ineligible group entity, or cancel an activeRUN_TO_PARITYslice on the yielder. This series builds on that behavior by crediting the target and cancelling slice protection, so the nominated entity is the onepick_eevdf()returns.
Cancel protection could make sense but I don't think you need it because the next buddy is checked before the slice protection. You only need to set a resched
And why would you add runtime credit? This breaks the fairness which we don't want to
Quoted 28 lines
2. KVM-side limitation:
kvm_vcpu_on_spin()selects a directed-yield target from coarse preempted / preempted-in-kernel state. It cannot distinguish a vCPU spinning on an IPI response from a vCPU spinning on a lock. When a vCPU sends an IPI and spins waiting for the response, the heuristic can boost an unrelated vCPU and miss the actual IPI receiver.These effects lengthen lock hold times and increase spin time, context-switch overhead and cache pressure in overcommitted environments, especially for workloads with fine-grained synchronization.
Solution Overview
-----------------Part 1: Scheduler EEVDF lag credit (patches 1-5)
Rather than penalizing the yielding vCPU, credit the nominated target so
pick_eevdf()honors the buddy hint.The mechanism is EEVDF-native and cgroup-hierarchy-aware:
- Credit bounded EEVDF lag to the nominated next-buddy so pick_eevdf()'s
PICK_BUDDYbranch returns it. Walk the same ancestor chain thatset_next_buddy()nominated and credit each not-yet-eligible level, so the hint is not dropped at the first ineligible group entity.
Quoted 5 lines
- Credit to a small positive-vlag margin, not merely the vlag = 0 eligibility boundary, so the target stays eligible across several scheduling decisions rather than a single pick. The margin scales with runqueue depth and is clamped to entity_lag()'s legal positive-lag bound, preserving EEVDF fairness.
There is no reason to add some credit to a task that already exhausted its CPU share
Quoted 308 lines
- Handle both the off-tree current entity (shifted in place, carrying any vprot window) and a queued (on-tree) entity (repositioned via the canonical place_entity()-paired requeue used by
requeue_delayed_entity(), keepingsum_w_vruntimeconsistent withentity_key()).- Force a local reschedule at the end of the credit path: cancel
RUN_TO_PARITYslice protection along the yielder'ssched_entitychain andresched_curr()the local CPU. Only this forced preemption is rate limited (once per 6ms per rq) to avoid excessive forced preemption on PLE-heavy guests; the lag credit itself runs on every directed yield.The mechanism is gated by
SCHED_FEAT(YIELD_TO_LAG_CREDIT)(default on). With the feature off,yield_to_task_fair()keeps the existing forfeit-only behavior.Part 2: KVM IPI-aware directed yield (patches 6-10)
KVM tracks recent unicast fixed IPI sender/receiver relationships and uses them to prioritize directed-yield targets.
- Record unicast fixed IPIs from both LAPIC delivery paths, the APIC-map fast path and the slow fallback, when exactly one destination vCPU accepts the interrupt.
- Use READ_ONCE()/WRITE_ONCE() accessors. The per-vCPU
ipi_contextstate is only a best-effort scheduling hint.- Age out stale relationships with a recency window (50ms default), and clear state on a matching-vector EOI without dropping unrelated pending IPI state.
Directed-yield candidate selection uses the following priority order:
1. A confirmed recent IPI receiver of the spinning vCPU.
2. The arch-specific pending-interrupt hint(kvm_arch_dy_has_pending_interrupt()).3. The existing preempted / preempted-in-kernel heuristic.
If the strict IPI-aware pass finds no eligible candidate, an optional second pass falls back to a relaxed preempted-only search. The fallback is controlled by the
enable_relaxed_boostmodule parameter (default on).Runtime controls:
*/sys/kernel/debug/sched/features(YIELD_TO_LAG_CREDIT)
*/sys/module/kvm/parameters/ipi_tracking_enabled
*/sys/module/kvm/parameters/ipi_window_ns
*/sys/module/kvm/parameters/enable_relaxed_boostHost-side deployment model
--------------------------The series is host-side by design. It requires no guest ABI, paravirtual driver, negotiated feature bit, or guest kernel change, so existing guests benefit without coordination between host and guest software.
That deployment model gives the mechanisms broad coverage. The scheduler lag credit applies to every
yield_to()the host already receives, including PLE and paravirtual spinlock paths. The KVM side observes the actual unicast-IPI sender/receiver relationship at software LAPIC delivery time, so it covers spin and IPI waits from spinlocks, RCU,smp_call_function()and IPI-based wakeups rather than a single paravirtualized operation such as TLB shootdown.The host-side approach also composes with existing paravirtualization. If a guest provides PV TLB shootdown or PV spinlocks, those interfaces reduce the amount of spinning that reaches the host; this series handles the residual
yield_to()and IPI waits that remain. It is runtime gated as described above and can be enabled or disabled per host.The scheduler side is independent of APICv, IPI virtualization and the LAPIC delivery path. The KVM side depends on software LAPIC delivery: when IPI/EOI virtualization handles the guest's ICR and EOI writes in hardware, no sender/receiver relationship is recorded, and candidate selection falls back to the pending-interrupt and preempted heuristics, plus the relaxed preempted-only pass added in patch 10. In that configuration the tracking state stays empty while the scheduler side remains fully active.
The design separates the consumer of the hint from its source. Software IPI tracking supplies the confirmed receiver on hosts where software LAPIC delivery is observable today; a future guest-cooperative scheduling hint could populate the same slot without changing the priority-ordered candidate selection.
Performance Results
-------------------Test environment: a 16-core x86-64 host, 16 vCPUs per guest. Host CPU overcommit is varied by co-locating 2, 3 and 4 guests (120 runs per point), with APICv disabled so the KVM side observes IPI delivery in software. Dbench reports throughput and reflects the scheduler-side lag credit; the PARSEC workloads report end-to-end latency reduction under the full series.
Dbench (filesystem metadata operations), throughput improvement:
2 VMs: +6.65% 3 VMs: +4.80% 4 VMs: +7.59%PARSEC Dedup, simlarge input (IPI-heavy synchronization), latency reduction:
2 VMs: +8.87%
3 VMs: +10.29%
4 VMs: +15.60%PARSEC VIPS, simlarge input (balanced sync and compute), latency reduction:
2 VMs: +10.23%
3 VMs: +6.63%
4 VMs: +4.50%Analysis:
- Dedup's gains grow with the VM count: as more runnable vCPUs compete for each physical CPU, a directed yield is more likely to land on a vCPU that is genuinely preempted while an IPI sender spins, so honoring the confirmed receiver matters more.
- Dedup, with its IPI-heavy synchronization, benefits most from the IPI-aware directed yield. Preferring the confirmed IPI receiver over the generic preempted-lock-holder heuristic shortens IPI response latency.
- VIPS mixes synchronization and compute, so its gains shrink as the VM count rises: at higher overcommit more of each run is spent in compute that a directed yield cannot accelerate, leaving less spin time to recover.
- Dbench benefits primarily from the scheduler-side lag credit; its lock patterns involve more direct lock-holder boosting than IPI spinning.
- No configuration regressed; the mechanisms degrade gracefully as contention rises.
The gains stem from three factors:
1. Lock holders receive sustained CPU time to complete critical sections, reducing lock hold duration and cascading contention.
2. IPI receivers are scheduled promptly when senders spin, reducing IPI response latency and wasted spin cycles.
3. Reduced context switching between lock waiters and holders improves cache utilization.
Scope of the scheduler-side benefit
-----------------------------------The lag credit takes effect only when the yielding vCPU and its target share a runqueue, i.e. when more runnable vCPUs than pCPUs contend for a CPU:
- Under CPU overcommit - co-located guests, or a VM whose vCPUs are pooled onto fewer pCPUs than it has vCPUs - the waiter and the lock-holder or IPI-receiver land on the same rq, and the buddy hint applies. The results here are from this regime, with guests co-located so their vCPUs contend for shared pCPUs.
- Without such contention - 1:1 vCPU:pCPU pinning, or a matched vCPU:pCPU count with no intra-VM overcommit - there is no eligible buddy to credit, so the path is inert and adds no overhead or regression.
Independent s390 testing (directed yield there uses the diag9c hypercall) shows the same pattern: under intra-VM vCPU pooling the yield-to hypercall rate falls by more than half with a few percent throughput gain, while 1:1 pinning and matched vCPU:pCPU configurations show no change either way.
Directed yield is a same-runqueue mechanism and cannot help a waiter whose target is on a different rq; extending it to cross-runqueue cases is left as future work.
Patch Organization
------------------Patches 1-5: Scheduler EEVDF lag credit
Patch 1: Add the
eevdf_credit_entity_vlag()primitive and the
YIELD_TO_LAG_CREDITfeature. Handles the off-tree current entity and has no functional effect on its own.Patch 2: Credit to a persistent, queue-depth-scaled positive-vlag
margin, clamped to entity_lag()'s legal bound.Patch 3: Extend the primitive to a queued (on-tree) entity via the
canonical place_entity()-paired requeue.Patch 4: Wire the credit walk into
yield_to_task_fair(), crediting each
level of the nominated ancestor chain.Patch 5: Force a local reschedule (cancel
RUN_TO_PARITYslice protection
andresched_curr()) so the credited buddy can be selected. Activation patch; rate-limits only the forced preemption.Patches 6-10: KVM IPI-aware directed yield
Patch 6: Add per-vCPU IPI tracking infrastructure, module parameters and
helper functions. Candidate selection is unchanged.Patch 7: Track unicast fixed IPI delivery from both LAPIC paths.
Patch 8: Clear IPI tracking on a matching-vector EOI.
Patch 9: Implement IPI-aware directed-yield candidate selection with the
priority order above.Patch 10: Add the relaxed preempted-only fallback as a safety net.
Testing
-------Workloads tested:
- Dbench (filesystem metadata stress)
- PARSEC benchmarks (Dedup, VIPS)
- Kernel compilation (make -j16 in each VM)No regressions observed on any configuration. The mechanisms show neutral to positive impact across diverse workloads.
Rate-limit policy
-----------------The scheduler-side forced reschedule is rate-limited to bound the cost of frequent VM exits. Under the kvm-full profile, PLE-heavy workloads such as PARSEC VIPS and Dedup take many PAUSE-loop exits; each exit can drive a
yield_to(), and thus a potential forced preemption. Forcing a reschedule on everyyield_to()would add needless preemption pressure and cache churn.The series limits only the forced preemption path (
cancel_protect_slice()plusresched_curr()) to once per 6ms per rq. The lag credit itself remains unthrottled, so each directed yield refreshes the buddy hint. The fixed 6ms interval is intentionally conservative; an adaptive limit based on the per-rq yield_to()/PLE-exit rate can be explored separately.Changelog:
v2 -> v3:
- Redesign the scheduler side. v2 applied a bounded vruntime penalty to the yielding vCPU (a "debooster"); v3 instead credits bounded EEVDF lag to the nominated next-buddy so pick_eevdf()'sPICK_BUDDYbranch returns it. Crediting the target is EEVDF-native, composes cleanly withRUN_TO_PARITY, and avoids the fairness reasoning required when shifting the yielder's vruntime in a cgroup hierarchy. The redesign also removes the bulk of the v2 machinery:
* Drop the cgroup LCA finder, reverse-pair debouncing, the per-rq penalty tracking and the dedicated debugfs sysctl. The mechanism is now gated bySCHED_FEAT(YIELD_TO_LAG_CREDIT).
* Credit to a queue-depth-scaled positive-vlag margin clamped to entity_lag()'s legal bound, keeping the target eligible across several picks while preserving EEVDF fairness.
* Handle the off-tree current entity (in-place shift) and a queued on-tree entity (canonical place_entity()-paired requeue) separately, sosum_w_vruntimestays consistent withentity_key().
* Add an explicit forced local reschedule that cancelsRUN_TO_PARITYslice protection so the credited buddy can be selected; only the forced preemption is rate limited (6ms/rq), the lag credit runs on every yield.
- KVM side keeps the v2 design; rebased and reorganized into five patches (infrastructure, track delivery, clear-on-EOI, candidate selection, relaxed fallback). Tracking now hooks both the APIC-map fast path and the slow fallback, and the EOI clear is vector-matched.
- Rebase onto v7.1-rc7.v1 -> v2:
- Rebase onto v6.19-rc1 (v1 was based on v6.18-rc4).
- Drop the "KVM: Fixlast_boosted_vcpuindex assignment bug" patch, as v6.19-rc1 already contains the fix.
- Scheduler side (the v2 vruntime debooster, since replaced in v3):
* Apply the deboost beforeyield_task_fair()to adapt to v6.19's EEVDF forfeit behavior (se->vruntime=se->deadline), which would otherwise inflate the yielder's vruntime before the penalty was computed.
* Userq->donorinstead ofrq->currfor correct EEVDF donor tracking.
* Useh_nr_queuedinstead ofnr_queuedfor accurate hierarchical task counting in the penalty cap.
* Drop the vlag assignment (recalculated on dequeue/enqueue) and theupdate_min_vruntime()call (the yielder iscfs_rq->curr, off-tree), and remove the unnecessarygran_floorsafeguard.
* Rename the debugfs knob tovcpu_debooster_enabled.
- KVM IPI tracking: improve module-parameter documentation and add thekvm_vcpu_is_ipi_receiver()declaration tox86.h.Wanpeng Li (10):
sched/fair: Add EEVDF lag credit primitive for nominated next-buddy sched/fair: Credit a persistent, queue-depth-scaled vlag margin sched/fair: Credit queued next-buddy via canonical requeue sched/fair: Credit nominated next-buddy inyield_to_task_fair()sched/fair: Force a local resched onyield_to()so the buddy is picked
KVM: x86: Add IPI tracking infrastructure for directed yield
KVM: x86/lapic: Track unicast fixed IPI delivery
KVM: x86/lapic: Clear IPI tracking on matching-vector EOI
KVM: Add IPI-aware directed-yield candidate selection
KVM: Add relaxed preempted-only fallback for directed yieldarch/x86/include/asm/kvm_host.h | 19 +++ arch/x86/kvm/lapic.c | 234 +++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 3 + arch/x86/kvm/x86.h | 8 ++ include/linux/kvm_host.h | 8 ++ kernel/sched/fair.c | 224 +++++++++++++++++++++++++++++- kernel/sched/features.h | 9 ++ kernel/sched/sched.h | 10 ++ virt/kvm/kvm_main.c | 95 +++++++++++-- 9 files changed, 594 insertions(+), 16 deletions(-)base-commit:
4549871118cf616eecdd2d939f78e3b9e1dddc48--
2.43.0