KVM: x86: kvmclock rate-limit removal causes IPI storm and high guest steal time
Subsystems (4)
Hi,
I am reporting a performance regression in Linux 6.19 that severely impacts KVM hosts running many Firecracker microVMs.
== Bisect result ==
446fcce2a52b533c543dabba26777813c347577c is the first bad commit commit 446fcce2a52b533c543dabba26777813c347577c
Author: Lei Chen <[email protected]> Date: Tue Aug 19 23:20:26 2025 +0800
Revert "x86: kvm: rate-limit global clock updates"
This reverts commit 7e44e4495a398eb553ce561f29f9148f40a3448f.
Commit 7e44e4495a39 ("x86: kvm: rate-limit global clock updates") intends to use a kvmclock_update_work to sync ntp corretion across all vcpus kvmclock, which is based on commit 0061d53daf26f ("KVM: x86: limit difference between kvmclock updates")
Since kvmclock has been switched to mono raw, this commit can be reverted.
arch/x86/include/asm/kvm_host.h | 1 - arch/x86/kvm/x86.c | 29 ++++------------------------- 2 files changed, 4 insertions(+), 26 deletions(-)
==== Symptoms ====
Measured on a KVM micro VM host running many Firecracker microVMs (node_exporter metrics, 2026-03-20):
kernel 6.19:
steal time inside guest VMs: 3–24% per vCPU (sustained)
host system CPU (kernel mode): 3–12 CPUs saturated
host steal: 3–8%kernel 6.18 (same host, same workload after rollback):
steal time inside guest VMs: < 0.02% per vCPU (~200x lower)
host system CPU (kernel mode): 2–3 CPUs
host steal: 0.3–0.5%==== Root cause (by AI analyze) ====
The regressing commit removes the rate-limiting from kvm_gen_kvmclock_update(). Previously this function deferred the all-vCPU kick via a 100ms delayed_work:
/* 6.18 */
static void kvm_gen_kvmclock_update(struct kvm_vcpu *v) {
kvm_make_request(KVM_REQ_CLOCK_UPDATE, v);
schedule_delayed_work(&kvm->arch.kvmclock_update_work,
KVMCLOCK_UPDATE_DELAY); /* 100ms */
}After the revert it kicks every vCPU of the VM synchronously on every call:
/* 6.19 */
static void kvm_gen_kvmclock_update(struct kvm_vcpu *v) {
kvm_for_each_vcpu(i, vcpu, kvm) {
kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu);
kvm_vcpu_kick(vcpu);
}
}KVM_REQ_GLOBAL_CLOCK_UPDATE, which calls kvm_gen_kvmclock_update(), is issued on every vCPU load when use_master_clock is false (arch/x86/kvm/x86.c, kvm_vcpu_load):
if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu == -1)
kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu);With many Firecracker microVMs, the vCPU scheduling rate is high. Each scheduling event now IPIs every sibling vCPU of the VM, instead of coalescing all-vCPU kicks into at most one per 100ms. This creates a continuous IPI storm on the host, visible as high kernel (system) CPU time and high steal time inside guest VMs.
The commit justifies the removal with "Since kvmclock has been switched to mono raw, this commit can be reverted." That reasoning is correct for the NTP-correction use case, but the 100ms rate-limit also protected against IPI storms when use_master_clock is false — a concern independent of clock source.
==== Full bisect log ====
git bisect start
# status: waiting for both good and bad commits
# good: [7d0a66e4bb9081d75c82ec4957c50034cb0ea449] Linux 6.18 git bisect good 7d0a66e4bb9081d75c82ec4957c50034cb0ea449
# status: waiting for bad commit, 1 good commit known
# bad: [05f7e89ab9731565d8a62e3b5d1ec206485eeb0b] Linux 6.19 git bisect bad 05f7e89ab9731565d8a62e3b5d1ec206485eeb0b
# good: [02892f90a9851f508e557b3c75e93fc178310d5f] Merge tag 'hwmon-for-v6.19' ofgit://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging
git bisect good 02892f90a9851f508e557b3c75e93fc178310d5f
# bad: [edf602a17b03e6bca31c48f34ac8fc3341503ac1] Merge tag 'tty-6.19-rc1' ofgit://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty
git bisect bad edf602a17b03e6bca31c48f34ac8fc3341503ac1
# bad: [09cab48db950b6fb8c114314a20c0fd5a80cf990] Merge tag 'soc-arm-6.19' ofgit://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect bad 09cab48db950b6fb8c114314a20c0fd5a80cf990
# good: [36492b7141b9abc967e92c991af32c670351dc16] Merge tag 'tracepoints-v6.19' ofgit://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
git bisect good 36492b7141b9abc967e92c991af32c670351dc16
# good: [7cd122b55283d3ceef71a5b723ccaa03a72284b4] Merge tag 'pull-persistency' ofgit://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect good 7cd122b55283d3ceef71a5b723ccaa03a72284b4
# bad: [63a9b0bc65d5d3ea96a57e7985ea22a8582fbbe5] Merge tag 'kvm-riscv-6.19-1' of https://github.com/kvm-riscv/linux into HEAD git bisect bad 63a9b0bc65d5d3ea96a57e7985ea22a8582fbbe5
# bad: [adc99a6cfcf76d670272dea64bbc2d43ecd12a2f] Merge tag 'kvm-x86-mmu-6.19' of https://github.com/kvm-x86/linux into HEAD git bisect bad adc99a6cfcf76d670272dea64bbc2d43ecd12a2f
# bad: [c09816f2afce0f89f176c4bc58dc57ec9f204998] KVM: x86: Remove unused declaration kvm_mmu_may_ignore_guest_pat()
git bisect bad c09816f2afce0f89f176c4bc58dc57ec9f204998
# bad: [f6106d41ec84e552a5e8adda1f8741cab96a5425] x86/bugs: Use an x86 feature to track the MMIO Stale Data mitigation
git bisect bad f6106d41ec84e552a5e8adda1f8741cab96a5425
# good: [9633f180ce994ab293ce4924a9b7aaf4673aa114] KVM: x86:
Explicitly set new periodic hrtimer expiration in apic_timer_fn() git bisect good 9633f180ce994ab293ce4924a9b7aaf4673aa114
# bad: [e78fb96b41c6ac85c1a02c7e9610d1ebaa9b5d98] KVM: x86: remove comment about ntp correction sync for
git bisect bad e78fb96b41c6ac85c1a02c7e9610d1ebaa9b5d98
# good: [a091fe60c2d3943b058132a64682a509d55bd325] KVM: x86: Grab lapic_timer in a local variable to cleanup periodic code git bisect good a091fe60c2d3943b058132a64682a509d55bd325
# bad: [446fcce2a52b533c543dabba26777813c347577c] Revert "x86: kvm:
rate-limit global clock updates"
git bisect bad 446fcce2a52b533c543dabba26777813c347577c
# good: [43ddbf16edf5c1790684b32d5eb920a1b0eea285] Revert "x86: kvm:
introduce periodic global clock updates"
git bisect good 43ddbf16edf5c1790684b32d5eb920a1b0eea285
# first bad commit: [446fcce2a52b533c543dabba26777813c347577c] Revert "x86: kvm: rate-limit global clock updates"
Best regards,
Jaroslav Pulchart
Hi Jaroslav,
Thanks for your test and report, I'm looking into this problem.
Best regards
Lei Chen
Quoted 152 lines · Jaroslav Pulchart
Hi,
I am reporting a performance regression in Linux 6.19 that severely impacts KVM hosts running many Firecracker microVMs.
== Bisect result ==
446fcce2a52b533c543dabba26777813c347577cis the first bad commit commit446fcce2a52b533c543dabba26777813c347577cAuthor: Lei Chen <[email protected]> Date: Tue Aug 19 23:20:26 2025 +0800Revert "x86: kvm: rate-limit global clock updates"
This reverts commit
7e44e4495a398eb553ce561f29f9148f40a3448f.Commit
7e44e4495a39("x86: kvm: rate-limit global clock updates") intends to use akvmclock_update_workto sync ntp corretion across all vcpus kvmclock, which is based on commit0061d53daf26f("KVM: x86: limit difference between kvmclock updates")Since kvmclock has been switched to mono raw, this commit can be reverted.
Signed-off-by Lei Chen <[email protected]>Signed-off-by Sean Christopherson <[email protected]>arch/x86/include/asm/kvm_host.h | 1 - arch/x86/kvm/x86.c | 29 ++++------------------------- 2 files changed, 4 insertions(+), 26 deletions(-)==== Symptoms ====
Measured on a KVM micro VM host running many Firecracker microVMs (
node_exportermetrics, 2026-03-20):kernel 6.19:
steal time inside guest VMs: 3–24% per vCPU (sustained) host system CPU (kernel mode): 3–12 CPUs saturated host steal: 3–8%kernel 6.18 (same host, same workload after rollback):
steal time inside guest VMs: < 0.02% per vCPU (~200x lower) host system CPU (kernel mode): 2–3 CPUs host steal: 0.3–0.5%==== Root cause (by AI analyze) ====
The regressing commit removes the rate-limiting from
kvm_gen_kvmclock_update(). Previously this function deferred the all-vCPU kick via a 100msdelayed_work:/* 6.18 */ static void kvm_gen_kvmclock_update(struct kvm_vcpu *v) { kvm_make_request(KVM_REQ_CLOCK_UPDATE, v); schedule_delayed_work(&kvm->arch.kvmclock_update_work, KVMCLOCK_UPDATE_DELAY); /* 100ms */ }After the revert it kicks every vCPU of the VM synchronously on every call:
/* 6.19 */ static void kvm_gen_kvmclock_update(struct kvm_vcpu *v) { kvm_for_each_vcpu(i, vcpu, kvm) { kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); kvm_vcpu_kick(vcpu); } }
KVM_REQ_GLOBAL_CLOCK_UPDATE, which callskvm_gen_kvmclock_update(), is issued on every vCPU load whenuse_master_clockis false (arch/x86/kvm/x86.c,kvm_vcpu_load):if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu == -1) kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu);With many Firecracker microVMs, the vCPU scheduling rate is high. Each scheduling event now IPIs every sibling vCPU of the VM, instead of coalescing all-vCPU kicks into at most one per 100ms. This creates a continuous IPI storm on the host, visible as high kernel (system) CPU time and high steal time inside guest VMs.
The commit justifies the removal with "Since kvmclock has been switched to mono raw, this commit can be reverted." That reasoning is correct for the NTP-correction use case, but the 100ms rate-limit also protected against IPI storms when
use_master_clockis false — a concern independent of clock source.==== Full bisect log ====
git bisect start
# status: waiting for both good and bad commits
# good: [7d0a66e4bb9081d75c82ec4957c50034cb0ea449] Linux 6.18 git bisect good7d0a66e4bb9081d75c82ec4957c50034cb0ea449
# status: waiting for bad commit, 1 good commit known
# bad: [05f7e89ab9731565d8a62e3b5d1ec206485eeb0b] Linux 6.19 git bisect bad05f7e89ab9731565d8a62e3b5d1ec206485eeb0b
# good: [02892f90a9851f508e557b3c75e93fc178310d5f] Merge tag 'hwmon-for-v6.19' ofgit://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging
git bisect good02892f90a9851f508e557b3c75e93fc178310d5f
# bad: [edf602a17b03e6bca31c48f34ac8fc3341503ac1] Merge tag 'tty-6.19-rc1' ofgit://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty
git bisect badedf602a17b03e6bca31c48f34ac8fc3341503ac1
# bad: [09cab48db950b6fb8c114314a20c0fd5a80cf990] Merge tag 'soc-arm-6.19' ofgit://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect bad09cab48db950b6fb8c114314a20c0fd5a80cf990
# good: [36492b7141b9abc967e92c991af32c670351dc16] Merge tag 'tracepoints-v6.19' ofgit://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
git bisect good36492b7141b9abc967e92c991af32c670351dc16
# good: [7cd122b55283d3ceef71a5b723ccaa03a72284b4] Merge tag 'pull-persistency' ofgit://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect good7cd122b55283d3ceef71a5b723ccaa03a72284b4
# bad: [63a9b0bc65d5d3ea96a57e7985ea22a8582fbbe5] Merge tag 'kvm-riscv-6.19-1' of https://github.com/kvm-riscv/linux into HEAD git bisect bad63a9b0bc65d5d3ea96a57e7985ea22a8582fbbe5
# bad: [adc99a6cfcf76d670272dea64bbc2d43ecd12a2f] Merge tag 'kvm-x86-mmu-6.19' of https://github.com/kvm-x86/linux into HEAD git bisect badadc99a6cfcf76d670272dea64bbc2d43ecd12a2f
# bad: [c09816f2afce0f89f176c4bc58dc57ec9f204998] KVM: x86: Remove unused declarationkvm_mmu_may_ignore_guest_pat()
git bisect badc09816f2afce0f89f176c4bc58dc57ec9f204998
# bad: [f6106d41ec84e552a5e8adda1f8741cab96a5425] x86/bugs: Use an x86 feature to track the MMIO Stale Data mitigation
git bisect badf6106d41ec84e552a5e8adda1f8741cab96a5425
# good: [9633f180ce994ab293ce4924a9b7aaf4673aa114] KVM: x86:
Explicitly set new periodic hrtimer expiration inapic_timer_fn()git bisect good9633f180ce994ab293ce4924a9b7aaf4673aa114
# bad: [e78fb96b41c6ac85c1a02c7e9610d1ebaa9b5d98] KVM: x86: remove comment about ntp correction sync for
git bisect bade78fb96b41c6ac85c1a02c7e9610d1ebaa9b5d98
# good: [a091fe60c2d3943b058132a64682a509d55bd325] KVM: x86: Grablapic_timerin a local variable to cleanup periodic code git bisect gooda091fe60c2d3943b058132a64682a509d55bd325
# bad: [446fcce2a52b533c543dabba26777813c347577c] Revert "x86: kvm:
rate-limit global clock updates"
git bisect bad446fcce2a52b533c543dabba26777813c347577c
# good: [43ddbf16edf5c1790684b32d5eb920a1b0eea285] Revert "x86: kvm:
introduce periodic global clock updates"
git bisect good43ddbf16edf5c1790684b32d5eb920a1b0eea285# first bad commit: [446fcce2a52b533c543dabba26777813c347577c] Revert "x86: kvm: rate-limit global clock updates"Best regards,
Jaroslav Pulchart
Hi Jaroslav,
I apologize for the late reply.
I have reviewed the code and identified two scenarios that currently trigger the KVM_REQ_GLOBAL_CLOCK_UPDATE request:
Scenario 1: kvm_write_system_time
This code path occurs when the hypervisor (such as QEMU) adjusts the time, or when the guest writes to the TSC.
Scenario 2: vcpu schedule in kvm_arch_vcpu_load
If this function triggers KVM_REQ_GLOBAL_CLOCK_UPDATE, it indicates that the virtual machine is not using the master_clock.
Those two cases are uncommon. Could you please provide your dmesg and help check which code path triggers KVM_REQ_GLOBAL_CLOCK_UPDATE?
Best regards,
Lei Chen
Quoted 162 lines · Lei Chen
Hi Jaroslav,
Thanks for your test and report, I'm looking into this problem.
Best regards
Lei ChenQuoted 152 lines · Jaroslav Pulchart
Hi,
I am reporting a performance regression in Linux 6.19 that severely impacts KVM hosts running many Firecracker microVMs.
== Bisect result ==
446fcce2a52b533c543dabba26777813c347577cis the first bad commit commit446fcce2a52b533c543dabba26777813c347577cAuthor: Lei Chen <[email protected]> Date: Tue Aug 19 23:20:26 2025 +0800Revert "x86: kvm: rate-limit global clock updates"
This reverts commit
7e44e4495a398eb553ce561f29f9148f40a3448f.Commit
7e44e4495a39("x86: kvm: rate-limit global clock updates") intends to use akvmclock_update_workto sync ntp corretion across all vcpus kvmclock, which is based on commit0061d53daf26f("KVM: x86: limit difference between kvmclock updates")Since kvmclock has been switched to mono raw, this commit can be reverted.
Signed-off-by Lei Chen <[email protected]>Signed-off-by Sean Christopherson <[email protected]>arch/x86/include/asm/kvm_host.h | 1 - arch/x86/kvm/x86.c | 29 ++++------------------------- 2 files changed, 4 insertions(+), 26 deletions(-)==== Symptoms ====
Measured on a KVM micro VM host running many Firecracker microVMs (
node_exportermetrics, 2026-03-20):kernel 6.19:
steal time inside guest VMs: 3–24% per vCPU (sustained) host system CPU (kernel mode): 3–12 CPUs saturated host steal: 3–8%kernel 6.18 (same host, same workload after rollback):
steal time inside guest VMs: < 0.02% per vCPU (~200x lower) host system CPU (kernel mode): 2–3 CPUs host steal: 0.3–0.5%==== Root cause (by AI analyze) ====
The regressing commit removes the rate-limiting from
kvm_gen_kvmclock_update(). Previously this function deferred the all-vCPU kick via a 100msdelayed_work:/* 6.18 */ static void kvm_gen_kvmclock_update(struct kvm_vcpu *v) { kvm_make_request(KVM_REQ_CLOCK_UPDATE, v); schedule_delayed_work(&kvm->arch.kvmclock_update_work, KVMCLOCK_UPDATE_DELAY); /* 100ms */ }After the revert it kicks every vCPU of the VM synchronously on every call:
/* 6.19 */ static void kvm_gen_kvmclock_update(struct kvm_vcpu *v) { kvm_for_each_vcpu(i, vcpu, kvm) { kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); kvm_vcpu_kick(vcpu); } }
KVM_REQ_GLOBAL_CLOCK_UPDATE, which callskvm_gen_kvmclock_update(), is issued on every vCPU load whenuse_master_clockis false (arch/x86/kvm/x86.c,kvm_vcpu_load):if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu == -1) kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu);With many Firecracker microVMs, the vCPU scheduling rate is high. Each scheduling event now IPIs every sibling vCPU of the VM, instead of coalescing all-vCPU kicks into at most one per 100ms. This creates a continuous IPI storm on the host, visible as high kernel (system) CPU time and high steal time inside guest VMs.
The commit justifies the removal with "Since kvmclock has been switched to mono raw, this commit can be reverted." That reasoning is correct for the NTP-correction use case, but the 100ms rate-limit also protected against IPI storms when
use_master_clockis false — a concern independent of clock source.==== Full bisect log ====
git bisect start
# status: waiting for both good and bad commits
# good: [7d0a66e4bb9081d75c82ec4957c50034cb0ea449] Linux 6.18 git bisect good7d0a66e4bb9081d75c82ec4957c50034cb0ea449
# status: waiting for bad commit, 1 good commit known
# bad: [05f7e89ab9731565d8a62e3b5d1ec206485eeb0b] Linux 6.19 git bisect bad05f7e89ab9731565d8a62e3b5d1ec206485eeb0b
# good: [02892f90a9851f508e557b3c75e93fc178310d5f] Merge tag 'hwmon-for-v6.19' ofgit://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging
git bisect good02892f90a9851f508e557b3c75e93fc178310d5f
# bad: [edf602a17b03e6bca31c48f34ac8fc3341503ac1] Merge tag 'tty-6.19-rc1' ofgit://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty
git bisect badedf602a17b03e6bca31c48f34ac8fc3341503ac1
# bad: [09cab48db950b6fb8c114314a20c0fd5a80cf990] Merge tag 'soc-arm-6.19' ofgit://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect bad09cab48db950b6fb8c114314a20c0fd5a80cf990
# good: [36492b7141b9abc967e92c991af32c670351dc16] Merge tag 'tracepoints-v6.19' ofgit://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
git bisect good36492b7141b9abc967e92c991af32c670351dc16
# good: [7cd122b55283d3ceef71a5b723ccaa03a72284b4] Merge tag 'pull-persistency' ofgit://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect good7cd122b55283d3ceef71a5b723ccaa03a72284b4
# bad: [63a9b0bc65d5d3ea96a57e7985ea22a8582fbbe5] Merge tag 'kvm-riscv-6.19-1' of https://github.com/kvm-riscv/linux into HEAD git bisect bad63a9b0bc65d5d3ea96a57e7985ea22a8582fbbe5
# bad: [adc99a6cfcf76d670272dea64bbc2d43ecd12a2f] Merge tag 'kvm-x86-mmu-6.19' of https://github.com/kvm-x86/linux into HEAD git bisect badadc99a6cfcf76d670272dea64bbc2d43ecd12a2f
# bad: [c09816f2afce0f89f176c4bc58dc57ec9f204998] KVM: x86: Remove unused declarationkvm_mmu_may_ignore_guest_pat()
git bisect badc09816f2afce0f89f176c4bc58dc57ec9f204998
# bad: [f6106d41ec84e552a5e8adda1f8741cab96a5425] x86/bugs: Use an x86 feature to track the MMIO Stale Data mitigation
git bisect badf6106d41ec84e552a5e8adda1f8741cab96a5425
# good: [9633f180ce994ab293ce4924a9b7aaf4673aa114] KVM: x86:
Explicitly set new periodic hrtimer expiration inapic_timer_fn()git bisect good9633f180ce994ab293ce4924a9b7aaf4673aa114
# bad: [e78fb96b41c6ac85c1a02c7e9610d1ebaa9b5d98] KVM: x86: remove comment about ntp correction sync for
git bisect bade78fb96b41c6ac85c1a02c7e9610d1ebaa9b5d98
# good: [a091fe60c2d3943b058132a64682a509d55bd325] KVM: x86: Grablapic_timerin a local variable to cleanup periodic code git bisect gooda091fe60c2d3943b058132a64682a509d55bd325
# bad: [446fcce2a52b533c543dabba26777813c347577c] Revert "x86: kvm:
rate-limit global clock updates"
git bisect bad446fcce2a52b533c543dabba26777813c347577c
# good: [43ddbf16edf5c1790684b32d5eb920a1b0eea285] Revert "x86: kvm:
introduce periodic global clock updates"
git bisect good43ddbf16edf5c1790684b32d5eb920a1b0eea285# first bad commit: [446fcce2a52b533c543dabba26777813c347577c] Revert "x86: kvm: rate-limit global clock updates"Best regards,
Jaroslav Pulchart
Sean Christopherson
maintainerQuoted 11 lines · Lei Chen
Hi Jaroslav,
I apologize for the late reply.
I have reviewed the code and identified two scenarios that currently trigger the
KVM_REQ_GLOBAL_CLOCK_UPDATErequest:Scenario 1:
kvm_write_system_time
This code path occurs when the hypervisor (such as QEMU) adjusts the time, or when the guest writes to the TSC.
Quoted 6 lines
Scenario 2: vcpu schedule in
kvm_arch_vcpu_load
If this function triggersKVM_REQ_GLOBAL_CLOCK_UPDATE, it indicates that the virtual machine is not using themaster_clock.Those two cases are uncommon. Could you please provide your dmesg and help check which code path triggers
KVM_REQ_GLOBAL_CLOCK_UPDATE?
I'm also mildly curious as to why KVM_REQ_GLOBAL_CLOCK_UPDATE is being triggered, but I don't know that it matters. E.g. fixing the underlying flaw (if one even exists) could fix Jaroslav's setup, but it won't fix setups where the "uncommon" cases are unavoidable, e.g. on setups that _can't_ use a master clock for whatever reason.
At a glance, explicitly ratelimiting kvm_gen_kvmclock_update() seems like the simplest option to address the regression.
PATCH 1/1 KVM: x86: Rate-limit global clock updates on vCPU load
commit 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting for KVM_REQ_GLOBAL_CLOCK_UPDATE.
As a result, kvm_arch_vcpu_load() can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.
Restore the throttling with a per-VM ratelimit state and gate KVM_REQ_GLOBAL_CLOCK_UPDATE through __ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.
--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/x86.c | 5 ++++- 2 files changed, 6 insertions(+), 1 deletion(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -1453,6 +1453,8 @@ struct kvm_arch {bool use_master_clock;u64 master_kernel_ns;u64 master_cycle_now; +/* how often to make KVM_REQ_GLOBAL_CLOCK_UPDATE on vcpu sched*/ +struct ratelimit_state kvmclock_update_rs;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;
arch/x86/kvm/x86.c +4 -1
@@ -5211,7 +5211,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)* kvmclock on vcpu->cpu migration */ if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) -kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs)) +kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +if (vcpu->cpu != cpu)kvm_make_request (KVM_REQ_MIGRATE_TIMER , vcpu); vcpu->cpu = cpu;
@@ -13189,6 +13191,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)raw_spin_lock_init (&kvm->arch.tsc_write_lock);mutex_init (&kvm->arch.apic_map_lock);seqcount_raw_spinlock_init (&kvm->arch.pvclock_sc, &kvm->arch.tsc_write_lock); +ratelimit_state_init (&kvm->arch.kvmclock_update_rs,HZ ,10 ); kvm->arch.kvmclock_offset = -get_kvmclock_base_ns ();raw_spin_lock_irqsave (&kvm->arch.tsc_write_lock, flags);
Signature · 1 line
2.51.0
Sean Christopherson
maintainerRe: [PATCH v1] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 26 lines · Lei Chen
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/x86.c | 5 ++++- 2 files changed, 6 insertions(+), 1 deletion(-)arch/x86/include/asm/kvm_host.h +0
-0
arch/x86/include/asm/kvm_host.h struct kvm_arch { · line 1456
Quoted 4 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -1453,6 +1453,8 @@ struct kvm_arch {bool use_master_clock;u64 master_kernel_ns;u64 master_cycle_now; +/* how often to make KVM_REQ_GLOBAL_CLOCK_UPDATE on vcpu sched*/
Eh, I would just omit this comment. If we want to document the ratelimit, the function comment above kvm_gen_kvmclock_update() is the best place for it.
Quoted 9 lines
arch/x86/include/asm/kvm_host.h +1
-0+struct ratelimit_state kvmclock_update_rs;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · line 5215
Quoted 6 lines
arch/x86/kvm/x86.c +2
-1@@ -5211,7 +5211,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)* kvmclock on vcpu->cpu migration */ if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) -kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs)) +kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu);
To maintain pre-revert compatibility, where KVM did this:
kvm_make_request(KVM_REQ_CLOCK_UPDATE, v); schedule_delayed_work(&kvm->arch.kvmclock_update_work, KVMCLOCK_UPDATE_DELAY);
the ratelimit should be on blasting KVM_REQ_CLOCK_UPDATE to *all* vCPUs, but KVM should still trigger KVM_REQ_CLOCK_UPDATE on the initiating vCPU so that the immediate update goes through.
That will also apply the ratelimiting to kvm_write_system_time(), though if a guest is changing system time that fast, it probably has other issues :-)
Completely untested, but this?
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 13 +++++-------- 2 files changed, 6 insertions(+), 8 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1504,6 +1504,7 @@ struct kvm_arch {bool use_master_clock;u64 master_kernel_ns;u64 master_cycle_now; +struct ratelimit_state kvmclock_update_rs;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;
arch/x86/kvm/x86.c +5 -8
@@ -3522,16 +3522,12 @@ uint64_t kvm_get_wall_clock_epoch(struct kvm *kvm)* The worst case for a remote vcpu to update its kvmclock * is then bounded by maximum nohz sleep latency. */ -static void kvm_gen_kvmclock_update (struct kvm_vcpu *v) +static void kvm_gen_kvmclock_update (struct kvm_vcpu *vcpu) { -unsigned long i; -struct kvm_vcpu *vcpu; -struct kvm *kvm = v->kvm; - -kvm_for_each_vcpu (i, vcpu, kvm) { +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs))kvm_make_request (KVM_REQ_CLOCK_UPDATE , vcpu); -kvm_vcpu_kick (vcpu); - } +else +kvm_make_all_cpus_request (vcpu->kvm,KVM_REQ_CLOCK_UPDATE ); }/* These helpers are safe iff @msr is known to be an MCx bank MSR. */
@@ -13366,6 +13362,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)raw_spin_lock_init (&kvm->arch.tsc_write_lock);mutex_init (&kvm->arch.apic_map_lock);seqcount_raw_spinlock_init (&kvm->arch.pvclock_sc, &kvm->arch.tsc_write_lock); +ratelimit_state_init (&kvm->arch.kvmclock_update_rs,HZ ,10 ); kvm->arch.kvmclock_offset = -get_kvmclock_base_ns ();raw_spin_lock_irqsave (&kvm->arch.tsc_write_lock, flags);
base-commit: b89df297a47e641581ee67793592e5c6ae0428f4
Re: [PATCH v1] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 59 lines · Sean Christopherson
Quoted 26 lines · Lei Chen
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/x86.c | 5 ++++- 2 files changed, 6 insertions(+), 1 deletion(-)arch/x86/include/asm/kvm_host.h +0
-0arch/x86/include/asm/kvm_host.h struct kvm_arch { · line 1456
Quoted 4 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -1453,6 +1453,8 @@ struct kvm_arch {bool use_master_clock;u64 master_kernel_ns;u64 master_cycle_now; +/* how often to make KVM_REQ_GLOBAL_CLOCK_UPDATE on vcpu sched*/ Eh, I would just omit this comment. If we want to document the ratelimit, the function comment above
kvm_gen_kvmclock_update()is the best place for it.Quoted 9 lines
arch/x86/include/asm/kvm_host.h +1
-0+struct ratelimit_state kvmclock_update_rs;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · line 5215
Quoted 6 lines
arch/x86/kvm/x86.c +2
-1@@ -5211,7 +5211,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)* kvmclock on vcpu->cpu migration */ if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) -kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs)) +kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu);To maintain pre-revert compatibility, where KVM did this:
kvm_make_request(KVM_REQ_CLOCK_UPDATE, v); schedule_delayed_work(&kvm->arch.kvmclock_update_work, KVMCLOCK_UPDATE_DELAY);
Quoted 7 lines
the ratelimit should be on blasting
KVM_REQ_CLOCK_UPDATEto *all* vCPUs, but KVM should still triggerKVM_REQ_CLOCK_UPDATEon the initiating vCPU so that the immediate update goes through.That will also apply the ratelimiting to
kvm_write_system_time(), though if a guest is changing system time that fast, it probably has other issues :-)
This patch does not impact kvm_write_system_time, which works as follows:
kvm_write_system_time
kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu);
vcpu_enter_guest
if (kvm_check_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu))
kvm_gen_kvmclock_update(vcpu);
kvm_for_each_vcpu
kvm_vcpu_kick(vcpu);This patch limits the rate of GLOBAL_CLOCK_UPDATE only in kvm_arch_vcpu_load.
Maybe I missed something?
Quoted 55 lines
Completely untested, but this?
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 13 +++++-------- 2 files changed, 6 insertions(+), 8 deletions(-)arch/x86/include/asm/kvm_host.h +1
-0@@ -1504,6 +1504,7 @@ struct kvm_arch {bool use_master_clock;u64 master_kernel_ns;u64 master_cycle_now; +struct ratelimit_state kvmclock_update_rs;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;arch/x86/kvm/x86.c +5
-8@@ -3522,16 +3522,12 @@ uint64_t kvm_get_wall_clock_epoch(struct kvm *kvm)* The worst case for a remote vcpu to update its kvmclock * is then bounded by maximum nohz sleep latency. */ -static void kvm_gen_kvmclock_update (struct kvm_vcpu *v) +static void kvm_gen_kvmclock_update (struct kvm_vcpu *vcpu) { -unsigned long i; -struct kvm_vcpu *vcpu; -struct kvm *kvm = v->kvm; - -kvm_for_each_vcpu (i, vcpu, kvm) { +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs))kvm_make_request (KVM_REQ_CLOCK_UPDATE , vcpu); -kvm_vcpu_kick (vcpu); - } +else +kvm_make_all_cpus_request (vcpu->kvm,KVM_REQ_CLOCK_UPDATE ); }/* These helpers are safe iff @msr is known to be an MCx bank MSR. */ @@ -13366,6 +13362,7 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)raw_spin_lock_init (&kvm->arch.tsc_write_lock);mutex_init (&kvm->arch.apic_map_lock);seqcount_raw_spinlock_init (&kvm->arch.pvclock_sc, &kvm->arch.tsc_write_lock); +ratelimit_state_init (&kvm->arch.kvmclock_update_rs,HZ ,10 ); kvm->arch.kvmclock_offset = -get_kvmclock_base_ns ();raw_spin_lock_irqsave (&kvm->arch.tsc_write_lock, flags);base-commit:
b89df297a47e641581ee67793592e5c6ae0428f4--
Re: [PATCH v1] KVM: x86: Rate-limit global clock updates on vCPU load
In reply to Sean ChristophersonHi Sean,
Thanks for your review.
Quoted 23 lines · Sean Christopherson
Quoted 21 lines · Lei Chen
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>--- arch/x86/include/asm/kvm_host.h | 2 ++ arch/x86/kvm/x86.c | 5 ++++- 2 files changed, 6 insertions(+), 1 deletion(-)
Quoted 13 lines
arch/x86/include/asm/kvm_host.h struct kvm_arch { · line 1456
Quoted 9 lines
arch/x86/include/asm/kvm_host.h +1
-0@@ -1453,6 +1453,8 @@ struct kvm_arch {bool use_master_clock;u64 master_kernel_ns;u64 master_cycle_now; +/* how often to make KVM_REQ_GLOBAL_CLOCK_UPDATE on vcpu sched*/ Eh, I would just omit this comment. If we want to document the ratelimit, the function comment above
kvm_gen_kvmclock_update()is the best place for it.
OK, I'll remove this comment in the next patch.
Quoted 18 lines
Quoted 9 lines
arch/x86/include/asm/kvm_host.h +1
-0+struct ratelimit_state kvmclock_update_rs;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · line 5215
Quoted 6 lines
arch/x86/kvm/x86.c +2
-1@@ -5211,7 +5211,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)* kvmclock on vcpu->cpu migration */ if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) -kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs)) +kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu);To maintain pre-revert compatibility, where KVM did this:
Quoted 9 lines
kvm_make_request(KVM_REQ_CLOCK_UPDATE, v); schedule_delayed_work(&kvm->arch.kvmclock_update_work, KVMCLOCK_UPDATE_DELAY);the ratelimit should be on blasting
KVM_REQ_CLOCK_UPDATEto *all* vCPUs, but KVM should still triggerKVM_REQ_CLOCK_UPDATEon the initiating vCPU so that the immediate update goes through.
Yes, this is a mistake, I'll fix it in the next patch.
Quoted 3 lines
That will also apply the ratelimiting to
kvm_write_system_time(), though if a guest is changing system time that fast, it probably has other issues :-)
This patch does not impact kvm_write_system_time, which works as follows:
kvm_write_system_time
kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu);
vcpu_enter_guest
if (kvm_check_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu))
kvm_gen_kvmclock_update(vcpu);
kvm_for_each_vcpu
kvm_vcpu_kick(vcpu);This patch limits the rate of GLOBAL_CLOCK_UPDATE only in kvm_arch_vcpu_load.
Maybe I missed something?
Best Regards,
Lei Chen
PATCH v2 1/1 KVM: x86: Rate-limit global clock updates on vCPU load
In reply to Sean Christophersoncommit 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting for KVM_REQ_GLOBAL_CLOCK_UPDATE.
As a result, kvm_arch_vcpu_load() can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.
Restore the throttling with a per-VM ratelimit state and gate KVM_REQ_GLOBAL_CLOCK_UPDATE through __ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.
---
CHANGELOG:
v2:
- remove comment of kvmclock_update_rs
- make sure kvm_arch_vcpu_load make KVM_REQ_CLOCK_UPDATE for this vcpu
- add RATELIMIT_MSG_ON_RELEASE to kvmclock_update_rs
v1:
- initial version(https://lore.kernel.org/all/[email protected]/)
--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1453,6 +1453,7 @@ struct kvm_arch {bool use_master_clock;u64 master_kernel_ns;u64 master_cycle_now; +struct ratelimit_state kvmclock_update_rs;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;
arch/x86/kvm/x86.c +9 -2
@@ -5210,8 +5210,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)* On a host with synchronized TSC, there is no need to update * kvmclock on vcpu->cpu migration */ -if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) -kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) { +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs)) +kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +else +kvm_make_request (KVM_REQ_CLOCK_UPDATE , vcpu); + } +if (vcpu->cpu != cpu)kvm_make_request (KVM_REQ_MIGRATE_TIMER , vcpu); vcpu->cpu = cpu;
@@ -13189,6 +13194,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)raw_spin_lock_init (&kvm->arch.tsc_write_lock);mutex_init (&kvm->arch.apic_map_lock);seqcount_raw_spinlock_init (&kvm->arch.pvclock_sc, &kvm->arch.tsc_write_lock); +ratelimit_state_init (&kvm->arch.kvmclock_update_rs,HZ ,10 ); +ratelimit_set_flags (&kvm->arch.kvmclock_update_rs,RATELIMIT_MSG_ON_RELEASE ); kvm->arch.kvmclock_offset = -get_kvmclock_base_ns ();raw_spin_lock_irqsave (&kvm->arch.tsc_write_lock, flags);
Signature · 1 line
2.51.0
Sean Christopherson
maintainerRe: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 47 lines · Lei Chen
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>---
CHANGELOG:
v2:
- remove comment ofkvmclock_update_rs
- make surekvm_arch_vcpu_loadmakeKVM_REQ_CLOCK_UPDATEfor this vcpu
- addRATELIMIT_MSG_ON_RELEASEtokvmclock_update_rsv1:
- initial version(https://lore.kernel.org/all/[email protected]/)--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +1
-0@@ -1453,6 +1453,7 @@ struct kvm_arch {bool use_master_clock;u64 master_kernel_ns;u64 master_cycle_now; +struct ratelimit_state kvmclock_update_rs;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;arch/x86/kvm/x86.c +0
-0
arch/x86/kvm/x86.c void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · line 5217
Quoted 10 lines
arch/x86/kvm/x86.c +5
-2@@ -5210,8 +5210,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)* On a host with synchronized TSC, there is no need to update * kvmclock on vcpu->cpu migration */ -if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) -kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) { +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs)) +kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +else +kvm_make_request (KVM_REQ_CLOCK_UPDATE , vcpu);
What I was trying to call out in my review of v1, is that prior to commit 446fcce2a52b, the effectively ratelimiting applied to *all* instances of KVM_REQ_GLOBAL_CLOCK_UPDATE. Which meant that KVM's existing behavior is that kvm_write_system_time() would be subject to the ratelimiting as well.
That said, I don't see any obvious problems with immediately honoring writes to MSR_KVM_SYSTEM_TIME{,_NEW}, and it's probably a much better experience for the guest. So I'm a-ok with this approach, but we should call out that skipping the synthetic MSR case is deliberate. No need for a v3, I'll add a blurb when applying.
arch/x86/kvm/x86.c int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) · line 13198
Quoted 11 lines
arch/x86/kvm/x86.c +4
-0+ } +if (vcpu->cpu != cpu)kvm_make_request (KVM_REQ_MIGRATE_TIMER , vcpu); vcpu->cpu = cpu;@@ -13189,6 +13194,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)raw_spin_lock_init (&kvm->arch.tsc_write_lock);mutex_init (&kvm->arch.apic_map_lock);seqcount_raw_spinlock_init (&kvm->arch.pvclock_sc, &kvm->arch.tsc_write_lock); +ratelimit_state_init (&kvm->arch.kvmclock_update_rs,HZ ,10 ); +ratelimit_set_flags (&kvm->arch.kvmclock_update_rs,RATELIMIT_MSG_ON_RELEASE );
IIUC, so long was KVM doesn't explicitly invoke ratelimit_state_exit(), setting RATELIMIT_MSG_ON_RELEASE means we won't get dmesg spam? To be clear, I'm 100% in favor of suppressing dmesg output.
Quoted 6 lines
arch/x86/kvm/x86.c +0
-0kvm->arch.kvmclock_offset = -get_kvmclock_base_ns ();raw_spin_lock_irqsave (&kvm->arch.tsc_write_lock, flags);Signature · 1 line
2.51.0
Re: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 13 lines · Sean Christopherson
Quoted 12 lines · Lei Chen
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.
Quoted 4 lines
Quoted 4 lines
Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>
Was this performance regression ever addressed? Looks like this fall through the cracks, but it's easy to miss something.
Ciao, Thorsten
Quoted 76 lines
Quoted 31 lines
---
CHANGELOG:
v2:
- remove comment ofkvmclock_update_rs
- make surekvm_arch_vcpu_loadmakeKVM_REQ_CLOCK_UPDATEfor this vcpu
- addRATELIMIT_MSG_ON_RELEASEtokvmclock_update_rsv1:
- initial version(https://lore.kernel.org/all/[email protected]/)--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +1
-0@@ -1453,6 +1453,7 @@ struct kvm_arch {bool use_master_clock;u64 master_kernel_ns;u64 master_cycle_now; +struct ratelimit_state kvmclock_update_rs;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;arch/x86/kvm/x86.c +0
-0arch/x86/kvm/x86.c void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · line 5217
Quoted 10 lines
arch/x86/kvm/x86.c +5
-2@@ -5210,8 +5210,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)* On a host with synchronized TSC, there is no need to update * kvmclock on vcpu->cpu migration */ -if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) -kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) { +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs)) +kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +else +kvm_make_request (KVM_REQ_CLOCK_UPDATE , vcpu);What I was trying to call out in my review of v1, is that prior to commit
446fcce2a52b, the effectively ratelimiting applied to *all* instances ofKVM_REQ_GLOBAL_CLOCK_UPDATE. Which meant that KVM's existing behavior is thatkvm_write_system_time()would be subject to the ratelimiting as well.That said, I don't see any obvious problems with immediately honoring writes to MSR_KVM_SYSTEM_TIME{,_NEW}, and it's probably a much better experience for the guest. So I'm a-ok with this approach, but we should call out that skipping the synthetic MSR case is deliberate. No need for a v3, I'll add a blurb when applying.
arch/x86/kvm/x86.c int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) · line 13198
Quoted 11 lines
arch/x86/kvm/x86.c +4
-0+ } +if (vcpu->cpu != cpu)kvm_make_request (KVM_REQ_MIGRATE_TIMER , vcpu); vcpu->cpu = cpu;@@ -13189,6 +13194,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)raw_spin_lock_init (&kvm->arch.tsc_write_lock);mutex_init (&kvm->arch.apic_map_lock);seqcount_raw_spinlock_init (&kvm->arch.pvclock_sc, &kvm->arch.tsc_write_lock); +ratelimit_state_init (&kvm->arch.kvmclock_update_rs,HZ ,10 ); +ratelimit_set_flags (&kvm->arch.kvmclock_update_rs,RATELIMIT_MSG_ON_RELEASE );IIUC, so long was KVM doesn't explicitly invoke
ratelimit_state_exit(), settingRATELIMIT_MSG_ON_RELEASEmeans we won't get dmesg spam? To be clear, I'm 100% in favor of suppressing dmesg output.Quoted 6 lines
arch/x86/kvm/x86.c +0
-0kvm->arch.kvmclock_offset = -get_kvmclock_base_ns ();raw_spin_lock_irqsave (&kvm->arch.tsc_write_lock, flags);Signature · 1 line
2.51.0
Sean Christopherson
maintainerRe: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 14 lines · Thorsten Leemhuis
Quoted 13 lines · Sean Christopherson
Quoted 12 lines · Lei Chen
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>Was this performance regression ever addressed?
Nope, not yet.
Quoted 1 line
Looks like this fall through the cracks, but it's easy to miss something.
It's in my list of patches to apply (probably for 7.2?). I didn't want to squeeze it into the initial 7.1 pull request for a variety of reasons.
Re: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 20 lines · Sean Christopherson
Quoted 19 lines · Thorsten Leemhuis
Quoted 17 lines · Sean Christopherson
Quoted 16 lines · Lei Chen
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>
Quoted 7 lines
Quoted 1 line
Was this performance regression ever addressed?
Nope, not yet.
Quoted 1 line
Looks like this fall through the cracks, but it's easy to miss something.
It's in my list of patches to apply (probably for 7.2?). I didn't want to squeeze it into the initial 7.1 pull request for a variety of reasons.
Hmmm. CCing Linus so he can speak up if he wants to about the following:
Given that this is a fix for a performance regression[1] I'd say it's not as urgent as a "something stopped working" case -- so I guess it's something where the "[fix] "within a week", preferably before the next rc" approach Linus recently mentioned does not need to be applied strictly.
But Jaroslav OTOH reported it more than 7 weeks ago already and back then called it something that "severely impacts KVM hosts running many Firecracker microVMs."[1]; and a potential fix exists for 4 weeks already. Due to that, 7.2 feels a bit too far away for me, as that is still ~15 weeks away. But maybe that's just me.
Ciao, Thorsten
[1]
https://lore.kernel.org/all/CAK8fFZ5gY8_Mw2A=iZVFNVKQNrXQzVsn-HTd+Me9K6ZfmdgA+Q@mail.gmail.com/
Sean Christopherson
maintainerRe: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 31 lines · Thorsten Leemhuis
Quoted 20 lines · Sean Christopherson
Quoted 19 lines · Thorsten Leemhuis
Quoted 17 lines · Sean Christopherson
Quoted 16 lines · Lei Chen
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>Quoted 7 lines
Quoted 1 line
Was this performance regression ever addressed?
Nope, not yet.
Quoted 1 line
Looks like this fall through the cracks, but it's easy to miss something.
It's in my list of patches to apply (probably for 7.2?). I didn't want to squeeze it into the initial 7.1 pull request for a variety of reasons.
Hmmm. CCing Linus so he can speak up if he wants to about the following:
Quoted 8 lines
Given that this is a fix for a performance regression[1] I'd say it's not as urgent as a "something stopped working" case -- so I guess it's something where the "[fix] "within a week", preferably before the next rc" approach Linus recently mentioned does not need to be applied strictly.
But Jaroslav OTOH reported it more than 7 weeks ago already and back then called it something that "severely impacts KVM hosts running many Firecracker microVMs."[1];
For a setup that is likely broken. On modern hardware, the path in question should never actually be hit. I do want to resolve the bug since older hardware and funky setups do rely on the old behavior, but it's not pants-on-fire urgent.
More importantly, the original reporter(s) hasn't responded to any of our questions, or to the proposed fix. I'm not going to rush in a fix if I don't actually *know* it's going to fix the original problem.
Quoted 3 lines
and a potential fix exists for 4 weeks already. Due to that, 7.2 feels a bit too far away for me, as that is still ~15 weeks away. But maybe that's just me.
The "user" is also a fairly sizeable company, not some random person that's trying to use KVM and is blocked. I highly doubt they are still actually running a buggy kernel. E.g. based on a "same workload after rollback" comment in the bug report, I assume they simply rolled back to the last good kernel (6.18).
Who knows, maybe they also took our hints/suggestions about theire setup being wonky and addressed whatever hiccup was sending them down the uncommon, already- slow path.
All in all, AFAICT the only difference between sending this into 7.1 vs. 7.2 is that the reporter won't be able to upgrade their kernel (without patching) for an extra ~8 weeks.
Re: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 42 lines · Thorsten Leemhuis
Quoted 31 lines · Sean Christopherson
Quoted 20 lines · Thorsten Leemhuis
Quoted 19 lines · Sean Christopherson
Quoted 17 lines · Lei Chen
Quoted 16 lines
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>Quoted 7 lines
Quoted 1 line
Was this performance regression ever addressed?
Nope, not yet.
Quoted 1 line
Looks like this fall through the cracks, but it's easy to miss something.
It's in my list of patches to apply (probably for 7.2?). I didn't want to squeeze it into the initial 7.1 pull request for a variety of reasons.
Hmmm. CCing Linus so he can speak up if he wants to about the following:
Quoted 8 lines
Given that this is a fix for a performance regression[1] I'd say it's not as urgent as a "something stopped working" case -- so I guess it's something where the "[fix] "within a week", preferably before the next rc" approach Linus recently mentioned does not need to be applied strictly.
But Jaroslav OTOH reported it more than 7 weeks ago already and back then called it something that "severely impacts KVM hosts running many Firecracker microVMs."[1];
Quoted 7 lines
For a setup that is likely broken. On modern hardware, the path in question should never actually be hit. I do want to resolve the bug since older hardware and funky setups do rely on the old behavior, but it's not pants-on-fire urgent.
More importantly, the original reporter(s) hasn't responded to any of our questions, or to the proposed fix. I'm not going to rush in a fix if I don't actually *know* it's going to fix the original problem.
Hi Sean, Thorsten,
sorry for the missing response from my side, this thread unfortunately ended up in trash due to mail filters on my side and I completely missed it. I currently don't have the full context loaded back in yet, but I'll re-read the thread and follow up properly once I do.
For additional context, we are currently running the latest 6.19/7.0.y kernels with a revert of the commits causing the reported regression, and the hardware is AMD EPYC 9454P 48-Core Processor.
Jaroslav
Quoted 17 lines
Quoted 4 lines
and a potential fix exists for 4 weeks already. Due to that, 7.2 feels a bit too far away for me, as that is still ~15 weeks away. But maybe that's just me.
The "user" is also a fairly sizeable company, not some random person that's trying to use KVM and is blocked. I highly doubt they are still actually running a buggy kernel. E.g. based on a "same workload after rollback" comment in the bug report, I assume they simply rolled back to the last good kernel (6.18).
Who knows, maybe they also took our hints/suggestions about theire setup being wonky and addressed whatever hiccup was sending them down the uncommon, already- slow path.
All in all, AFAICT the only difference between sending this into 7.1 vs. 7.2 is that the reporter won't be able to upgrade their kernel (without patching) for an extra ~8 weeks.
Sean Christopherson
maintainerRe: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 49 lines · Jaroslav Pulchart
Quoted 41 lines · Thorsten Leemhuis
Quoted 31 lines · Sean Christopherson
Quoted 20 lines · Thorsten Leemhuis
Quoted 19 lines · Sean Christopherson
Quoted 17 lines · Lei Chen
Quoted 16 lines
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>Quoted 7 lines
Quoted 1 line
Was this performance regression ever addressed?
Nope, not yet.
Quoted 1 line
Looks like this fall through the cracks, but it's easy to miss something.
It's in my list of patches to apply (probably for 7.2?). I didn't want to squeeze it into the initial 7.1 pull request for a variety of reasons.
Hmmm. CCing Linus so he can speak up if he wants to about the following:
Quoted 8 lines
Given that this is a fix for a performance regression[1] I'd say it's not as urgent as a "something stopped working" case -- so I guess it's something where the "[fix] "within a week", preferably before the next rc" approach Linus recently mentioned does not need to be applied strictly.
But Jaroslav OTOH reported it more than 7 weeks ago already and back then called it something that "severely impacts KVM hosts running many Firecracker microVMs."[1];
Quoted 7 lines
For a setup that is likely broken. On modern hardware, the path in question should never actually be hit. I do want to resolve the bug since older hardware and funky setups do rely on the old behavior, but it's not pants-on-fire urgent.
More importantly, the original reporter(s) hasn't responded to any of our questions, or to the proposed fix. I'm not going to rush in a fix if I don't actually *know* it's going to fix the original problem.
Quoted 5 lines
Hi Sean, Thorsten,
sorry for the missing response from my side, this thread unfortunately ended up in trash due to mail filters on my side and I completely missed it.
No worries, gmail's Spam filter is my nemesis :-)
Quoted 2 lines
I currently don't have the full context loaded back in yet, but I'll re-read the thread and follow up properly once I do.
I think the only remaining question is why/how KVM's master clock is getting disabled. But that's more of a question for your deployment than it is a question for upstream; it's possible there's a different KVM bug lurking, but it's more likely that something in your setup is incompatible with using the master clock.
Note, it's certainly not "wrong" for the master clock to be disabled, but it's quite suprising, especially for Firecracker VMs. It's worth investigating as there might be an underlying issue that's very easy to address, and "fixing" it should provide (very) small performance benefits.
Re: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 67 lines · Jaroslav Pulchart
Quoted 49 lines
Quoted 41 lines · Thorsten Leemhuis
Quoted 31 lines · Sean Christopherson
Quoted 20 lines · Thorsten Leemhuis
Quoted 19 lines · Sean Christopherson
Quoted 17 lines · Lei Chen
Quoted 16 lines
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>Quoted 7 lines
Quoted 1 line
Was this performance regression ever addressed?
Nope, not yet.
Quoted 1 line
Looks like this fall through the cracks, but it's easy to miss something.
It's in my list of patches to apply (probably for 7.2?). I didn't want to squeeze it into the initial 7.1 pull request for a variety of reasons.
Hmmm. CCing Linus so he can speak up if he wants to about the following:
Quoted 8 lines
Given that this is a fix for a performance regression[1] I'd say it's not as urgent as a "something stopped working" case -- so I guess it's something where the "[fix] "within a week", preferably before the next rc" approach Linus recently mentioned does not need to be applied strictly.
But Jaroslav OTOH reported it more than 7 weeks ago already and back then called it something that "severely impacts KVM hosts running many Firecracker microVMs."[1];
Quoted 7 lines
For a setup that is likely broken. On modern hardware, the path in question should never actually be hit. I do want to resolve the bug since older hardware and funky setups do rely on the old behavior, but it's not pants-on-fire urgent.
More importantly, the original reporter(s) hasn't responded to any of our questions, or to the proposed fix. I'm not going to rush in a fix if I don't actually *know* it's going to fix the original problem.
Quoted 5 lines
Hi Sean, Thorsten,
sorry for the missing response from my side, this thread unfortunately ended up in trash due to mail filters on my side and I completely missed it.
No worries, gmail's Spam filter is my nemesis :-)
Quoted 2 lines
I currently don't have the full context loaded back in yet, but I'll re-read the thread and follow up properly once I do.
I think the only remaining question is why/how KVM's master clock is getting disabled. But that's more of a question for your deployment than it is a question for upstream; it's possible there's a different KVM bug lurking, but it's more likely that something in your setup is incompatible with using the master clock.
Quoted 4 lines
Note, it's certainly not "wrong" for the master clock to be disabled, but it's quite suprising, especially for Firecracker VMs. It's worth investigating as there might be an underlying issue that's very easy to address, and "fixing" it should provide (very) small performance benefits.
I've dug into the "master clock question" and have an idea.
Our Firecracker hosts are themselves L1 KVM VMs (nested virtualisation) running on AMD EPYC 9454P and EPYC 9455 hardware. Even though the compute nodes use cpu_mode=host-passthrough in qemu kvm, the invtsc CPUID bit is filtered out by QEMU, which I hadn't realized. Without it the guest kernel marks the TSC unstable at boot:
tsc: Marking TSC unstable due to TSCs unsynchronized
and falls back to kvm-clock as its clocksource.
I suppose that in turn prevents KVM from enabling the master clock for any L2 guests (the Firecracker microVMs), am I right?
I have resolved the issue by explicitly adding +invtsc to cpu_model_extra_flags in our OpenStack nova.conf. After this change the L1 VMs now correctly show constant_tsc and nonstop_tsc in /proc/cpuinfo and switch clocksource to tsc. I also confirmed the IPI storm disappears without the v2 patch when +invtsc is present, and returns when it is absent on a vanilla 7.0.3 kernel.
So could this be the answer to your question: "the master clock was disabled because QEMU silently drops invtsc even in host-passthrough mode"?
Sean Christopherson
maintainerRe: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 32 lines · Jaroslav Pulchart
Quoted 11 lines · Jaroslav Pulchart
Quoted 1 line
Quoted 1 line · Thorsten Leemhuis
I think the only remaining question is why/how KVM's master clock is getting disabled. But that's more of a question for your deployment than it is a question for upstream; it's possible there's a different KVM bug lurking, but it's more likely that something in your setup is incompatible with using the master clock.
Note, it's certainly not "wrong" for the master clock to be disabled, but it's quite suprising, especially for Firecracker VMs. It's worth investigating as there might be an underlying issue that's very easy to address, and "fixing" it should provide (very) small performance benefits.
I've dug into the "master clock question" and have an idea.
Our Firecracker hosts are themselves L1 KVM VMs (nested virtualisation) running on AMD EPYC 9454P and EPYC 9455 hardware. Even though the compute nodes use cpu_mode=host-passthrough in qemu kvm, the invtsc CPUID bit is filtered out by QEMU, which I hadn't realized. Without it the guest kernel marks the TSC unstable at boot:
tsc: Marking TSC unstable due to TSCs unsynchronized
and falls back to kvm-clock as its clocksource.I suppose that in turn prevents KVM from enabling the master clock for any L2 guests (the Firecracker microVMs), am I right?
I have resolved the issue by explicitly adding +invtsc to
cpu_model_extra_flagsin our OpenStack nova.conf. After this change the L1 VMs now correctly showconstant_tscandnonstop_tscin/proc/cpuinfoand switch clocksource to tsc. I also confirmed the IPI storm disappears without the v2 patch when +invtsc is present, and returns when it is absent on a vanilla 7.0.3 kernel.
Quoted 3 lines
So could this be the answer to your question: "the master clock was disabled because QEMU silently drops invtsc even in host-passthrough mode"?
Yep, that'd do it. Linux-as-a-guest will prefer kvmclock over TSC if the TSC
isnt constant and non-stop. That in turn will prevent KVM (as the L1 hypervisor) from using the master clock, since it sees the kernel clocksource as not being (directly) based on TSC.
Thanks for the follow-up!
Re: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
In reply to Lei ChenQuoted 73 lines
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.Restore the throttling with a per-VM ratelimit state and gate
KVM_REQ_GLOBAL_CLOCK_UPDATEthrough__ratelimit(), so frequent vCPU scheduling does not generate a steady stream of redundant clock update requests.Fixes 446fcce2a52b ("Revert "x86: kvm: rate-limit global clock updates"")Signed-off-by Lei Chen <[email protected]>Reported-by Jaroslav Pulchart <[email protected]>---
CHANGELOG:
v2:
- remove comment ofkvmclock_update_rs
- make surekvm_arch_vcpu_loadmakeKVM_REQ_CLOCK_UPDATEfor this vcpu
- addRATELIMIT_MSG_ON_RELEASEtokvmclock_update_rsv1:
- initial version(https://lore.kernel.org/all/[email protected]/)--- arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/x86.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)arch/x86/include/asm/kvm_host.h +1
-0@@ -1453,6 +1453,7 @@ struct kvm_arch {bool use_master_clock;u64 master_kernel_ns;u64 master_cycle_now; +struct ratelimit_state kvmclock_update_rs;#ifdef CONFIG_KVM_HYPERV struct kvm_hv hyperv;arch/x86/kvm/x86.c +9
-3@@ -5210,8 +5210,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)* On a host with synchronized TSC, there is no need to update * kvmclock on vcpu->cpu migration */ -if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) -kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) { +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs)) +kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +else +kvm_make_request (KVM_REQ_CLOCK_UPDATE , vcpu); + } +if (vcpu->cpu != cpu)kvm_make_request (KVM_REQ_MIGRATE_TIMER , vcpu); vcpu->cpu = cpu;@@ -13189,6 +13194,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)raw_spin_lock_init (&kvm->arch.tsc_write_lock);mutex_init (&kvm->arch.apic_map_lock);seqcount_raw_spinlock_init (&kvm->arch.pvclock_sc, &kvm->arch.tsc_write_lock); +ratelimit_state_init (&kvm->arch.kvmclock_update_rs,HZ ,10 ); +ratelimit_set_flags (&kvm->arch.kvmclock_update_rs,RATELIMIT_MSG_ON_RELEASE ); kvm->arch.kvmclock_offset = -get_kvmclock_base_ns ();raw_spin_lock_irqsave (&kvm->arch.tsc_write_lock, flags); --
Quoted 2 lines
2.51.0
Hi Lei, I tested the v2 patch on a Firecracker host running vanilla kernel 7.0.3 with this patch applied, the IPI storm on vCPU load is gone!
Re: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
In reply to Lei ChenQuoted 8 lines · Lei Chen
commit
446fcce2a52b("Revert "x86: kvm: rate-limit global clock updates"") dropped the rate limiting forKVM_REQ_GLOBAL_CLOCK_UPDATE.As a result,
kvm_arch_vcpu_load()can queue global clock update requests every time a vCPU is scheduled when the master clock is disabled or when the vCPU is loaded for the first time.[...]
Applied to kvm-x86 fixes, thanks!
[1/1] KVM: x86: Rate-limit global clock updates on vCPU load
https://github.com/kvm-x86/linux/commit/8fe2e698fce4
Re: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
In reply to Lei ChenQuoted 9 lines · Lei Chen
arch/x86/kvm/x86.c +0
-2@@ -5210,8 +5210,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)* On a host with synchronized TSC, there is no need to update * kvmclock on vcpu->cpu migration */ -if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) -kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu);
arch/x86/kvm/x86.c void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · line 5222
Quoted 10 lines
arch/x86/kvm/x86.c +7
-0+if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) { +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs)) +kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +else +kvm_make_request (KVM_REQ_CLOCK_UPDATE , vcpu); + } + if (vcpu->cpu != cpu) kvm_make_request(KVM_REQ_MIGRATE_TIMER, vcpu); vcpu->cpu = cpu;
I don't like this.
We do the GLOBAL_CLOCK_UPDATE in non-masterclock mode because theoretically, a *failure* to do so could cause the guest to observe non-monotonicity across vCPUs.
With this rate-limiting, we *sometimes* protect against that non- monotonicity. Why even bother? Why not just rip it out completely, in that case? AIUI the guest should cope with anyway if its PVCLOCK_TSC_STABLE_BIT isn't set, and enforce monotonicity for itself side in __pvclock_clocksource_read() ?
But also, I would *love* to kill non-masterclock mode completely.
In this nested case, the L1 TSCs *are* actually in sync; it just don't have the +invtsc CPUID to *promise* that they will be.
Could we find a way for L1 KVM to use masterclock mode for its nested L2 VMs even in this case? Could we make it use PVCLOCK_TSC_STABLE_BIT to enable masterclock mode, instead?
At at the very least, could we find a way for L1 to trigger the GLOBAL_CLOCK_UPDATE only if an *actual* discrepancy in its TSCs is found? (Or, if vdso mode is not VDSO_CLOCKMODE_PVCLOCK?)
Sean Christopherson
maintainerRe: [PATCH v2] KVM: x86: Rate-limit global clock updates on vCPU load
Quoted 14 lines · David Woodhouse
Quoted 13 lines · Lei Chen
arch/x86/kvm/x86.c +4
-2@@ -5210,8 +5210,13 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)* On a host with synchronized TSC, there is no need to update * kvmclock on vcpu->cpu migration */ -if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) -kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +if (!vcpu->kvm->arch.use_master_clock || vcpu->cpu ==-1 ) { +if (__ratelimit (&vcpu->kvm->arch.kvmclock_update_rs)) +kvm_make_request (KVM_REQ_GLOBAL_CLOCK_UPDATE , vcpu); +else
Quoted 8 lines
arch/x86/kvm/x86.c void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) · line 5222
Quoted 6 lines
arch/x86/kvm/x86.c +3
-0+kvm_make_request (KVM_REQ_CLOCK_UPDATE , vcpu); + } + if (vcpu->cpu != cpu) kvm_make_request(KVM_REQ_MIGRATE_TIMER, vcpu); vcpu->cpu = cpu;I don't like this.
I don't think anyone likes this :-)
Quoted 9 lines
We do the
GLOBAL_CLOCK_UPDATEin non-masterclock mode because theoretically, a *failure* to do so could cause the guest to observe non-monotonicity across vCPUs.With this rate-limiting, we *sometimes* protect against that non- monotonicity. Why even bother? Why not just rip it out completely, in that case? AIUI the guest should cope with anyway if its
PVCLOCK_TSC_STABLE_BITisn't set, and enforce monotonicity for itself side in__pvclock_clocksource_read()?
Because we needed a fix somewhat urgently, and I didn't want to risk more breakage by going with an aggressive change.
Quoted 1 line
But also, I would *love* to kill non-masterclock mode completely.
That's not feasible until KVM drops support for hardware that, per Linux standards, isn't _that_ old, is it?
Quoted 10 lines
In this nested case, the L1 TSCs *are* actually in sync; it just don't have the +invtsc CPUID to *promise* that they will be.
Could we find a way for L1 KVM to use masterclock mode for its nested L2 VMs even in this case? Could we make it use
PVCLOCK_TSC_STABLE_BITto enable masterclock mode, instead?At at the very least, could we find a way for L1 to trigger the
GLOBAL_CLOCK_UPDATEonly if an *actual* discrepancy in its TSCs is found? (Or, if vdso mode is notVDSO_CLOCKMODE_PVCLOCK?)
Or just advertise invtsc? I appreciate the desire to eliminate mostly-useless overhead like this, but at some point doesn't this boil down to "don't mess up your setup"?