KVM: x86: CPU isolation and direct interrupts delivery to guests
v2 cover KVM: x86: CPU isolation and direct interrupts delivery to guests
This RFC patch series provides facility to dedicate CPUs to KVM guests and enable the guests to handle interrupts from passed-through PCI devices directly (without VM exit and relay by the host).
With this feature, we can improve throughput and response time of the device and the host's CPU usage by reducing the overhead of interrupt handling. This is good for the application using very high throughput/frequent interrupt device (e.g. 10GbE NIC).
Real-time applicatoins also gets benefit from CPU isolation feature, which reduces interfare from host kernel tasks and scheduling delay.
The overview of this patch series is presented in CloudOpen 2012. The slides are available at:
http://events.linuxfoundation.org/images/stories/pdf/lcna_co2012_sekiyama.pdf
* Changes from v1 ( https://lkml.org/lkml/2012/6/28/30 )
- SMP guest is supported
- Direct EOI is added, that eliminate VM exit on EOI
- Direct local APIC timer access from guests is added, which pass-through the physical timer of a dedicated CPU to the guest.
- Rebased on v3.6-rc4
* How to test
- Create a guest VM with 1 CPU and some PCI passthrough devices (which supports MSI/MSI-X).
No VGA display will be better...
- Apply the patch at the end of this mail to qemu-kvm. (This patch is just for simple testing, and dedicated CPU ID for the guest is hard-coded.)
- Run the guest once to ensure the PCI passthrough works correctly.
- Make the specified CPU offline.
# echo 0 > /sys/devices/system/cpu/cpu3/online - Launch qemu-kvm with -no-kvm-pit option. The offlined CPU is booted as a slave CPU and guest is runs on that CPU.
* To-do
- Enable slave CPUs to handle access fault
- Support AMD SVM
- Support non-Linux guests
---
Tomoki Sekiyama (21):
x86: request TLB flush to slave CPU using NMI
KVM: Pass-through local APIC timer of on slave CPUs to guest VM
KVM: Enable direct EOI for directly routed interrupts to guests
KVM: route assigned devices' MSI/MSI-X directly to guests on slave CPUs
KVM: add kvm_arch_vcpu_prevent_run to prevent VM ENTER when NMI is received
KVM: vmx: Add definitions PIN_BASED_PREEMPTION_TIMER
KVM: add tracepoint on enabling/disabling direct interrupt delivery
KVM: Directly handle interrupts by guests without VM EXIT on slave CPUs x86/apic: IRQ vector remapping on slave for slave CPUs x86/apic: Enable external interrupt routing to slave CPUs
KVM: no exiting from guest when slave CPU halted
KVM: proxy slab operations for slave CPUs on online CPUs
KVM: Go back to online CPU on VM exit by external interrupt
KVM: Add KVM_GET_SLAVE_CPU and KVM_SET_SLAVE_CPU to vCPU ioctl
KVM: handle page faults of slave guests on online CPUs
KVM: Add facility to run guests on slave CPUs
KVM: Enable/Disable virtualization on slave CPUs are activated/dying
x86: Avoid RCU warnings on slave CPUs
x86: Support hrtimer on slave CPUs
x86: Add a facility to use offlined CPUs as slave CPUs
x86: Split memory hotplug function from cpu_up() as cpu_memory_up()
arch/x86/Kconfig | 10 + arch/x86/include/asm/apic.h | 10 + arch/x86/include/asm/irq.h | 15 + arch/x86/include/asm/kvm_host.h | 59 +++++ arch/x86/include/asm/tlbflush.h | 5 arch/x86/include/asm/vmx.h | 3 arch/x86/kernel/apic/apic.c | 11 + arch/x86/kernel/apic/io_apic.c | 111 ++++++++- arch/x86/kernel/apic/x2apic_cluster.c | 8 - arch/x86/kernel/cpu/common.c | 5 arch/x86/kernel/smp.c | 2 arch/x86/kernel/smpboot.c | 264 ++++++++++++++++++++++- arch/x86/kvm/irq.c | 136 ++++++++++++ arch/x86/kvm/lapic.c | 56 +++++ arch/x86/kvm/lapic.h | 2 arch/x86/kvm/mmu.c | 63 ++++- arch/x86/kvm/mmu.h | 4 arch/x86/kvm/trace.h | 19 ++ arch/x86/kvm/vmx.c | 180 +++++++++++++++ arch/x86/kvm/x86.c | 387 +++++++++++++++++++++++++++++++-- arch/x86/kvm/x86.h | 9 + arch/x86/mm/tlb.c | 94 ++++++++ drivers/iommu/intel_irq_remapping.c | 32 ++- include/linux/cpu.h | 36 +++ include/linux/cpumask.h | 26 ++ include/linux/kvm.h | 4 include/linux/kvm_host.h | 2 kernel/cpu.c | 83 +++++-- kernel/hrtimer.c | 14 + kernel/irq/manage.c | 4 kernel/irq/migration.c | 2 kernel/irq/proc.c | 2 kernel/rcutree.c | 14 + kernel/smp.c | 9 + virt/kvm/assigned-dev.c | 8 + virt/kvm/async_pf.c | 17 + virt/kvm/kvm_main.c | 32 +++ 37 files changed, 1629 insertions(+), 109 deletions(-)
* Patch for qemu-kvm-1.0
qemu-kvm-1.0/linux-headers/linux/kvm.h +5 -0
@@ -558,6 +558,7 @@ struct kvm_ppc_pvinfo {#define KVM_CAP_PPC_PAPR 68#define KVM_CAP_SW_TLB 69#define KVM_CAP_ONE_REG 70 +#define KVM_CAP_SLAVE_CPU 81#ifdef KVM_CAP_IRQ_ROUTING
@@ -811,6 +812,10 @@ struct kvm_one_reg {/* Available with KVM_CAP_ONE_REG */ #define KVM_GET_ONE_REG _IOWR(KVMIO, 0xab, struct kvm_one_reg)#define KVM_SET_ONE_REG _IOW(KVMIO, 0xac, struct kvm_one_reg) +/* Available with KVM_CAP_SLAVE_CPU */ +#define KVM_GET_SLAVE_CPU _IO(KVMIO, 0xae) +#define KVM_SET_SLAVE_CPU _IO(KVMIO, 0xaf) +#define KVM_DEV_ASSIGN_ENABLE_IOMMU (1 << 0)
qemu-kvm-1.0/qemu-kvm-x86.c +16 -0
@@ -139,12 +139,28 @@ static int kvm_enable_tpr_access_reportireturn kvm_vcpu_ioctl (env,KVM_TPR_ACCESS_REPORTING , &tac); } +static int kvm_set_slave_cpu (CPUState *env) +{ +int r, slave = env->cpu_index ==0 ?2 : env->cpu_index ==1 ?3 :-1 ; + + r =kvm_ioctl (env->kvm_state,KVM_CHECK_EXTENSION ,KVM_CAP_SLAVE_CPU ); +if (r <=0 ) { +return -ENOSYS ; + } + r =kvm_vcpu_ioctl (env,KVM_SET_SLAVE_CPU , slave); +if (r <0 ) +perror ("kvm_set_slave_cpu" ); +return r; +} +static int _kvm_arch_init_vcpu (CPUState *env) {kvm_arch_reset_vcpu (env);kvm_enable_tpr_access_reporting (env); +kvm_set_slave_cpu (env); +return kvm_update_ioport_access (env); }
RFC PATCH v2 1/21 x86: Split memory hotplug function from cpu_up() as cpu_memory_up()
Split memory hotplug function from cpu_up() as cpu_memory_up(), which will be used for assigning memory area to off-lined cpus at following patch in this series.
---
include/linux/cpu.h | 9 +++++++++ kernel/cpu.c | 46 +++++++++++++++++++++++++++------------------- 2 files changed, 36 insertions(+), 19 deletions(-)
include/linux/cpu.h +9 -0
@@ -146,6 +146,15 @@ void notify_cpu_starting(unsigned int cpu);extern void cpu_maps_update_begin (void );extern void cpu_maps_update_done (void ); +#ifdef CONFIG_MEMORY_HOTPLUG +extern int cpu_memory_up (unsigned int cpu); +#else +static inline int cpu_memory_up (unsigned int cpu) +{ +return 0 ; +} +#endif +#else /* CONFIG_SMP */ #define cpu_notifier (fn, pri) do { (void)(fn); } while (0)
kernel/cpu.c +27 -19
67 lines not on this page · raw message
Re: [RFC v2 PATCH 01/21] x86: Split memory hotplug function from cpu_up() as cpu_memory_up()
Quoted 4 lines · Tomoki Sekiyama
Split memory hotplug function from
cpu_up()ascpu_memory_up(), which will be used for assigning memory area to off-lined cpus at following patch in this series.
Can post a summary containing both the general outline for people reading this for the first time, or who have forgotten it, and the list of changes from v1?
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [RFC v2 PATCH 01/21] x86: Split memory hotplug function from cpu_up() as cpu_memory_up()
Quoted 10 lines · Avi Kivity
Quoted 4 lines · Tomoki Sekiyama
Split memory hotplug function from
cpu_up()ascpu_memory_up(), which will be used for assigning memory area to off-lined cpus at following patch in this series.Can post a summary containing both the general outline for people reading this for the first time, or who have forgotten it, and the list of changes from v1?
Never mind, I see it was posted, just that I wasn't copied on it.
Signature · 1 line
error compiling committee.c: too many arguments to function
RFC PATCH v2 2/21 x86: Add a facility to use offlined CPUs as slave CPUs
In reply to Tomoki SekiyamaAdd a facility of using offlined CPUs as slave CPUs. Slave CPUs are specialized to exclusively run functions specified by online CPUs, which do not run user processes.
To use this feature, build the kernel with CONFIG_SLAVE_CPU=y.
A slave CPU is launched by calling cpu_slave_up() when the CPU is offlined. Once launched, the slave CPU waits for IPI in the idle thread context. Users of the slave CPU can run specific kernel function by sending IPI using slave_cpu_call_function().
When cpu_slave_down() is called, the slave cpu will go offline state again.
Cpumask cpu_slave_mask' is provided to manage whether CPU is slave. In addition, cpu_online_or_slave_mask' is also provided for convenence of APIC handling, etc.
---
arch/x86/Kconfig | 10 ++ arch/x86/include/asm/cpu.h | 11 ++ arch/x86/kernel/cpu/common.c | 5 + arch/x86/kernel/smpboot.c | 190 ++++++++++++++++++++++++++++++++++++++++-- include/linux/cpumask.h | 26 ++++++ kernel/cpu.c | 37 ++++++++ kernel/smp.c | 8 +- 7 files changed, 275 insertions(+), 12 deletions(-)
arch/x86/Kconfig +10 -0
@@ -1678,6 +1678,16 @@ config HOTPLUG_CPU automatically on SMP systems. ) Say N if you want to disable CPU hotplug. +config SLAVE_CPU + bool "Support for slave CPUs (EXPERIMENTAL)" + depends on EXPERIMENTAL && HOTPLUG_CPU + ---help--- + Say Y here to allow use some of CPUs as slave processors. + Slave CPUs are controlled from another CPU and do some tasks + and cannot run user processes. Slave processors can be + specified through /sys/devices/system/cpu. + Say N if you want to disable slave CPU support. + config COMPAT_VDSO def_bool y prompt "Compat VDSO support"
arch/x86/include/asm/cpu.h +11 -0
@@ -30,6 +30,17 @@ extern int arch_register_cpu(int num);extern void arch_unregister_cpu (int );#endif +#ifdef CONFIG_SLAVE_CPU +#define CPU_SLAVE_UP_PREPARE 0xff00 +#define CPU_SLAVE_UP 0xff01 +#define CPU_SLAVE_DEAD 0xff02 + +extern int slave_cpu_up (unsigned int cpu); +extern int slave_cpu_down (unsigned int cpu); +extern void slave_cpu_call_function (unsigned int cpu, +void (*f)(void *),void *arg); +#endif +DECLARE_PER_CPU (int, cpu_state);int mwait_usable (const struct cpuinfo_x86 *);
arch/x86/kernel/cpu/common.c +4 -1
@@ -913,7 +913,10 @@ static void __cpuinit identify_cpu(struct cpuinfo_x86 *c) }/* Init Machine Check Exception if available. */ -mcheck_cpu_init (c); +#ifdef CONFIG_SLAVE_CPU + if (per_cpu(cpu_state, smp_processor_id()) != CPU_SLAVE_UP_PREPARE) +#endif +mcheck_cpu_init (c);select_idle_routine (c);
arch/x86/kernel/smpboot.c +183 -7
258 lines not on this page · raw message
include/linux/cpumask.h +26 -0
@@ -81,6 +81,19 @@ extern const struct cpumask *const cpu_online_mask;extern const struct cpumask *const cpu_present_mask;extern const struct cpumask *const cpu_active_mask; +/* +* cpu_online_or_slave_mask represents cpu_online_mask | cpu_slave_mask. +* This mask indicates the cpu can hanlde irq. +* if CONFIG_SLAVE_CPU is not defined, cpu_slave is defined as 0, +* and cpu_online_or_slave_mask is equals to cpu_online_mask. +*/ +#ifdef CONFIG_SLAVE_CPU +extern const struct cpumask *const cpu_slave_mask; +extern const struct cpumask *const cpu_online_or_slave_mask; +#else +#define cpu_online_or_slave_mask cpu_online_mask +#endif +#if NR_CPUS >1 #define num_online_cpus () cpumask_weight(cpu_online_mask)#define num_possible_cpus () cpumask_weight(cpu_possible_mask)
@@ -101,6 +114,12 @@ extern const struct cpumask *const cpu_active_mask;#define cpu_active (cpu) ((cpu) == 0)#endif +#if defined(CONFIG_SLAVE_CPU ) &&NR_CPUS >1 +#define cpu_slave (cpu) cpumask_test_cpu((cpu), cpu_slave_mask) +#else +#define cpu_slave (cpu) 0 +#endif +/* verify cpu argument to cpumask_* operators */ static inline unsigned int cpumask_check (unsigned int cpu) {
@@ -716,6 +735,13 @@ void init_cpu_present(const struct cpumask *src);void init_cpu_possible (const struct cpumask *src);void init_cpu_online (const struct cpumask *src); +#ifdef CONFIG_SLAVE_CPU +#define for_each_slave_cpu (cpu) for_each_cpu((cpu), cpu_slave_mask) +void set_cpu_slave (unsigned int cpu,bool slave); +#else +#define for_each_slave_cpu (cpu) while (0) +#endif +/** * to_cpumask - convert an NR_CPUS bitmap to a struct cpumask * * @bitmap: the bitmap
kernel/cpu.c +36 -1
@@ -343,7 +343,7 @@ static int __cpuinit _cpu_up(unsigned int cpu, int tasks_frozen)unsigned long mod = tasks_frozen ?CPU_TASKS_FROZEN :0 ;struct task_struct *idle; -if (cpu_online (cpu) || !cpu_present (cpu)) +if (cpu_online (cpu) ||cpu_slave (cpu) || !cpu_present (cpu))return -EINVAL ;cpu_hotplug_begin ();
@@ -685,6 +685,17 @@ static DECLARE_BITMAP(cpu_active_bits, CONFIG_NR_CPUS) __read_mostly;const struct cpumask *const cpu_active_mask =to_cpumask (cpu_active_bits);EXPORT_SYMBOL (cpu_active_mask); +#ifdef CONFIG_SLAVE_CPU +static DECLARE_BITMAP(cpu_slave_bits, CONFIG_NR_CPUS) __read_mostly; +const struct cpumask *const cpu_slave_mask =to_cpumask (cpu_slave_bits); +EXPORT_SYMBOL (cpu_slave_mask); + +static DECLARE_BITMAP(cpu_online_or_slave_bits, CONFIG_NR_CPUS) __read_mostly; +const struct cpumask *const cpu_online_or_slave_mask = +to_cpumask (cpu_online_or_slave_bits); +EXPORT_SYMBOL (cpu_online_or_slave_mask); +#endif +void set_cpu_possible (unsigned int cpu,bool possible) {if (possible)
@@ -707,6 +718,13 @@ void set_cpu_online(unsigned int cpu, bool online)cpumask_set_cpu (cpu,to_cpumask (cpu_online_bits)); elsecpumask_clear_cpu (cpu ,to_cpumask (cpu_online_bits )); + +#ifdef CONFIG_SLAVE_CPU +if (online) +cpumask_set_cpu (cpu,to_cpumask (cpu_online_or_slave_bits)); +else +cpumask_clear_cpu (cpu,to_cpumask (cpu_online_or_slave_bits)); +#endif }void set_cpu_active (unsigned int cpu,bool active)
@@ -717,6 +735,19 @@ void set_cpu_active(unsigned int cpu, bool active)cpumask_clear_cpu (cpu,to_cpumask (cpu_active_bits)); } +#ifdef CONFIG_SLAVE_CPU +void set_cpu_slave (unsigned int cpu,bool slave) +{ +if (slave) { +cpumask_set_cpu (cpu,to_cpumask (cpu_slave_bits)); +cpumask_set_cpu (cpu,to_cpumask (cpu_online_or_slave_bits)); + }else { +cpumask_clear_cpu (cpu,to_cpumask (cpu_slave_bits)); +cpumask_clear_cpu (cpu,to_cpumask (cpu_online_or_slave_bits)); + } +} +#endif +void init_cpu_present (const struct cpumask *src) {cpumask_copy (to_cpumask (cpu_present_bits), src);
@@ -730,4 +761,8 @@ void init_cpu_possible(const struct cpumask *src)void init_cpu_online (const struct cpumask *src) {cpumask_copy (to_cpumask (cpu_online_bits), src); + +#ifdef CONFIG_SLAVE_CPU +cpumask_copy (to_cpumask (cpu_online_or_slave_bits), src); +#endif }
kernel/smp.c +5 -3
@@ -177,7 +177,7 @@ void generic_smp_call_function_interrupt(void)/* * Shouldn't receive this interrupt on a cpu that is not yet online. */ -WARN_ON_ONCE (!cpu_online (cpu)); +WARN_ON_ONCE (!cpu_online (cpu) && !cpu_slave (cpu));/* * Ensure entry is visible on call_function_queue after we have
@@ -257,7 +257,8 @@ void generic_smp_call_function_single_interrupt(void)/* * Shouldn't receive this interrupt on a cpu that is not yet online. */ -WARN_ON_ONCE (!cpu_online (smp_processor_id ())); +WARN_ON_ONCE (!cpu_online (smp_processor_id ()) && + !cpu_slave (smp_processor_id ()));raw_spin_lock (&q->lock);list_replace_init (&q->list, &list);
@@ -326,7 +327,8 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,func (info);local_irq_restore (flags); }else { -if ((unsigned )cpu < nr_cpu_ids &&cpu_online (cpu)) { +if ((unsigned )cpu < nr_cpu_ids && + (cpu_online (cpu) ||cpu_slave (cpu))) {struct call_single_data *data = &d;if (!wait)
RFC PATCH v2 3/21 x86: Support hrtimer on slave CPUs
In reply to Tomoki SekiyamaAdds a facility to use hrtimer on slave CPUs.
To initialize hrtimer when slave CPUs are activated, and to shutdown hrtimer when slave CPUs are stopped, this patch adds the slave cpu notifier chain, which call registered callbacks when slave CPUs are up, dying, and died.
The registered callbacks are called with CPU_SLAVE_UP when a slave CPU becomes active. When the slave CPU is stopped, callbacks are called with CPU_SLAVE_DYING on slave CPUs, and with CPU_SLAVE_DEAD on online CPUs.
---
arch/x86/include/asm/cpu.h | 11 ----------- arch/x86/kernel/smpboot.c | 37 +++++++++++++++++++++++++++++++++++++ include/linux/cpu.h | 22 ++++++++++++++++++++++ kernel/hrtimer.c | 14 ++++++++++++++ 4 files changed, 73 insertions(+), 11 deletions(-)
arch/x86/include/asm/cpu.h +0 -11
@@ -30,17 +30,6 @@ extern int arch_register_cpu(int num);extern void arch_unregister_cpu (int );#endif -#ifdef CONFIG_SLAVE_CPU -#define CPU_SLAVE_UP_PREPARE 0xff00 -#define CPU_SLAVE_UP 0xff01 -#define CPU_SLAVE_DEAD 0xff02 - -extern int slave_cpu_up (unsigned int cpu); -extern int slave_cpu_down (unsigned int cpu); -extern void slave_cpu_call_function (unsigned int cpu, -void (*f)(void *),void *arg); -#endif -DECLARE_PER_CPU (int, cpu_state);int mwait_usable (const struct cpuinfo_x86 *);
arch/x86/kernel/smpboot.c +37 -0
@@ -127,6 +127,36 @@ EXPORT_PER_CPU_SYMBOL(cpu_info);atomic_t init_deasserted; +static void __ref remove_cpu_from_maps (int cpu); + + +#ifdef CONFIG_SLAVE_CPU +/* Notify slave cpu up and down */ +static RAW_NOTIFIER_HEAD (slave_cpu_chain); + +int register_slave_cpu_notifier (struct notifier_block *nb) +{ +return raw_notifier_chain_register (&slave_cpu_chain, nb); +} +EXPORT_SYMBOL (register_slave_cpu_notifier); + +void unregister_slave_cpu_notifier (struct notifier_block *nb) +{ +raw_notifier_chain_unregister (&slave_cpu_chain, nb); +} +EXPORT_SYMBOL (unregister_slave_cpu_notifier); + +static int slave_cpu_notify (unsigned long val,int cpu) +{ +int ret; + + ret =__raw_notifier_call_chain (&slave_cpu_chain, val, + (void *)(long )cpu,-1 ,NULL ); + +return notifier_to_errno (ret); +} +#endif +/* * Report back to the Boot Processor. * Running on AP.
@@ -307,6 +337,7 @@ notrace static void __cpuinit start_slave_cpu(void *unused)* most necessary things. */ cpu_init (); + x86_cpuinit.early_percpu_clock_init ();preempt_disable ();smp_callin (0 );
@@ -333,10 +364,14 @@ notrace static void __cpuinit start_slave_cpu(void *unused)/* to prevent fake stack check failure */ boot_init_stack_canary (); + x86_cpuinit.setup_percpu_clockev (); +tick_nohz_idle_enter (); +/* announce slave CPU started */ pr_info ("Slave CPU %d is up\n" , cpu);__this_cpu_write (cpu_state,CPU_SLAVE_UP );set_cpu_slave (cpu, true); +slave_cpu_notify (CPU_SLAVE_UP , cpu);wmb ();/* wait for slave_cpu_call_function or slave_cpu_down */
@@ -363,6 +398,7 @@ notrace static void __cpuinit start_slave_cpu(void *unused)local_irq_disable ();native_cpu_disable ();set_cpu_slave (cpu, false); +slave_cpu_notify (CPU_SLAVE_DYING , cpu); native_play_dead(); }#endif
@@ -995,6 +1031,7 @@ int slave_cpu_down(unsigned int cpu)return -EBUSY ; } +slave_cpu_notify (CPU_SLAVE_DEAD , cpu);return 0 ; }EXPORT_SYMBOL (slave_cpu_down);
include/linux/cpu.h +22 -0
@@ -221,4 +221,26 @@ static inline int disable_nonboot_cpus(void) { return 0; }static inline void enable_nonboot_cpus (void ) {}#endif /* !CONFIG_PM_SLEEP_SMP */ +#ifdef CONFIG_SLAVE_CPU +int register_slave_cpu_notifier (struct notifier_block *nb); +void unregister_slave_cpu_notifier (struct notifier_block *nb); + +/* CPU notifier constants for slave processors */ +#define CPU_SLAVE_UP_PREPARE 0xff00 +#define CPU_SLAVE_UP 0xff01 +#define CPU_SLAVE_DEAD 0xff02 +#define CPU_SLAVE_DYING 0xff03 + +extern int slave_cpu_up (unsigned int cpu); +extern int slave_cpu_down (unsigned int cpu); +extern void slave_cpu_call_function (unsigned int cpu, +void (*f)(void *),void *arg); +#else +static inline int register_slave_cpu_notifier (struct notifier_block *nb) +{ +return 0 ; +} +static inline void unregister_slave_cpu_notifier (struct notifier_block *nb) {} +#endif +#endif /* _LINUX_CPU_H_ */
kernel/hrtimer.c +14 -0
@@ -1727,16 +1727,25 @@ static int __cpuinit hrtimer_cpu_notify(struct notifier_block *self,case CPU_UP_PREPARE :case CPU_UP_PREPARE_FROZEN : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_UP : +#endif init_hrtimers_cpu (scpu);break ;#ifdef CONFIG_HOTPLUG_CPU case CPU_DYING :case CPU_DYING_FROZEN : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_DYING : +#endif clockevents_notify (CLOCK_EVT_NOTIFY_CPU_DYING , &scpu);break ;case CPU_DEAD :case CPU_DEAD_FROZEN : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_DEAD : +#endif {clockevents_notify (CLOCK_EVT_NOTIFY_CPU_DEAD , &scpu);migrate_hrtimers (scpu);
@@ -1755,11 +1764,16 @@ static struct notifier_block __cpuinitdata hrtimers_nb = { .notifier_call = hrtimer_cpu_notify, }; +static struct notifier_block __cpuinitdata hrtimers_slave_nb = { + .notifier_call = hrtimer_cpu_notify, +}; +void __init hrtimers_init (void ) {hrtimer_cpu_notify (&hrtimers_nb, (unsigned long )CPU_UP_PREPARE , (void *)(long )smp_processor_id ());register_cpu_notifier (&hrtimers_nb); +register_slave_cpu_notifier (&hrtimers_slave_nb);#ifdef CONFIG_HIGH_RES_TIMERS open_softirq (HRTIMER_SOFTIRQ , run_hrtimer_softirq);#endif
RFC PATCH v2 4/21 x86: Avoid RCU warnings on slave CPUs
In reply to Tomoki SekiyamaInitialize rcu related variables to avoid warnings about RCU usage while slave CPUs is running specified functions. Also notify RCU subsystem before the slave CPU is entered into idle state.
---
arch/x86/kernel/smpboot.c | 4 ++++ kernel/rcutree.c | 14 ++++++++++++++ 2 files changed, 18 insertions(+), 0 deletions(-)
arch/x86/kernel/smpboot.c +4 -0
@@ -382,6 +382,8 @@ notrace static void __cpuinit start_slave_cpu(void *unused) f =per_cpu (slave_cpu_func, cpu);per_cpu (slave_cpu_func, cpu).func =NULL ; +rcu_note_context_switch (cpu); +if (!f.func) {native_safe_halt ();continue ;
@@ -1005,6 +1007,8 @@ int __cpuinit slave_cpu_up(unsigned int cpu)if (IS_ERR (idle))return PTR_ERR (idle); +slave_cpu_notify (CPU_SLAVE_UP_PREPARE , cpu); + ret =__native_cpu_up (cpu, idle,1 );cpu_maps_update_done ();
kernel/rcutree.c +14 -0
@@ -2589,6 +2589,9 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,switch (action) {case CPU_UP_PREPARE :case CPU_UP_PREPARE_FROZEN : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_UP_PREPARE : +#endif rcu_prepare_cpu (cpu);rcu_prepare_kthreads (cpu);break ;
@@ -2603,6 +2606,9 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,break ;case CPU_DYING :case CPU_DYING_FROZEN : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_DYING : +#endif /* * The whole machine is "stopped" except this CPU, so we can * touch any data without introducing corruption. We send the
@@ -2616,6 +2622,9 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,case CPU_DEAD_FROZEN :case CPU_UP_CANCELED :case CPU_UP_CANCELED_FROZEN : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_DEAD : +#endif for_each_rcu_flavor (rsp)rcu_cleanup_dead_cpu (cpu, rsp);break ;
@@ -2797,6 +2806,10 @@ static void __init rcu_init_geometry(void) rcu_num_nodes -= n; } +static struct notifier_block __cpuinitdata rcu_slave_nb = { + .notifier_call = rcu_cpu_notify, +}; +void __init rcu_init (void ) {int cpu;
@@ -2814,6 +2827,7 @@ void __init rcu_init(void)* or the scheduler are operational. */ cpu_notifier (rcu_cpu_notify,0 ); +register_slave_cpu_notifier (&rcu_slave_nb);for_each_online_cpu (cpu)rcu_cpu_notify (NULL ,CPU_UP_PREPARE , (void *)(long )cpu);check_cpu_stall_init ();
Re: [RFC v2 PATCH 04/21] x86: Avoid RCU warnings on slave CPUs
Quoted 3 lines · Tomoki Sekiyama
Initialize rcu related variables to avoid warnings about RCU usage while slave CPUs is running specified functions. Also notify RCU subsystem before the slave CPU is entered into idle state.
Hello, Tomoki,
A few questions and comments interspersed below.
Thanx, Paul
Quoted 17 lines
Signed-off-by Tomoki Sekiyama <[email protected]>Cc Avi Kivity <[email protected]>Cc Marcelo Tosatti <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc "H. Peter Anvin" <[email protected]>---
arch/x86/kernel/smpboot.c | 4 ++++ kernel/rcutree.c | 14 ++++++++++++++ 2 files changed, 18 insertions(+), 0 deletions(-)arch/x86/kernel/smpboot.c +0
-0
arch/x86/kernel/smpboot.c notrace static void __cpuinit start_slave_cpu(void *unused) · line 386
Quoted 5 lines
arch/x86/kernel/smpboot.c +2
-0@@ -382,6 +382,8 @@ notrace static void __cpuinit start_slave_cpu(void *unused) f =per_cpu (slave_cpu_func, cpu);per_cpu (slave_cpu_func, cpu).func =NULL ; +rcu_note_context_switch (cpu); +
Why not use rcu_idle_enter() and rcu_idle_exit()? These would tell RCU to ignore the slave CPU for the duration of its idle period. The way you have it, if a slave CPU stayed idle for too long, you would get RCU CPU stall warnings, and possibly system hangs as well.
Or is this being called from some task that is not the idle task? If so, you instead want the new rcu_user_enter() and rcu_user_exit() that are hopefully on their way into 3.7. Or maybe better, use a real idle task, so that idle_task(smp_processor_id()) returns true and RCU stops complaining. ;-)
Note that CPUs that RCU believes to be idle are not permitted to contain RCU read-side critical sections, which in turn means no entering the scheduler, no sleeping, and so on. There is an RCU_NONIDLE() macro to tell RCU to pay attention to the CPU only for the duration of the statement passed to RCU_NONIDLE, and there are also an _rcuidle variant of the tracing statement to allow tracing from idle.
Quoted 17 lines
arch/x86/kernel/smpboot.c +2
-0if (!f.func) {native_safe_halt ();continue ;@@ -1005,6 +1007,8 @@ int __cpuinit slave_cpu_up(unsigned int cpu)if (IS_ERR (idle))return PTR_ERR (idle); +slave_cpu_notify (CPU_SLAVE_UP_PREPARE , cpu); + ret =__native_cpu_up (cpu, idle,1 );cpu_maps_update_done ();kernel/rcutree.c +0
-0
kernel/rcutree.c static int __cpuinit rcu_cpu_notify(struct notifier_block *self, · line 2594
Quoted 6 lines
kernel/rcutree.c +3
-0@@ -2589,6 +2589,9 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,switch (action) {case CPU_UP_PREPARE :case CPU_UP_PREPARE_FROZEN : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_UP_PREPARE : +#endif
Why do you need #ifdef here? Why not define CPU_SLAVE_UP_PREPARE unconditionally? Then if CONFIG_SLAVE_CPU=n, rcu_cpu_notify() would never be invoked with CPU_SLAVE_UP_PREPARE, so no problems.
kernel/rcutree.c static int __cpuinit rcu_cpu_notify(struct notifier_block *self, · line 2611
Quoted 10 lines
kernel/rcutree.c +3
-0rcu_prepare_cpu (cpu);rcu_prepare_kthreads (cpu);break ;@@ -2603,6 +2606,9 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,break ;case CPU_DYING :case CPU_DYING_FROZEN : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_DYING : +#endif
Same here.
kernel/rcutree.c static int __cpuinit rcu_cpu_notify(struct notifier_block *self, · line 2627
Quoted 10 lines
kernel/rcutree.c +3
-0/* * The whole machine is "stopped" except this CPU, so we can * touch any data without introducing corruption. We send the @@ -2616,6 +2622,9 @@ static int __cpuinit rcu_cpu_notify(struct notifier_block *self,case CPU_DEAD_FROZEN :case CPU_UP_CANCELED :case CPU_UP_CANCELED_FROZEN : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_DEAD : +#endif
And here.
Quoted 30 lines
kernel/rcutree.c +5
-1for_each_rcu_flavor (rsp)rcu_cleanup_dead_cpu (cpu, rsp);break ;@@ -2797,6 +2806,10 @@ static void __init rcu_init_geometry(void) rcu_num_nodes -= n; } +static struct notifier_block __cpuinitdata rcu_slave_nb = { + .notifier_call = rcu_cpu_notify, +}; +void __init rcu_init (void ) {int cpu;@@ -2814,6 +2827,7 @@ void __init rcu_init(void)* or the scheduler are operational. */ cpu_notifier (rcu_cpu_notify,0 ); +register_slave_cpu_notifier (&rcu_slave_nb);for_each_online_cpu (cpu)rcu_cpu_notify (NULL ,CPU_UP_PREPARE , (void *)(long )cpu);check_cpu_stall_init (); --To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [RFC v2 PATCH 04/21] x86: Avoid RCU warnings on slave CPUs
Hi Paul,
Thank you for your comments, and sorry for my late reply.
Quoted 17 lines · Paul E. McKenney
Quoted 3 lines · Tomoki Sekiyama
Initialize rcu related variables to avoid warnings about RCU usage while slave CPUs is running specified functions. Also notify RCU subsystem before the slave CPU is entered into idle state.
Hello, Tomoki,
A few questions and comments interspersed below.
Quoted 9 lines
<snip>
arch/x86/kernel/smpboot.c +0
-0@@ -382,6 +382,8 @@ notrace static void __cpuinit start_slave_cpu(void *unused) f =per_cpu (slave_cpu_func, cpu);per_cpu (slave_cpu_func, cpu).func =NULL ;
Quoted 7 lines
arch/x86/kernel/smpboot.c notrace static void __cpuinit start_slave_cpu(void *unused) · line 386
Quoted 2 lines
arch/x86/kernel/smpboot.c +2
-0+rcu_note_context_switch (cpu); +Why not use
rcu_idle_enter()andrcu_idle_exit()? These would tell RCU to ignore the slave CPU for the duration of its idle period. The way you have it, if a slave CPU stayed idle for too long, you would get RCU CPU stall warnings, and possibly system hangs as well.
That's true, rcu_idle_enter() and rcu_idle_exit() should be used when the slave cpu is idle. Thanks.
Quoted 6 lines
Or is this being called from some task that is not the idle task? If so, you instead want the new
rcu_user_enter()andrcu_user_exit()that are hopefully on their way into 3.7. Or maybe better, use a real idle task, so that idle_task(smp_processor_id()) returns true and RCU stops complaining. ;-)
Quoted 6 lines
Note that CPUs that RCU believes to be idle are not permitted to contain RCU read-side critical sections, which in turn means no entering the scheduler, no sleeping, and so on. There is an
RCU_NONIDLE()macro to tell RCU to pay attention to the CPU only for the duration of the statement passed toRCU_NONIDLE, and there are also an_rcuidlevariant of the tracing statement to allow tracing from idle.
This was for KVM is called as `func', which contains RCU read-side critical sections, and rcu_virt_note_context_switch() (that is rcu_note_context_switch(cpu)) before entering guest.
Maybe it should be replaced by rcu_user_enter() and rcu_user_exit() in the future.
Quoted 5 lines
Quoted 5 lines
kernel/rcutree.c +0
-0@@ -2589,6 +2589,9 @@ static int __cpuinit rcu_cpu_notify(struc tnotifier_block *self,switch (action) {case CPU_UP_PREPARE :
Quoted 8 lines
kernel/rcutree.c static int __cpuinit rcu_cpu_notify(struct notifier_block *self, · line 2594
Quoted 4 lines
kernel/rcutree.c +3
-0case CPU_UP_PREPARE_FROZEN : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_UP_PREPARE : +#endif Why do you need #ifdef here? Why not define
CPU_SLAVE_UP_PREPAREunconditionally? Then ifCONFIG_SLAVE_CPU=n,rcu_cpu_notify()would never be invoked withCPU_SLAVE_UP_PREPARE, so no problems.
Agreed. That will make the code simpler.
Thank you again,
Signature · 3 lines
Tomoki Sekiyama <[email protected]> Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory
RFC PATCH v2 5/21 KVM: Enable/Disable virtualization on slave CPUs are activated/dying
In reply to Tomoki SekiyamaEnable virtualization when slave CPUs are activated, and disable when the CPUs are dying using slave CPU notifier call chain.
In x86, TSC kHz must also be initialized by tsc_khz_changed when the new slave CPUs are activated.
---
arch/x86/kvm/x86.c | 16 ++++++++++++++++ virt/kvm/kvm_main.c | 30 ++++++++++++++++++++++++++++-- 2 files changed, 44 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +16 -0
@@ -61,6 +61,7 @@#include <asm/xcr.h> #include <asm/pvclock.h> #include <asm/div64.h> +#include <asm/cpu.h> #define MAX_IO_MSRS 256#define KVM_MAX_MCE_BANKS 32
@@ -4782,9 +4783,15 @@ static int kvmclock_cpu_notifier(struct notifier_block *nfb,switch (action) {case CPU_ONLINE :case CPU_DOWN_FAILED : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_UP : +#endif smp_call_function_single (cpu, tsc_khz_changed,NULL ,1 );break ;case CPU_DOWN_PREPARE : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_DYING : +#endif smp_call_function_single (cpu, tsc_bad,NULL ,1 );break ; }
@@ -4796,12 +4803,18 @@ static struct notifier_block kvmclock_cpu_notifier_block = { .priority = -INT_MAX }; +static struct notifier_block kvmclock_slave_cpu_notifier_block = { + .notifier_call = kvmclock_cpu_notifier, + .priority = -INT_MAX +}; +static void kvm_timer_init (void ) {int cpu; max_tsc_khz = tsc_khz;register_hotcpu_notifier (&kvmclock_cpu_notifier_block); +register_slave_cpu_notifier (&kvmclock_slave_cpu_notifier_block);if (!boot_cpu_has (X86_FEATURE_CONSTANT_TSC )) {#ifdef CONFIG_CPU_FREQ struct cpufreq_policy policy;
@@ -4818,6 +4831,8 @@ static void kvm_timer_init(void)pr_debug ("kvm: max_tsc_khz = %ld\n" , max_tsc_khz);for_each_online_cpu (cpu)smp_call_function_single (cpu, tsc_khz_changed,NULL ,1 ); +for_each_slave_cpu (cpu) +smp_call_function_single (cpu, tsc_khz_changed,NULL ,1 ); } static DEFINE_PER_CPU(struct kvm_vcpu *, current_vcpu);
@@ -4943,6 +4958,7 @@ void kvm_arch_exit(void)cpufreq_unregister_notifier (&kvmclock_cpufreq_notifier_block,CPUFREQ_TRANSITION_NOTIFIER );unregister_hotcpu_notifier (&kvmclock_cpu_notifier_block); +unregister_slave_cpu_notifier (&kvmclock_slave_cpu_notifier_block); kvm_x86_ops =NULL ;kvm_mmu_module_exit (); }
virt/kvm/kvm_main.c +28 -2
@@ -54,6 +54,9 @@#include <asm/io.h> #include <asm/uaccess.h> #include <asm/pgtable.h> +#ifdef CONFIG_X86 +#include <asm/cpu.h> +#endif #include "coalesced_mmio.h" #include "async_pf.h"
@@ -2336,11 +2339,17 @@ static void hardware_disable(void *junk)static void hardware_disable_all_nolock (void ) { +int cpu; +BUG_ON (!kvm_usage_count); kvm_usage_count--; -if (!kvm_usage_count) +if (!kvm_usage_count) {on_each_cpu (hardware_disable_nolock,NULL ,1 ); +for_each_slave_cpu (cpu) +smp_call_function_single (cpu, hardware_disable_nolock, +NULL ,1 ); + } }static void hardware_disable_all (void )
@@ -2353,6 +2362,7 @@ static void hardware_disable_all(void)static int hardware_enable_all (void ) {int r =0 ; +int cpu;raw_spin_lock (&kvm_lock);
@@ -2360,6 +2370,9 @@ static int hardware_enable_all(void)if (kvm_usage_count ==1 ) {atomic_set (&hardware_enable_failed,0 );on_each_cpu (hardware_enable_nolock,NULL ,1 ); +for_each_slave_cpu (cpu) +smp_call_function_single (cpu, hardware_enable_nolock, +NULL ,1 );if (atomic_read (&hardware_enable_failed)) {hardware_disable_all_nolock ();
@@ -2383,11 +2396,17 @@ static int kvm_cpu_hotplug(struct notifier_block *notifier, unsigned long val, val &= ~CPU_TASKS_FROZEN ;switch (val) {case CPU_DYING : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_DYING : +#endif printk (KERN_INFO "kvm: disabling virtualization on CPU%d\n" , cpu);hardware_disable (NULL );break ;case CPU_STARTING : +#ifdef CONFIG_SLAVE_CPU +case CPU_SLAVE_UP : +#endif printk (KERN_INFO "kvm: enabling virtualization on CPU%d\n" , cpu);hardware_enable (NULL );
@@ -2605,6 +2624,10 @@ static struct notifier_block kvm_cpu_notifier = { .notifier_call = kvm_cpu_hotplug, }; +static struct notifier_block kvm_slave_cpu_notifier = { + .notifier_call = kvm_cpu_hotplug, +}; +static int vm_stat_get (void *_offset,u64 *val) {unsigned offset = (long )_offset;
@@ -2768,7 +2791,7 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align,if (r <0 ) goto out_free_0a; -for_each_online_cpu (cpu) { +for_each_cpu (cpu, cpu_online_or_slave_mask) {smp_call_function_single (cpu, kvm_arch_check_processor_compat, &r,1 );
@@ -2779,6 +2802,7 @@ int kvm_init(void *opaque, unsigned vcpu_size, unsigned vcpu_align, r =register_cpu_notifier (&kvm_cpu_notifier);if (r) goto out_free_2; +register_slave_cpu_notifier (&kvm_slave_cpu_notifier);register_reboot_notifier (&kvm_reboot_notifier);/* A kmem cache lets us meet the alignment requirements of fx_save. */
@@ -2826,6 +2850,7 @@ out_free:kmem_cache_destroy (kvm_vcpu_cache); out_free_3:unregister_reboot_notifier (&kvm_reboot_notifier); +unregister_slave_cpu_notifier (&kvm_slave_cpu_notifier);unregister_cpu_notifier (&kvm_cpu_notifier); out_free_2: out_free_1:
@@ -2853,6 +2878,7 @@ void kvm_exit(void)kvm_async_pf_deinit ();unregister_syscore_ops (&kvm_syscore_ops);unregister_reboot_notifier (&kvm_reboot_notifier); +unregister_slave_cpu_notifier (&kvm_slave_cpu_notifier);unregister_cpu_notifier (&kvm_cpu_notifier);on_each_cpu (hardware_disable_nolock,NULL ,1 );kvm_arch_hardware_unsetup ();
RFC PATCH v2 6/21 KVM: Add facility to run guests on slave CPUs
In reply to Tomoki SekiyamaAdd path to migrate execution of vcpu_enter_guest to a slave CPU when vcpu->arch.slave_cpu is set.
After moving to the slave CPU, it goes back to the online CPU when the guest is exited by reasons that cannot be handled by the slave CPU only (e.g. handling async page faults).
On migration, kvm_arch_vcpu_put_migrate is used to avoid using IPI to clear loaded vmcs from the old CPU. Instead, this immediately clears vmcs.
---
arch/x86/include/asm/kvm_host.h | 9 ++ arch/x86/kernel/smp.c | 2 arch/x86/kvm/vmx.c | 10 ++ arch/x86/kvm/x86.c | 189 ++++++++++++++++++++++++++++++++++----- arch/x86/kvm/x86.h | 9 ++ include/linux/kvm_host.h | 1 kernel/smp.c | 1 virt/kvm/async_pf.c | 9 +- virt/kvm/kvm_main.c | 3 - 9 files changed, 203 insertions(+), 30 deletions(-)
arch/x86/include/asm/kvm_host.h +9 -0
@@ -354,6 +354,14 @@ struct kvm_vcpu_arch {u64 ia32_misc_enable_msr;bool tpr_access_reporting; +#ifdef CONFIG_SLAVE_CPU +/* slave cpu dedicated to this vcpu */ +int slave_cpu; +#endif + +/* user process tied to each vcpu */ +struct task_struct *task; +/* * Paging state of the vcpu *
@@ -617,6 +625,7 @@ struct kvm_x86_ops {void (*prepare_guest_switch)(struct kvm_vcpu *vcpu);void (*vcpu_load)(struct kvm_vcpu *vcpu,int cpu);void (*vcpu_put)(struct kvm_vcpu *vcpu); +void (*vcpu_put_migrate)(struct kvm_vcpu *vcpu);void (*set_guest_debug)(struct kvm_vcpu *vcpu,struct kvm_guest_debug *dbg);
arch/x86/kernel/smp.c +1 -1
@@ -119,7 +119,7 @@ static bool smp_no_nmi_ipi = false;*/ static void native_smp_send_reschedule (int cpu) { -if (unlikely (cpu_is_offline (cpu))) { +if (unlikely (cpu_is_offline (cpu) && !cpu_slave (cpu))) {WARN_ON (1 );return ; }
arch/x86/kvm/vmx.c +9 -1
@@ -1557,6 +1557,13 @@ static void vmx_vcpu_put(struct kvm_vcpu *vcpu) } } +static void vmx_vcpu_put_migrate (struct kvm_vcpu *vcpu) +{ +vmx_vcpu_put (vcpu); +__loaded_vmcs_clear (to_vmx (vcpu)->loaded_vmcs); + vcpu->cpu =-1 ; +} +static void vmx_fpu_activate (struct kvm_vcpu *vcpu) {ulong cr0;
@@ -5017,7 +5024,7 @@ static int handle_invalid_guest_state(struct kvm_vcpu *vcpu)return 0 ; } -if (signal_pending (current)) +if (signal_pending (vcpu->arch.task)) goto out;if (need_resched ())schedule ();
@@ -7263,6 +7270,7 @@ static struct kvm_x86_ops vmx_x86_ops = { .prepare_guest_switch = vmx_save_host_state, .vcpu_load = vmx_vcpu_load, .vcpu_put = vmx_vcpu_put, + .vcpu_put_migrate = vmx_vcpu_put_migrate, .set_guest_debug = set_guest_debug, .get_msr = vmx_get_msr,
arch/x86/kvm/x86.c +164 -25
270 lines not on this page · raw message
arch/x86/kvm/x86.h +9 -0
@@ -108,6 +108,15 @@ static inline bool vcpu_match_mmio_gpa(struct kvm_vcpu *vcpu, gpa_t gpa)return false; } +static inline bool vcpu_has_slave_cpu (struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_SLAVE_CPU +return vcpu->arch.slave_cpu >=0 ; +#else +return 0 ; +#endif +} +void kvm_before_handle_nmi (struct kvm_vcpu *vcpu);void kvm_after_handle_nmi (struct kvm_vcpu *vcpu);int kvm_inject_realmode_interrupt (struct kvm_vcpu *vcpu,int irq,int inc_eip);
include/linux/kvm_host.h +1 -0
@@ -119,6 +119,7 @@ struct kvm_async_pf { };void kvm_clear_async_pf_completion_queue (struct kvm_vcpu *vcpu); +int kvm_can_complete_async_pf (struct kvm_vcpu *vcpu);void kvm_check_async_pf_completion (struct kvm_vcpu *vcpu);int kvm_setup_async_pf (struct kvm_vcpu *vcpu,gva_t gva,gfn_t gfn,struct kvm_arch_async_pf *arch);
kernel/smp.c +1 -0
@@ -431,6 +431,7 @@ void __smp_call_function_single(int cpu, struct call_single_data *data, }put_cpu (); } +EXPORT_SYMBOL (__smp_call_function_single);/** * smp_call_function_many(): Run a function on a set of other CPUs.
virt/kvm/async_pf.c +7 -2
@@ -120,12 +120,17 @@ void kvm_clear_async_pf_completion_queue(struct kvm_vcpu *vcpu) vcpu->async_pf.queued =0 ; } +int kvm_can_complete_async_pf (struct kvm_vcpu *vcpu) +{ +return !list_empty_careful (&vcpu->async_pf.done) && +kvm_arch_can_inject_async_page_present (vcpu); +} +void kvm_check_async_pf_completion (struct kvm_vcpu *vcpu) {struct kvm_async_pf *work; -while (!list_empty_careful (&vcpu->async_pf.done) && -kvm_arch_can_inject_async_page_present (vcpu)) { +while (kvm_can_complete_async_pf (vcpu)) {spin_lock (&vcpu->async_pf.lock); work =list_first_entry (&vcpu->async_pf.done,typeof (*work), link);
virt/kvm/kvm_main.c +2 -1
@@ -1543,7 +1543,8 @@ void kvm_vcpu_kick(struct kvm_vcpu *vcpu) } me =get_cpu (); -if (cpu != me && (unsigned )cpu < nr_cpu_ids &&cpu_online (cpu)) +if (cpu != me && (unsigned )cpu < nr_cpu_ids && + (cpu_online (cpu) ||cpu_slave (cpu)))if (kvm_arch_vcpu_should_kick (vcpu))smp_send_reschedule (cpu);put_cpu ();
RFC PATCH v2 7/21 KVM: handle page faults of slave guests on online CPUs
In reply to Tomoki SekiyamaPage faults which occured by the guest running on slave CPUs cannot be handled on slave CPUs because it is running on idle process context.
With this patch, the page fault happened in a slave CPU is notified to online CPU using struct kvm_access_fault, and is handled after the user-process for the guest is resumed on an online CPU.
---
arch/x86/include/asm/kvm_host.h | 15 +++++++++++++++ arch/x86/kvm/mmu.c | 13 +++++++++++++ arch/x86/kvm/x86.c | 10 ++++++++++ 3 files changed, 38 insertions(+), 0 deletions(-)
arch/x86/include/asm/kvm_host.h +15 -0
@@ -67,6 +67,11 @@#define UNMAPPED_GVA (~(gpa_t)0) +#ifdef CONFIG_SLAVE_CPU +/* Requests to handle VM exit on online cpu */ +#define KVM_REQ_HANDLE_PF 32 +#endif +/* KVM Hugepage definitions for x86 */ #define KVM_NR_PAGE_SIZES 3#define KVM_HPAGE_GFN_SHIFT (x) (((x) - 1) * 9)
@@ -413,6 +418,16 @@ struct kvm_vcpu_arch {u8 nr; } interrupt; +#ifdef CONFIG_SLAVE_CPU +/* used for recording page fault on offline CPU */ +struct kvm_access_fault { +gva_t cr2; +u32 error_code; +void *insn; +int insn_len; + } page_fault; +#endif +int halt_request;/* real mode on Intel only */ int cpuid_nent;
arch/x86/kvm/mmu.c +13 -0
@@ -3946,6 +3946,19 @@ int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gva_t cr2, u32 error_code,int r, emulation_type =EMULTYPE_RETRY ;enum emulation_result er; +#ifdef CONFIG_SLAVE_CPU +if (cpu_slave (smp_processor_id ())) { +/* Page fault must be handled on user-process context. */ + r = -EFAULT ; + vcpu->arch.page_fault.cr2 = cr2; + vcpu->arch.page_fault.error_code = error_code; + vcpu->arch.page_fault.insn = insn; + vcpu->arch.page_fault.insn_len = insn_len; +kvm_make_request (KVM_REQ_HANDLE_PF , vcpu); + goto out; + } +#endif + r = vcpu->arch.mmu.page_fault (vcpu, cr2, error_code, false);if (r <0 ) goto out;
arch/x86/kvm/x86.c +10 -0
@@ -5561,6 +5561,16 @@ static int vcpu_enter_guest_slave(struct kvm_vcpu *vcpu, r = arg.ret; *apf_pending = arg.apf_pending; +if (r == -EFAULT &&kvm_check_request (KVM_REQ_HANDLE_PF , vcpu)) { +pr_debug ("handling page fault request @%p\n" , + (void *)vcpu->arch.page_fault.cr2); + r =kvm_mmu_page_fault (vcpu, + vcpu->arch.page_fault.cr2, + vcpu->arch.page_fault.error_code, + vcpu->arch.page_fault.insn, + vcpu->arch.page_fault.insn_len); + } +return r; }
RFC PATCH v2 8/21 KVM: Add KVM_GET_SLAVE_CPU and KVM_SET_SLAVE_CPU to vCPU ioctl
In reply to Tomoki SekiyamaAdd an interface to set/get slave CPU dedicated to the vCPUs.
By calling ioctl with KVM_GET_SLAVE_CPU, users can get the slave CPU id for the vCPU. -1 is returned if a slave CPU is not set.
By calling ioctl with KVM_SET_SLAVE_CPU, users can dedicate the specified slave CPU to the vCPU. The CPU must be offlined before calling ioctl. The CPU is activated as slave CPU for the vCPU when the correct id is set. The slave CPU is freed and offlined by setting -1 as slave CPU id.
Whether getting/setting slave CPUs are supported by KVM or not can be known by checking KVM_CAP_SLAVE_CPU.
---
arch/x86/include/asm/kvm_host.h | 2 + arch/x86/kvm/vmx.c | 7 +++++ arch/x86/kvm/x86.c | 58 +++++++++++++++++++++++++++++++++++++++ include/linux/kvm.h | 4 +++ 4 files changed, 71 insertions(+), 0 deletions(-)
arch/x86/include/asm/kvm_host.h +2 -0
@@ -718,6 +718,8 @@ struct kvm_x86_ops {int (*check_intercept)(struct kvm_vcpu *vcpu,struct x86_instruction_info *info,enum x86_intercept_stage stage); + +void (*set_slave_mode)(struct kvm_vcpu *vcpu,bool slave); };struct kvm_arch_async_pf {
arch/x86/kvm/vmx.c +7 -0
@@ -1698,6 +1698,11 @@ static void skip_emulated_instruction(struct kvm_vcpu *vcpu)vmx_set_interrupt_shadow (vcpu,0 ); } +static void vmx_set_slave_mode (struct kvm_vcpu *vcpu,bool slave) +{ +/* Nothing */ +} +/* * KVM wants to inject page-faults which it got to the guest. This function * checks whether in a nested guest, we need to inject them to L1 or L2.
@@ -7344,6 +7349,8 @@ static struct kvm_x86_ops vmx_x86_ops = { .set_tdp_cr3 = vmx_set_cr3, .check_intercept = vmx_check_intercept, + + .set_slave_mode = vmx_set_slave_mode, };static int __init vmx_init (void )
arch/x86/kvm/x86.c +58 -0
82 lines not on this page · raw message
include/linux/kvm.h +4 -0
@@ -618,6 +618,7 @@ struct kvm_ppc_smmu_info {#define KVM_CAP_PPC_GET_SMMU_INFO 78#define KVM_CAP_S390_COW 79#define KVM_CAP_PPC_ALLOC_HTAB 80 +#define KVM_CAP_SLAVE_CPU 81#ifdef KVM_CAP_IRQ_ROUTING
@@ -904,6 +905,9 @@ struct kvm_s390_ucas_mapping {#define KVM_SET_ONE_REG _IOW(KVMIO, 0xac, struct kvm_one_reg)/* VM is being stopped by host */ #define KVM_KVMCLOCK_CTRL _IO(KVMIO, 0xad) +/* Available with KVM_CAP_SLAVE_CPU */ +#define KVM_GET_SLAVE_CPU _IO(KVMIO, 0xae) +#define KVM_SET_SLAVE_CPU _IO(KVMIO, 0xaf)#define KVM_DEV_ASSIGN_ENABLE_IOMMU (1 << 0)#define KVM_DEV_ASSIGN_PCI_2_3 (1 << 1)
RFC PATCH v2 9/21 KVM: Go back to online CPU on VM exit by external interrupt
In reply to Tomoki SekiyamaIf the slave CPU receives an interrupt in running a guest, current implementation must once go back to onilne CPUs to handle the interupt.
This behavior will be replaced by later patch, which introduces direct interrupt handling mechanism by the guest.
---
arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/vmx.c | 1 + arch/x86/kvm/x86.c | 6 ++++++ 3 files changed, 8 insertions(+), 0 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -358,6 +358,7 @@ struct kvm_vcpu_arch {int sipi_vector;u64 ia32_misc_enable_msr;bool tpr_access_reporting; +bool interrupted;#ifdef CONFIG_SLAVE_CPU /* slave cpu dedicated to this vcpu */
arch/x86/kvm/vmx.c +1 -0
@@ -4408,6 +4408,7 @@ static int handle_exception(struct kvm_vcpu *vcpu)static int handle_external_interrupt (struct kvm_vcpu *vcpu) { + vcpu->arch.interrupted = true; ++vcpu->stat.irq_exits;return 1 ; }
arch/x86/kvm/x86.c +6 -0
@@ -5566,6 +5566,12 @@ static void __vcpu_enter_guest_slave(void *_arg)break ;/* determine if slave cpu can handle the exit alone */ +if (vcpu->arch.interrupted) { + vcpu->arch.interrupted = false; + arg->ret =LOOP_ONLINE ; +break ; + } + r =vcpu_post_run (vcpu, arg->task, &arg->apf_pending); if (r == LOOP_SLAVE &&
RFC PATCH v2 10/21 KVM: proxy slab operations for slave CPUs on online CPUs
In reply to Tomoki SekiyamaAdd some fix-ups that proxy slab operations on online CPUs for the guest, in order to avoid touching slab on slave CPUs where some slab functions are not activated.
Currently, slab may be touched on slave CPUs in following 3 cases. For each cases, the fix-ups below are introduced:
* kvm_mmu_commit_zap_page
With this patch, Instead of commiting zap pages, the pages are added into invalid_mmu_pages list and make KVM_REQ_COMMIT_ZAP_PAGE request. Then, the pages are freed on online CPUs after the execution of vCPU thread is resumed.
* mmu_topup_memory_caches
Preallocate caches for mmu operations in vcpu_enter_guest_slave, which is done by online CPUs before entering guests.
* kvm_async_pf_wakeup_all
If this function is called on slave CPUs, it makes KVM_REQ_WAKEUP_APF. The request is handled by calling kvm_async_pf_wakeup_all on online CPUs.
---
arch/x86/include/asm/kvm_host.h | 5 ++++ arch/x86/kvm/mmu.c | 52 ++++++++++++++++++++++++++++----------- arch/x86/kvm/mmu.h | 4 +++ arch/x86/kvm/x86.c | 15 +++++++++++ virt/kvm/async_pf.c | 8 ++++++ 5 files changed, 69 insertions(+), 15 deletions(-)
arch/x86/include/asm/kvm_host.h +5 -0
@@ -70,6 +70,8 @@#ifdef CONFIG_SLAVE_CPU /* Requests to handle VM exit on online cpu */ #define KVM_REQ_HANDLE_PF 32 +#define KVM_REQ_COMMIT_ZAP_PAGE 33 +#define KVM_REQ_WAKEUP_APF 34#endif /* KVM Hugepage definitions for x86 */
@@ -542,6 +544,9 @@ struct kvm_arch {* Hash table of struct kvm_mmu_page. */ struct list_head active_mmu_pages; +#ifdef CONFIG_SLAVE_CPU +struct list_head invalid_mmu_pages; +#endif struct list_head assigned_dev_head;struct iommu_domain *iommu_domain;int iommu_flags;
arch/x86/kvm/mmu.c +37 -15
160 lines not on this page · raw message
arch/x86/kvm/mmu.h +4 -0
@@ -52,6 +52,10 @@ int kvm_mmu_get_spte_hierarchy(struct kvm_vcpu *vcpu, u64 addr, u64 sptes[4]);void kvm_mmu_set_mmio_spte_mask (u64 mmio_mask);int handle_mmio_page_fault_common (struct kvm_vcpu *vcpu,u64 addr,bool direct);int kvm_init_shadow_mmu (struct kvm_vcpu *vcpu,struct kvm_mmu *context); +int mmu_topup_memory_caches (struct kvm_vcpu *vcpu); +#ifdef CONFIG_SLAVE_CPU +void kvm_mmu_commit_zap_page_late (struct kvm_vcpu *vcpu); +#endif static inline unsigned int kvm_mmu_available_pages (struct kvm *kvm) {
arch/x86/kvm/x86.c +15 -0
@@ -5571,6 +5571,11 @@ static void __vcpu_enter_guest_slave(void *_arg) arg->ret =LOOP_ONLINE ;break ; } +if (test_bit (KVM_REQ_COMMIT_ZAP_PAGE , &vcpu->requests) || +test_bit (KVM_REQ_WAKEUP_APF , &vcpu->requests)) { + r =LOOP_ONLINE ; +break ; + } r =vcpu_post_run (vcpu, arg->task, &arg->apf_pending);
@@ -5598,6 +5603,9 @@ static int vcpu_enter_guest_slave(struct kvm_vcpu *vcpu,BUG_ON ((unsigned )slave >= nr_cpu_ids || !cpu_slave (slave)); +/* Refill memory caches here to avoid calling slab on slave cpu */ +mmu_topup_memory_caches (vcpu); +preempt_disable ();preempt_notifier_unregister (&vcpu->preempt_notifier);kvm_arch_vcpu_put_migrate (vcpu);
@@ -5631,6 +5639,10 @@ static int vcpu_enter_guest_slave(struct kvm_vcpu *vcpu, vcpu->arch.page_fault.insn, vcpu->arch.page_fault.insn_len); } +if (kvm_check_request (KVM_REQ_WAKEUP_APF , vcpu)) +kvm_async_pf_wakeup_all (vcpu); +if (kvm_check_request (KVM_REQ_COMMIT_ZAP_PAGE , vcpu)) +kvm_mmu_commit_zap_page_late (vcpu);return r; }
@@ -6486,6 +6498,9 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)return -EINVAL ;INIT_LIST_HEAD (&kvm->arch.active_mmu_pages); +#ifdef CONFIG_SLAVE_CPU +INIT_LIST_HEAD (&kvm->arch.invalid_mmu_pages); +#endif INIT_LIST_HEAD (&kvm->arch.assigned_dev_head);/* Reserve bit 0 of irq_sources_bitmap for userspace irq source */
virt/kvm/async_pf.c +8 -0
@@ -204,6 +204,14 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu)if (!list_empty_careful (&vcpu->async_pf.done))return 0 ; +#ifdef CONFIG_SLAVE_CPU +if (cpu_slave (raw_smp_processor_id ())) { +/* Redo on online cpu to avoid kmem_cache_alloc on slave cpu */ +kvm_make_request (KVM_REQ_WAKEUP_APF , vcpu); +return -ENOMEM ; + } +#endif + work =kmem_cache_zalloc (async_pf_cache,GFP_ATOMIC );if (!work)return -ENOMEM ;
RFC PATCH v2 11/21 KVM: no exiting from guest when slave CPU halted
In reply to Tomoki SekiyamaAvoid exiting from a guest on slave CPU even if HLT instruction is executed. Since the slave CPU is dedicated to a vCPU, exit on HLT is not required, and avoiding VM exit will improve the guest's performance.
This is a partial revert of
10166744b80a ("KVM: VMX: remove yield_on_hlt")---
arch/x86/kvm/vmx.c | 25 ++++++++++++++++++++++++- 1 files changed, 24 insertions(+), 1 deletions(-)
arch/x86/kvm/vmx.c +24 -1
@@ -1698,9 +1698,29 @@ static void skip_emulated_instruction(struct kvm_vcpu *vcpu)vmx_set_interrupt_shadow (vcpu,0 ); } +static inline void vmx_clear_hlt (struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_SLAVE_CPU +/* Ensure that we clear the HLT state in the VMCS. We don't need to +* explicitly skip the instruction because if the HLT state is set, +* then the instruction is already executing and RIP has already been +* advanced. */ +if (vcpu->arch.slave_cpu >=0 && +vmcs_read32 (GUEST_ACTIVITY_STATE ) ==GUEST_ACTIVITY_HLT ) +vmcs_write32 (GUEST_ACTIVITY_STATE ,GUEST_ACTIVITY_ACTIVE ); +#endif +} +static void vmx_set_slave_mode (struct kvm_vcpu *vcpu,bool slave) { -/* Nothing */ +/* Don't intercept the guest's halt on slave CPU */ +if (slave) { +vmcs_clear_bits (CPU_BASED_VM_EXEC_CONTROL , +CPU_BASED_HLT_EXITING ); + }else { +vmcs_set_bits (CPU_BASED_VM_EXEC_CONTROL , +CPU_BASED_HLT_EXITING ); + } }/*
@@ -1755,6 +1775,7 @@ static void vmx_queue_exception(struct kvm_vcpu *vcpu, unsigned nr, intr_info |=INTR_TYPE_HARD_EXCEPTION ;vmcs_write32 (VM_ENTRY_INTR_INFO_FIELD , intr_info); +vmx_clear_hlt (vcpu); }static bool vmx_rdtscp_supported (void )
@@ -4125,6 +4146,7 @@ static void vmx_inject_irq(struct kvm_vcpu *vcpu) }else intr |=INTR_TYPE_EXT_INTR ;vmcs_write32 (VM_ENTRY_INTR_INFO_FIELD , intr); +vmx_clear_hlt (vcpu); }static void vmx_inject_nmi (struct kvm_vcpu *vcpu)
@@ -4156,6 +4178,7 @@ static void vmx_inject_nmi(struct kvm_vcpu *vcpu) }vmcs_write32 (VM_ENTRY_INTR_INFO_FIELD ,INTR_TYPE_NMI_INTR |INTR_INFO_VALID_MASK |NMI_VECTOR ); +vmx_clear_hlt (vcpu); }static int vmx_nmi_allowed (struct kvm_vcpu *vcpu)
RFC PATCH v2 12/21 x86/apic: Enable external interrupt routing to slave CPUs
In reply to Tomoki SekiyamaEnable APIC to handle interrupts on slave CPUs, and enables interrupt routing to slave CPUs by setting IRQ affinity.
As slave CPUs which run a KVM guest handle external interrupts directly in the vCPUs, the guest's vector/IRQ mapping is different from the host's. That requires interrupts to be routed either online CPUs or slave CPUs.
In this patch, if online CPUs are contained in specified affinity settings, the affinity settings will be only applied to online CPUs. If every specified CPU is slave, IRQ will be routed to slave CPUs.
---
arch/x86/include/asm/apic.h | 6 ++--- arch/x86/kernel/apic/io_apic.c | 43 ++++++++++++++++++++++++--------- arch/x86/kernel/apic/x2apic_cluster.c | 8 +++--- drivers/iommu/intel_irq_remapping.c | 30 +++++++++++++++++++---- kernel/irq/manage.c | 4 ++- kernel/irq/migration.c | 2 +- kernel/irq/proc.c | 2 +- 7 files changed, 67 insertions(+), 28 deletions(-)
arch/x86/include/asm/apic.h +3 -3
@@ -535,7 +535,7 @@ extern void generic_bigsmp_probe(void);static inline const struct cpumask *default_target_cpus (void ) {#ifdef CONFIG_SMP -return cpu_online_mask; +return cpu_online_or_slave_mask;#else return cpumask_of (0 );#endif
@@ -543,7 +543,7 @@ static inline const struct cpumask *default_target_cpus(void)static inline const struct cpumask *online_target_cpus (void ) { -return cpu_online_mask; +return cpu_online_or_slave_mask; }DECLARE_EARLY_PER_CPU_READ_MOSTLY (u16, x86_bios_cpu_apicid);
@@ -602,7 +602,7 @@ flat_cpu_mask_to_apicid_and(const struct cpumask *cpumask, {unsigned long cpu_mask =cpumask_bits (cpumask)[0 ] &cpumask_bits (andmask)[0 ] & -cpumask_bits (cpu_online_mask)[0 ] & +cpumask_bits (cpu_online_or_slave_mask)[0 ] &APIC_ALL_CPUS ;if (likely (cpu_mask)) {
arch/x86/kernel/apic/io_apic.c +31 -12
101 lines not on this page · raw message
arch/x86/kernel/apic/x2apic_cluster.c +4 -4
@@ -106,7 +106,7 @@ x2apic_cpu_mask_to_apicid_and(const struct cpumask *cpumask,int i;for_each_cpu_and (i, cpumask, andmask) { -if (!cpumask_test_cpu (i, cpu_online_mask)) +if (!cpumask_test_cpu (i, cpu_online_or_slave_mask))continue ; dest =per_cpu (x86_cpu_to_logical_apicid, i); cluster =x2apic_cluster (i);
@@ -117,7 +117,7 @@ x2apic_cpu_mask_to_apicid_and(const struct cpumask *cpumask,return -EINVAL ;for_each_cpu_and (i, cpumask, andmask) { -if (!cpumask_test_cpu (i, cpu_online_mask)) +if (!cpumask_test_cpu (i, cpu_online_or_slave_mask))continue ;if (cluster !=x2apic_cluster (i))continue ;
@@ -137,7 +137,7 @@ static void init_x2apic_ldr(void)per_cpu (x86_cpu_to_logical_apicid, this_cpu) =apic_read (APIC_LDR );__cpu_set (this_cpu,per_cpu (cpus_in_cluster, this_cpu)); -for_each_online_cpu (cpu) { +for_each_cpu (cpu, cpu_online_or_slave_mask) {if (x2apic_cluster (this_cpu) !=x2apic_cluster (cpu))continue ;__cpu_set (this_cpu,per_cpu (cpus_in_cluster, cpu));
@@ -169,7 +169,7 @@ update_clusterinfo(struct notifier_block *nfb, unsigned long action, void *hcpu)case CPU_UP_CANCELED :case CPU_UP_CANCELED_FROZEN :case CPU_DEAD : -for_each_online_cpu (cpu) { +for_each_cpu (cpu, cpu_online_or_slave_mask) {if (x2apic_cluster (this_cpu) !=x2apic_cluster (cpu))continue ;__cpu_clear (this_cpu,per_cpu (cpus_in_cluster, cpu));
drivers/iommu/intel_irq_remapping.c +25 -5
@@ -931,26 +931,43 @@ intel_ioapic_set_affinity(struct irq_data *data, const struct cpumask *mask,struct irq_cfg *cfg = data->chip_data;unsigned int dest, irq = data->irq;struct irte irte; +cpumask_var_t new_mask; +bool mask_ok = false;int err;if (!config_enabled (CONFIG_SMP ))return -EINVAL ; -if (!cpumask_intersects (mask, cpu_online_mask)) +if (!cpumask_intersects (mask, cpu_online_or_slave_mask))return -EINVAL ;if (get_irte (irq, &irte))return -EBUSY ; +#ifdef CONFIG_SLAVE_CPU +/* Set affinity to either online cpus only or slave cpus only */ + mask_ok =alloc_cpumask_var (&new_mask,GFP_ATOMIC ); +if (mask_ok) { +cpumask_and (new_mask, mask, cpu_online_mask); +if (cpumask_empty (new_mask)) +cpumask_copy (new_mask, mask); + err =assign_irq_vector (irq, cfg, new_mask); + }else if (!cpumask_intersects (mask, cpu_online_mask) || + !cpumask_intersects (mask, cpu_slave_mask)) + err =assign_irq_vector (irq, cfg, mask); +else +return -ENOMEM ; +#else err =assign_irq_vector (irq, cfg, mask); +#endif if (err) -return err; + goto error; err = apic->cpu_mask_to_apicid_and (cfg->domain, mask, &dest);if (err) {if (assign_irq_vector (irq, cfg, data->affinity))pr_err ("Failed to recover vector for irq %d\n" , irq); -return err; + goto error; } irte.vector = cfg->vector;
@@ -970,8 +987,11 @@ intel_ioapic_set_affinity(struct irq_data *data, const struct cpumask *mask,if (cfg->move_in_progress)send_cleanup_vector (cfg); -cpumask_copy (data->affinity, mask); -return 0 ; +cpumask_copy (data->affinity, mask_ok ? new_mask : mask); +error: +if (mask_ok) +free_cpumask_var (new_mask); +return err; } static void intel_compose_msi_msg(struct pci_dev *pdev,
kernel/irq/manage.c +2 -2
@@ -308,13 +308,13 @@ setup_affinity(unsigned int irq, struct irq_desc *desc, struct cpumask *mask)*/ if (irqd_has_set (&desc->irq_data,IRQD_AFFINITY_SET )) {if (cpumask_intersects (desc->irq_data.affinity, - cpu_online_mask)) + cpu_online_or_slave_mask)) set = desc->irq_data.affinity;else irqd_clear (&desc->irq_data,IRQD_AFFINITY_SET ); } -cpumask_and (mask, cpu_online_mask, set); +cpumask_and (mask, cpu_online_or_slave_mask, set);if (node !=NUMA_NO_NODE ) {const struct cpumask *nodemask =cpumask_of_node (node);
kernel/irq/migration.c +1 -1
@@ -42,7 +42,7 @@ void irq_move_masked_irq(struct irq_data *idata)* For correct operation this depends on the caller * masking the irqs. */ -if (cpumask_any_and (desc->pending_mask, cpu_online_mask) < nr_cpu_ids) +if (cpumask_any_and (desc->pending_mask, cpu_online_or_slave_mask) < nr_cpu_ids)irq_do_set_affinity (&desc->irq_data, desc->pending_mask, false);cpumask_clear (desc->pending_mask);
kernel/irq/proc.c +1 -1
@@ -103,7 +103,7 @@ static ssize_t write_irq_affinity(int type, struct file *file,* way to make the system unusable accidentally :-) At least * one online CPU still has to be targeted. */ -if (!cpumask_intersects (new_value, cpu_online_mask)) { +if (!cpumask_intersects (new_value, cpu_online_or_slave_mask)) {/* Special case for empty set - allow the architecture code to set default SMP affinity. */ err =irq_select_affinity_usr (irq, new_value) ? -EINVAL : count;
RFC PATCH v2 13/21 x86/apic: IRQ vector remapping on slave for slave CPUs
In reply to Tomoki SekiyamaAdd a facility to use IRQ vector different from online CPUs on slave CPUs.
When alternative vector for IRQ is registered by remap_slave_vector_irq() and the IRQ affinity is set only to slave CPUs, the device is configured to use the alternative vector.
Current patch only supports MSI and Intel CPU with IRQ remapper of IOMMU.
This is intended to be used to routing interrupts directly to KVM guest which is running on slave CPUs which do not cause VM EXIT by external interrupts.
---
arch/x86/include/asm/irq.h | 15 ++++++++ arch/x86/kernel/apic/io_apic.c | 68 ++++++++++++++++++++++++++++++++++- drivers/iommu/intel_irq_remapping.c | 2 + 3 files changed, 83 insertions(+), 2 deletions(-)
arch/x86/include/asm/irq.h +15 -0
@@ -41,4 +41,19 @@ extern int vector_used_by_percpu_irq(unsigned int vector);extern void init_ISA_irqs (void ); +#ifdef CONFIG_SLAVE_CPU +extern void remap_slave_vector_irq (int irq,int vector, +const struct cpumask *mask); +extern void revert_slave_vector_irq (int irq,const struct cpumask *mask); +extern u8 get_remapped_slave_vector (u8 vector,unsigned int irq, +const struct cpumask *mask); +#else +static inline u8 get_remapped_slave_vector (u8 vector,unsigned int irq, +const struct cpumask *mask) +{ +return vector; +} +#endif + +#endif /* _ASM_X86_IRQ_H */
arch/x86/kernel/apic/io_apic.c +67 -1
87 lines not on this page · raw message
drivers/iommu/intel_irq_remapping.c +1 -1
@@ -970,7 +970,7 @@ intel_ioapic_set_affinity(struct irq_data *data, const struct cpumask *mask, goto error; } - irte.vector = cfg->vector; + irte.vector =get_remapped_slave_vector (cfg->vector, irq, mask); irte.dest_id =IRTE_DEST (dest);/*
RFC PATCH v2 14/21 KVM: Directly handle interrupts by guests without VM EXIT on slave CPUs
In reply to Tomoki SekiyamaMake interrupts on slave CPUs handled by guests without VM EXIT. This reduces CPU usage by the host to transfer interrupts of assigned PCI devices from the host to guests. It also reduces cost of VM EXIT and quickens response of guests to the interrupts.
When a slave CPU is dedicated to a vCPU, exit on external interrupts is disabled. Unfortunately, we can only enable/disable exits for whole external interrupts except NMIs and cannot switch exits based on IRQ# or vectors. Thus, to avoid IPIs from online CPUs transferred to guests, this patch modify kvm_vcpu_kick() to use NMI for guests on slave CPUs.
---
arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kvm/lapic.c | 5 +++++ arch/x86/kvm/vmx.c | 19 ++++++++++++++++++ arch/x86/kvm/x86.c | 41 +++++++++++++++++++++++++++++++++++++++ include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 5 +++-- 6 files changed, 70 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
@@ -725,6 +725,7 @@ struct kvm_x86_ops {struct x86_instruction_info *info, enum x86_intercept_stage stage); +void (*set_direct_interrupt)(struct kvm_vcpu *vcpu,bool enabled);void (*set_slave_mode)(struct kvm_vcpu *vcpu,bool slave); };
arch/x86/kvm/lapic.c +5 -0
@@ -601,6 +601,9 @@ static int apic_set_eoi(struct kvm_lapic *apic)kvm_ioapic_update_eoi (apic->vcpu->kvm, vector, trigger_mode); }kvm_make_request (KVM_REQ_EVENT , apic->vcpu); +if (vcpu_has_slave_cpu (apic->vcpu) && + kvm_x86_ops->set_direct_interrupt) + kvm_x86_ops->set_direct_interrupt (apic->vcpu,1 );return vector; }
@@ -1569,6 +1572,8 @@ int kvm_lapic_enable_pv_eoi(struct kvm_vcpu *vcpu, u64 data)u64 addr = data & ~KVM_MSR_ENABLED ;if (!IS_ALIGNED (addr,4 ))return 1 ; +if (vcpu_has_slave_cpu (vcpu)) +return 1 ; vcpu->arch.pv_eoi.msr_val = data;if (!pv_eoi_enabled (vcpu))
arch/x86/kvm/vmx.c +19 -0
@@ -1711,6 +1711,16 @@ static inline void vmx_clear_hlt(struct kvm_vcpu *vcpu)#endif } +static void vmx_set_direct_interrupt (struct kvm_vcpu *vcpu,bool enabled) +{ +if (enabled) +vmcs_clear_bits (PIN_BASED_VM_EXEC_CONTROL , +PIN_BASED_EXT_INTR_MASK ); +else +vmcs_set_bits (PIN_BASED_VM_EXEC_CONTROL , +PIN_BASED_EXT_INTR_MASK ); +} +static void vmx_set_slave_mode (struct kvm_vcpu *vcpu,bool slave) {/* Don't intercept the guest's halt on slave CPU */
@@ -1721,6 +1731,8 @@ static void vmx_set_slave_mode(struct kvm_vcpu *vcpu, bool slave)vmcs_set_bits (CPU_BASED_VM_EXEC_CONTROL ,CPU_BASED_HLT_EXITING ); } + +vmx_set_direct_interrupt (vcpu, slave); }/*
@@ -1776,6 +1788,8 @@ static void vmx_queue_exception(struct kvm_vcpu *vcpu, unsigned nr,vmcs_write32 (VM_ENTRY_INTR_INFO_FIELD , intr_info);vmx_clear_hlt (vcpu); +if (vcpu_has_slave_cpu (vcpu)) +vmx_set_direct_interrupt (vcpu,0 ); }static bool vmx_rdtscp_supported (void )
@@ -4147,6 +4161,8 @@ static void vmx_inject_irq(struct kvm_vcpu *vcpu) intr |=INTR_TYPE_EXT_INTR ;vmcs_write32 (VM_ENTRY_INTR_INFO_FIELD , intr);vmx_clear_hlt (vcpu); +if (vcpu_has_slave_cpu (vcpu)) +vmx_set_direct_interrupt (vcpu,0 ); }static void vmx_inject_nmi (struct kvm_vcpu *vcpu)
@@ -4179,6 +4195,8 @@ static void vmx_inject_nmi(struct kvm_vcpu *vcpu)vmcs_write32 (VM_ENTRY_INTR_INFO_FIELD ,INTR_TYPE_NMI_INTR |INTR_INFO_VALID_MASK |NMI_VECTOR );vmx_clear_hlt (vcpu); +if (vcpu_has_slave_cpu (vcpu)) +vmx_set_direct_interrupt (vcpu,0 ); }static int vmx_nmi_allowed (struct kvm_vcpu *vcpu)
@@ -7374,6 +7392,7 @@ static struct kvm_x86_ops vmx_x86_ops = { .check_intercept = vmx_check_intercept, + .set_direct_interrupt = vmx_set_direct_interrupt, .set_slave_mode = vmx_set_slave_mode, };
arch/x86/kvm/x86.c +41 -0
77 lines not on this page · raw message
include/linux/kvm_host.h +1 -0
@@ -539,6 +539,7 @@ void kvm_arch_hardware_unsetup(void);void kvm_arch_check_processor_compat (void *rtn);int kvm_arch_vcpu_runnable (struct kvm_vcpu *vcpu);int kvm_arch_vcpu_should_kick (struct kvm_vcpu *vcpu); +int kvm_arch_vcpu_kick_slave (struct kvm_vcpu *vcpu);void kvm_free_physmem (struct kvm *kvm);
virt/kvm/kvm_main.c +3 -2
@@ -1543,10 +1543,11 @@ void kvm_vcpu_kick(struct kvm_vcpu *vcpu) } me =get_cpu (); -if (cpu != me && (unsigned )cpu < nr_cpu_ids && - (cpu_online (cpu) ||cpu_slave (cpu))) +if (cpu != me && (unsigned )cpu < nr_cpu_ids &&cpu_online (cpu))if (kvm_arch_vcpu_should_kick (vcpu))smp_send_reschedule (cpu); +if (cpu != me && (unsigned )cpu < nr_cpu_ids &&cpu_slave (cpu)) +kvm_arch_vcpu_kick_slave (vcpu);put_cpu (); }#endif /* !CONFIG_S390 */
RFC PATCH v2 15/21 KVM: add tracepoint on enabling/disabling direct interrupt delivery
In reply to Tomoki SekiyamaAdd trace event "kvm_set_direct_interrupt" to trace enabling/disabling direct interrupt delivery on slave CPUs. At the event, the guest rip and whether the feature is enabled or not is logged.
---
arch/x86/kvm/trace.h | 18 ++++++++++++++++++ arch/x86/kvm/vmx.c | 2 ++ arch/x86/kvm/x86.c | 1 + 3 files changed, 21 insertions(+), 0 deletions(-)
arch/x86/kvm/trace.h +18 -0
@@ -551,6 +551,24 @@ TRACE_EVENT(kvm_pv_eoi,TP_printk ("apicid %x vector %d" , __entry->apicid, __entry->vector) ); +TRACE_EVENT (kvm_set_direct_interrupt, + TP_PROTO(struct kvm_vcpu *vcpu, bool enabled), +TP_ARGS (vcpu, enabled), + +TP_STRUCT__entry ( + __field( unsigned long, guest_rip ) +__field ( bool, enabled ) + ), + +TP_fast_assign ( + __entry->guest_rip = kvm_rip_read(vcpu); + __entry->enabled = enabled; + ), + +TP_printk ("rip 0x%lx enabled %d" , + __entry->guest_rip, __entry->enabled) +); +/* * Tracepoint for nested VMRUN */
arch/x86/kvm/vmx.c +2 -0
@@ -1719,6 +1719,8 @@ static void vmx_set_direct_interrupt(struct kvm_vcpu *vcpu, bool enabled)else vmcs_set_bits (PIN_BASED_VM_EXEC_CONTROL ,PIN_BASED_EXT_INTR_MASK ); + +trace_kvm_set_direct_interrupt (vcpu, enabled); }static void vmx_set_slave_mode (struct kvm_vcpu *vcpu,bool slave)
arch/x86/kvm/x86.c +1 -0
@@ -6936,3 +6936,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_nested_intr_vmexit);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_invlpga);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_skinit);EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_nested_intercepts); +EXPORT_TRACEPOINT_SYMBOL_GPL (kvm_set_direct_interrupt);
RFC PATCH v2 16/21 KVM: vmx: Add definitions PIN_BASED_PREEMPTION_TIMER
In reply to Tomoki SekiyamaAdd some definitions to use PIN_BASED_PREEMPTION_TIMER.
When PIN_BASED_PREEMPTION_TIMER is enabled, the guest will exit with reason=EXIT_REASON_PREEMPTION_TIMER when the counter specified in VMX_PREEMPTION_TIMER_VALUE becomes 0.
This patch also adds a dummy handler for EXIT_REASON_PREEMPTION_TIMER, which just goes back to VM execution soon.
These are currently intended only to be used with avoid entering the guest on a slave CPU when vmx_prevent_run(vcpu, 1) is called.
---
arch/x86/include/asm/vmx.h | 3 +++ arch/x86/kvm/trace.h | 1 + arch/x86/kvm/vmx.c | 7 +++++++ 3 files changed, 11 insertions(+), 0 deletions(-)
arch/x86/include/asm/vmx.h +3 -0
@@ -66,6 +66,7 @@#define PIN_BASED_EXT_INTR_MASK 0x00000001#define PIN_BASED_NMI_EXITING 0x00000008#define PIN_BASED_VIRTUAL_NMIS 0x00000020 +#define PIN_BASED_PREEMPTION_TIMER 0x00000040#define VM_EXIT_SAVE_DEBUG_CONTROLS 0x00000002#define VM_EXIT_HOST_ADDR_SPACE_SIZE 0x00000200
@@ -196,6 +197,7 @@ enum vmcs_field {GUEST_INTERRUPTIBILITY_INFO =0x00004824 ,GUEST_ACTIVITY_STATE =0X00004826 ,GUEST_SYSENTER_CS =0x0000482A , +VMX_PREEMPTION_TIMER_VALUE =0x0000482E ,HOST_IA32_SYSENTER_CS =0x00004c00 ,CR0_GUEST_HOST_MASK =0x00006000 , CR4_GUEST_HOST_MASK = 0x00006002,
@@ -280,6 +282,7 @@ enum vmcs_field {#define EXIT_REASON_APIC_ACCESS 44#define EXIT_REASON_EPT_VIOLATION 48#define EXIT_REASON_EPT_MISCONFIG 49 +#define EXIT_REASON_PREEMPTION_TIMER 52#define EXIT_REASON_WBINVD 54#define EXIT_REASON_XSETBV 55#define EXIT_REASON_INVPCID 58
arch/x86/kvm/trace.h +1 -0
@@ -218,6 +218,7 @@ TRACE_EVENT(kvm_apic, { EXIT_REASON_APIC_ACCESS, "APIC_ACCESS" }, \ { EXIT_REASON_EPT_VIOLATION, "EPT_VIOLATION" }, \ { EXIT_REASON_EPT_MISCONFIG, "EPT_MISCONFIG" }, \ + { EXIT_REASON_PREEMPTION_TIMER, "PREEMPTION_TIMER" }, \ { EXIT_REASON_WBINVD, "WBINVD" } #define SVM_EXIT_REASONS \
arch/x86/kvm/vmx.c +7 -0
@@ -4456,6 +4456,12 @@ static int handle_external_interrupt(struct kvm_vcpu *vcpu)return 1 ; } +static int handle_preemption_timer (struct kvm_vcpu *vcpu) +{ +/* Nothing */ +return 1 ; +} +static int handle_triple_fault (struct kvm_vcpu *vcpu) { vcpu->run->exit_reason =KVM_EXIT_SHUTDOWN ;
@@ -5768,6 +5774,7 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = { [EXIT_REASON_VMON] = handle_vmon, [EXIT_REASON_TPR_BELOW_THRESHOLD] = handle_tpr_below_threshold, [EXIT_REASON_APIC_ACCESS] = handle_apic_access, + [EXIT_REASON_PREEMPTION_TIMER] = handle_preemption_timer, [EXIT_REASON_WBINVD] = handle_wbinvd, [EXIT_REASON_XSETBV] = handle_xsetbv, [EXIT_REASON_TASK_SWITCH] = handle_task_switch,
RFC PATCH v2 17/21 KVM: add kvm_arch_vcpu_prevent_run to prevent VM ENTER when NMI is received
In reply to Tomoki SekiyamaSince NMI can not be disabled around VM enter, there is a race between receiving NMI to kick a guest and entering the guests on slave CPUs.If the NMI is received just before entering VM, after the NMI handler is invoked, it continues entering the guest and the effect of the NMI will be lost.
This patch adds kvm_arch_vcpu_prevent_run(), which causes VM exit right after VM enter. The NMI handler uses this to ensure the execution of the guest is cancelled after NMI.
---
arch/x86/include/asm/kvm_host.h | 6 ++++++ arch/x86/kvm/vmx.c | 42 ++++++++++++++++++++++++++++++++++++++- arch/x86/kvm/x86.c | 31 +++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+), 1 deletions(-)
arch/x86/include/asm/kvm_host.h +6 -0
@@ -429,6 +429,9 @@ struct kvm_vcpu_arch {void *insn; int insn_len; } page_fault; + +bool prevent_run; +bool prevent_needed;#endif int halt_request;/* real mode on Intel only */
@@ -681,6 +684,7 @@ struct kvm_x86_ops {void (*run)(struct kvm_vcpu *vcpu);int (*handle_exit)(struct kvm_vcpu *vcpu); +void (*prevent_run)(struct kvm_vcpu *vcpu,int prevent);void (*skip_emulated_instruction)(struct kvm_vcpu *vcpu);void (*set_interrupt_shadow)(struct kvm_vcpu *vcpu,int mask);u32 (*get_interrupt_shadow)(struct kvm_vcpu *vcpu,int mask);
@@ -1027,4 +1031,6 @@ int kvm_pmu_read_pmc(struct kvm_vcpu *vcpu, unsigned pmc, u64 *data);void kvm_handle_pmu_event (struct kvm_vcpu *vcpu);void kvm_deliver_pmi (struct kvm_vcpu *vcpu); +int kvm_arch_vcpu_run_prevented (struct kvm_vcpu *vcpu); +#endif /* _ASM_X86_KVM_HOST_H */
arch/x86/kvm/vmx.c +41 -1
96 lines not on this page · raw message
arch/x86/kvm/x86.c +31 -0
@@ -2667,6 +2667,19 @@ static DEFINE_PER_CPU(struct kvm_vcpu *, slave_vcpu);static int kvm_arch_kicked_by_nmi (unsigned int cmd,struct pt_regs *regs); +struct kvm_vcpu *get_slave_vcpu (int cpu) +{ +return per_cpu (slave_vcpu, cpu); +} + +static int kvm_arch_vcpu_prevent_run (struct kvm_vcpu *vcpu,int prevent) +{ + vcpu->arch.prevent_run = prevent; +if (!prevent || vcpu->arch.prevent_needed) + kvm_x86_ops->prevent_run (vcpu, prevent); +return 1 ; +} +static int kvm_arch_vcpu_ioctl_set_slave_cpu (struct kvm_vcpu *vcpu,int slave,int set_slave_mode) {
@@ -4873,10 +4886,12 @@ static struct notifier_block kvmclock_cpu_notifier_block = { .priority = -INT_MAX }; +#ifdef CONFIG_SLAVE_CPU static struct notifier_block kvmclock_slave_cpu_notifier_block = { .notifier_call = kvmclock_cpu_notifier, .priority = -INT_MAX }; +#endif static void kvm_timer_init (void ) {
@@ -5350,6 +5365,11 @@ static int kvm_arch_kicked_by_nmi(unsigned int cmd, struct pt_regs *regs)if (vcpu->mode ==OUTSIDE_GUEST_MODE ||kvm_is_in_guest ())return NMI_HANDLED ; +/* +* We may be about to entering VM. To prevent entering, +* mark to exit as soon as possible. +*/ +kvm_arch_vcpu_prevent_run (vcpu,1 ); return NMI_HANDLED; }#endif
@@ -5592,6 +5612,8 @@ static void __vcpu_enter_guest_slave(void *_arg)kvm_arch_vcpu_load (vcpu, cpu);while (r ==LOOP_SLAVE ) { + vcpu->arch.prevent_run =0 ; + r =vcpu_enter_guest (vcpu, arg->task);if (r <=0 )
@@ -5618,6 +5640,7 @@ static void __vcpu_enter_guest_slave(void *_arg) } } +kvm_arch_vcpu_prevent_run (vcpu,0 );kvm_arch_vcpu_put_migrate (vcpu);unuse_mm (arg->task->mm);srcu_read_unlock (&vcpu->kvm->srcu, vcpu->srcu_idx);
@@ -6733,6 +6756,14 @@ int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu) kvm_cpu_has_interrupt(vcpu)); } +int kvm_arch_vcpu_run_prevented (struct kvm_vcpu *vcpu) +{ + kvm_x86_ops->prevent_run (vcpu,0 ); + vcpu->arch.interrupted = true; +return 1 ; +} +EXPORT_SYMBOL_GPL (kvm_arch_vcpu_run_prevented); +int kvm_arch_vcpu_should_kick (struct kvm_vcpu *vcpu) {return kvm_vcpu_exiting_guest_mode (vcpu) ==IN_GUEST_MODE ;
RFC PATCH v2 18/21 KVM: route assigned devices' MSI/MSI-X directly to guests on slave CPUs
In reply to Tomoki SekiyamaWhen a PCI device is assigned to a guest running on slave CPUs, this routes the device's MSI/MSI-X interrupts directly to the guest.
Because the guest uses a different interrupt vector from the host, vector remapping is required. This is safe because slave CPUs only handles interrupts for the assigned guest.
The slave CPU may receive interrupts for the guest while the guest is not running. In that case, the host IRQ handler is invoked and the interrupt is transfered as vIRQ.
If the guest receive the direct interrupt from the devices, EOI to physical APIC is required. To handle this, if the guest issues EOI when there are no in-service interrupts in the virtual APIC, physical EOI is issued.
---
arch/x86/include/asm/kvm_host.h | 19 +++++ arch/x86/kvm/irq.c | 136 +++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/lapic.c | 6 +- arch/x86/kvm/x86.c | 12 +++ virt/kvm/assigned-dev.c | 8 ++ 5 files changed, 179 insertions(+), 2 deletions(-)
arch/x86/include/asm/kvm_host.h +19 -0
@@ -1033,4 +1033,23 @@ void kvm_deliver_pmi(struct kvm_vcpu *vcpu);int kvm_arch_vcpu_run_prevented (struct kvm_vcpu *vcpu); +#ifdef CONFIG_SLAVE_CPU +void kvm_get_slave_cpu_mask (struct kvm *kvm,struct cpumask *mask); + +struct kvm_assigned_dev_kernel ; +extern void assign_slave_msi (struct kvm *kvm, +struct kvm_assigned_dev_kernel *assigned_dev); +extern void deassign_slave_msi (struct kvm *kvm, +struct kvm_assigned_dev_kernel *assigned_dev); +extern void assign_slave_msix (struct kvm *kvm, +struct kvm_assigned_dev_kernel *assigned_dev); +extern void deassign_slave_msix (struct kvm *kvm, +struct kvm_assigned_dev_kernel *assigned_dev); +#else +#define assign_slave_msi (kvm, assigned_dev) +#define deassign_slave_msi (kvm, assigned_dev) +#define assign_slave_msix (kvm, assigned_dev) +#define deassign_slave_msix (kvm, assigned_dev) +#endif +#endif /* _ASM_X86_KVM_HOST_H */
arch/x86/kvm/irq.c +136 -0
145 lines not on this page · raw message
arch/x86/kvm/lapic.c +5 -1
@@ -585,8 +585,12 @@ static int apic_set_eoi(struct kvm_lapic *apic)* Not every write EOI will has corresponding ISR, * one example is when Kernel check timer on setup_IO_APIC */ -if (vector ==-1 ) +if (vector ==-1 ) { +/* On slave cpu, it can be EOI for a direct interrupt */ +if (cpu_slave (smp_processor_id ())) +ack_APIC_irq ();return vector; + }apic_clear_isr (vector, apic);apic_update_ppr (apic);
arch/x86/kvm/x86.c +12 -0
@@ -5349,6 +5349,17 @@ static void process_nmi(struct kvm_vcpu *vcpu) }#ifdef CONFIG_SLAVE_CPU + +void kvm_get_slave_cpu_mask (struct kvm *kvm,struct cpumask *mask) +{ +int i; +struct kvm_vcpu *vcpu; + +kvm_for_each_vcpu (i, vcpu, kvm) +if (vcpu_has_slave_cpu (vcpu)) +cpumask_set_cpu (vcpu->arch.slave_cpu, mask); +} +static int kvm_arch_kicked_by_nmi (unsigned int cmd,struct pt_regs *regs) {struct kvm_vcpu *vcpu;
@@ -5372,6 +5383,7 @@ static int kvm_arch_kicked_by_nmi(unsigned int cmd, struct pt_regs *regs)kvm_arch_vcpu_prevent_run (vcpu,1 );return NMI_HANDLED ; } +#endif enum vcpu_enter_guest_slave_retval {
virt/kvm/assigned-dev.c +7 -1
@@ -225,8 +225,12 @@ static void deassign_host_irq(struct kvm *kvm,free_irq (assigned_dev->host_irq, assigned_dev); -if (assigned_dev->irq_requested_type &KVM_DEV_IRQ_HOST_MSI ) +if (assigned_dev->irq_requested_type &KVM_DEV_IRQ_HOST_MSI ) {pci_disable_msi (assigned_dev->dev); +deassign_slave_msi (kvm, assigned_dev); + } +if (assigned_dev->irq_requested_type &KVM_DEV_IRQ_HOST_MSIX ) +deassign_slave_msix (kvm, assigned_dev); } assigned_dev->irq_requested_type &= ~(KVM_DEV_IRQ_HOST_MASK );
@@ -417,6 +421,7 @@ static int assigned_device_enable_guest_msi(struct kvm *kvm, { dev->guest_irq = irq->guest_irq; dev->ack_notifier.gsi =-1 ; +assign_slave_msi (kvm, dev);return 0 ; }#endif
@@ -428,6 +433,7 @@ static int assigned_device_enable_guest_msix(struct kvm *kvm, { dev->guest_irq = irq->guest_irq; dev->ack_notifier.gsi =-1 ; +assign_slave_msix (kvm, dev);return 0 ; }#endif
RFC PATCH v2 19/21 KVM: Enable direct EOI for directly routed interrupts to guests
In reply to Tomoki SekiyamaEnable direct access to EOI MSR of x2apic to accelerate guests. This accelerate handling of interrupts delivered directly to guest from passed-through PCI devices. When a virtual IRQ is injected, this feature is disabled in order to route following EOI to virtual APIC. Then, it is enabled again after every virtual IRQ is handled.
---
arch/x86/kvm/vmx.c | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 files changed, 67 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx.c +67 -2
@@ -636,6 +636,10 @@ static unsigned long *vmx_io_bitmap_a;static unsigned long *vmx_io_bitmap_b;static unsigned long *vmx_msr_bitmap_legacy;static unsigned long *vmx_msr_bitmap_longmode; +#ifdef CONFIG_SLAVE_CPU +static unsigned long *vmx_msr_bitmap_slave_legacy; +static unsigned long *vmx_msr_bitmap_slave_longmode; +#endif static bool cpu_has_load_ia32_efer;static bool cpu_has_load_perf_global_ctrl;
@@ -912,6 +916,11 @@ static void nested_vmx_entry_failure(struct kvm_vcpu *vcpu,struct vmcs12 *vmcs12, u32 reason, unsigned long qualification); +static void vmx_disable_intercept_for_msr (u32 msr,bool longmode_only); +#ifdef CONFIG_SLAVE_CPU +static void vmx_disable_intercept_for_msr_slave (u32 msr,bool longmode_only); +#endif +static int __find_msr_index (struct vcpu_vmx *vmx,u32 msr) {int i;
@@ -1716,13 +1725,28 @@ static void vmx_set_direct_interrupt(struct kvm_vcpu *vcpu, bool enabled)#ifdef CONFIG_SLAVE_CPU void *msr_bitmap; -if (enabled) +if (enabled) {vmcs_clear_bits (PIN_BASED_VM_EXEC_CONTROL ,PIN_BASED_EXT_INTR_MASK ); -else + +if (cpu_has_vmx_msr_bitmap ()) { + msr_bitmap =is_long_mode (vcpu) ? + vmx_msr_bitmap_slave_longmode : + vmx_msr_bitmap_slave_legacy; +vmcs_write64 (MSR_BITMAP ,__pa (msr_bitmap)); + } + }else {vmcs_set_bits (PIN_BASED_VM_EXEC_CONTROL ,PIN_BASED_EXT_INTR_MASK ); +if (cpu_has_vmx_msr_bitmap ()) { + msr_bitmap =is_long_mode (vcpu) ? + vmx_msr_bitmap_longmode : + vmx_msr_bitmap_legacy; +vmcs_write64 (MSR_BITMAP ,__pa (msr_bitmap)); + } + } +trace_kvm_set_direct_interrupt (vcpu, enabled);#endif }
@@ -3771,6 +3795,16 @@ static void vmx_disable_intercept_for_msr(u32 msr, bool longmode_only)__vmx_disable_intercept_for_msr (vmx_msr_bitmap_longmode, msr); } +#ifdef CONFIG_SLAVE_CPU +static void vmx_disable_intercept_for_msr_slave (u32 msr,bool longmode_only) +{ +if (!longmode_only) +__vmx_disable_intercept_for_msr (vmx_msr_bitmap_slave_legacy, + msr); +__vmx_disable_intercept_for_msr (vmx_msr_bitmap_slave_longmode, msr); +} +#endif +/* * Set up the vmcs's constant host-state fields, i.e., host-state fields that * will not change in the lifetime of the guest.
@@ -7474,6 +7508,22 @@ static int __init vmx_init(void) goto out2; +#ifdef CONFIG_SLAVE_CPU + vmx_msr_bitmap_slave_legacy = + (unsigned long *)__get_free_page (GFP_KERNEL ); +if (!vmx_msr_bitmap_slave_legacy) { + r = -ENOMEM ; + goto out1s; + } + + vmx_msr_bitmap_slave_longmode = + (unsigned long *)__get_free_page (GFP_KERNEL ); +if (!vmx_msr_bitmap_slave_longmode) { + r = -ENOMEM ; + goto out2s; + } +#endif +/* * Allow direct access to the PC debug port (it is often used for I/O * delays, but the vmexits simply slow things down).
@@ -7500,6 +7550,15 @@ static int __init vmx_init(void)vmx_disable_intercept_for_msr (MSR_IA32_SYSENTER_ESP , false);vmx_disable_intercept_for_msr (MSR_IA32_SYSENTER_EIP , false); +#ifdef CONFIG_SLAVE_CPU +memcpy (vmx_msr_bitmap_slave_legacy, + vmx_msr_bitmap_legacy,PAGE_SIZE ); +memcpy (vmx_msr_bitmap_slave_longmode, + vmx_msr_bitmap_longmode,PAGE_SIZE ); +vmx_disable_intercept_for_msr_slave ( +APIC_BASE_MSR + (APIC_EOI >>4 ), false); +#endif +if (enable_ept) { kvm_mmu_set_mask_ptes(0ull, (enable_ept_ad_bits) ? VMX_EPT_ACCESS_BIT : 0ull,
@@ -7513,6 +7572,12 @@ static int __init vmx_init(void)return 0 ; out3: +#ifdef CONFIG_SLAVE_CPU +free_page ((unsigned long )vmx_msr_bitmap_slave_longmode); +out2s: +free_page ((unsigned long )vmx_msr_bitmap_slave_legacy); +out1s: +#endif free_page ((unsigned long )vmx_msr_bitmap_longmode); out2:free_page ((unsigned long )vmx_msr_bitmap_legacy);
RFC PATCH v2 20/21 KVM: Pass-through local APIC timer of on slave CPUs to guest VM
In reply to Tomoki SekiyamaProvide direct control of local APIC timer of slave CPUs to the guest. The timer interrupt does not cause VM exit if direct interrupt delivery is enabled. To handle the timer correctly, this makes the guest occupy the local APIC timer.
If the host supports x2apic, this expose TMICT and TMCCT to the guest in order to allow guests to start the timer and to read the timer count without VM exit. Otherwise, it sets APIC registers to specified values. LVTT is not passed-through to avoid modifying timer interrupt vector.
Currently the guest timer interrupt vector remapping is not supported, and guest must use the same vector as host.
---
arch/x86/include/asm/apic.h | 4 +++ arch/x86/include/asm/kvm_host.h | 1 + arch/x86/kernel/apic/apic.c | 11 ++++++++++ arch/x86/kernel/smpboot.c | 30 ++++++++++++++++++++++++++ arch/x86/kvm/lapic.c | 45 +++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/lapic.h | 2 ++ arch/x86/kvm/vmx.c | 6 +++++ arch/x86/kvm/x86.c | 3 +++ include/linux/cpu.h | 5 ++++ kernel/hrtimer.c | 2 +- 10 files changed, 108 insertions(+), 1 deletions(-)
arch/x86/include/asm/apic.h +4 -0
@@ -44,6 +44,8 @@ static inline void generic_apic_probe(void)#ifdef CONFIG_X86_LOCAL_APIC +struct clock_event_device ; +extern unsigned int apic_verbosity;extern int local_apic_timer_c2_ok;
@@ -245,6 +247,8 @@ extern void init_apic_mappings(void);void register_lapic_address (unsigned long address);extern void setup_boot_APIC_clock (void );extern void setup_secondary_APIC_clock (void ); +extern void override_local_apic_timer (int cpu, +void (*handler)(struct clock_event_device *));extern int APIC_init_uniprocessor (void );extern int apic_force_enable (unsigned long addr);
arch/x86/include/asm/kvm_host.h +1 -0
@@ -1035,6 +1035,7 @@ int kvm_arch_vcpu_run_prevented(struct kvm_vcpu *vcpu);#ifdef CONFIG_SLAVE_CPU void kvm_get_slave_cpu_mask (struct kvm *kvm,struct cpumask *mask); +struct kvm_vcpu *get_slave_vcpu (int cpu);struct kvm_assigned_dev_kernel ; extern void assign_slave_msi(struct kvm *kvm,
arch/x86/kernel/apic/apic.c +11 -0
@@ -901,6 +901,17 @@ void __irq_entry smp_apic_timer_interrupt(struct pt_regs *regs)set_irq_regs (old_regs); } +void override_local_apic_timer (int cpu, +void (*handler)(struct clock_event_device *)) +{ +unsigned long flags; + +local_irq_save (flags); +per_cpu (lapic_events, cpu).event_handler = handler; +local_irq_restore (flags); +} +EXPORT_SYMBOL_GPL (override_local_apic_timer); +int setup_profiling_timer (unsigned int multiplier) {return -EINVAL ;
arch/x86/kernel/smpboot.c +30 -0
@@ -133,6 +133,7 @@ static void __ref remove_cpu_from_maps(int cpu);#ifdef CONFIG_SLAVE_CPU /* Notify slave cpu up and down */ static RAW_NOTIFIER_HEAD (slave_cpu_chain); +struct notifier_block *slave_timer_nb;int register_slave_cpu_notifier (struct notifier_block *nb) {
@@ -140,6 +141,13 @@ int register_slave_cpu_notifier(struct notifier_block *nb) }EXPORT_SYMBOL (register_slave_cpu_notifier); +int register_slave_cpu_timer_notifier (struct notifier_block *nb) +{ + slave_timer_nb = nb; +return register_slave_cpu_notifier (nb); +} +EXPORT_SYMBOL (register_slave_cpu_timer_notifier); +void unregister_slave_cpu_notifier (struct notifier_block *nb) {raw_notifier_chain_unregister (&slave_cpu_chain, nb);
@@ -155,6 +163,8 @@ static int slave_cpu_notify(unsigned long val, int cpu) return notifier_to_errno(ret); } + +static void slave_cpu_disable_timer (int cpu);#endif /*
@@ -1013,10 +1023,30 @@ int __cpuinit slave_cpu_up(unsigned int cpu)cpu_maps_update_done (); +/* Timer may be used only in starting the slave CPU */ +slave_cpu_disable_timer (cpu); +return ret; }EXPORT_SYMBOL (slave_cpu_up); +static void __slave_cpu_disable_timer (void *hcpu) +{ +int cpu = (long )hcpu; + +pr_info ("Disabling timer on slave cpu %d\n" , cpu); +BUG_ON (!slave_timer_nb); + slave_timer_nb->notifier_call (slave_timer_nb,CPU_SLAVE_DYING , hcpu); +} + +static void slave_cpu_disable_timer (int cpu) +{ +void *hcpu = (void *)(long )cpu; + +slave_cpu_call_function (cpu, __slave_cpu_disable_timer, hcpu); + slave_timer_nb->notifier_call (slave_timer_nb,CPU_SLAVE_DEAD , hcpu); +} +static void __slave_cpu_down (void *dummy) {__this_cpu_write (cpu_state,CPU_DYING );
arch/x86/kvm/lapic.c +45 -0
87 lines not on this page · raw message
arch/x86/kvm/lapic.h +2 -0
@@ -55,6 +55,8 @@ int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu);u64 kvm_get_lapic_tscdeadline_msr (struct kvm_vcpu *vcpu);void kvm_set_lapic_tscdeadline_msr (struct kvm_vcpu *vcpu,u64 data); +void slave_lapic_timer_fn (struct clock_event_device *dev); +void kvm_lapic_set_vapic_addr (struct kvm_vcpu *vcpu,gpa_t vapic_addr);void kvm_lapic_sync_from_vapic (struct kvm_vcpu *vcpu);void kvm_lapic_sync_to_vapic (struct kvm_vcpu *vcpu);
arch/x86/kvm/vmx.c +6 -0
@@ -7557,6 +7557,12 @@ static int __init vmx_init(void) vmx_msr_bitmap_longmode, PAGE_SIZE);vmx_disable_intercept_for_msr_slave (APIC_BASE_MSR + (APIC_EOI >>4 ), false); + +/* Allow guests on slave CPU to access local APIC timer directly */ +vmx_disable_intercept_for_msr_slave ( +APIC_BASE_MSR + (APIC_TMICT >>4 ), false); +vmx_disable_intercept_for_msr_slave ( +APIC_BASE_MSR + (APIC_TMCCT >>4 ), false);#endif if (enable_ept) {
arch/x86/kvm/x86.c +3 -0
@@ -2707,6 +2707,9 @@ static int kvm_arch_vcpu_ioctl_set_slave_cpu(struct kvm_vcpu *vcpu, goto out;BUG_ON (!cpu_slave (slave));per_cpu (slave_vcpu, slave) = vcpu; + +if (vcpu->arch.apic) +override_local_apic_timer (slave, slave_lapic_timer_fn); } vcpu->arch.slave_cpu = slave;
include/linux/cpu.h +5 -0
@@ -223,6 +223,7 @@ static inline void enable_nonboot_cpus(void) {}#ifdef CONFIG_SLAVE_CPU int register_slave_cpu_notifier (struct notifier_block *nb); +int register_slave_cpu_timer_notifier (struct notifier_block *nb);void unregister_slave_cpu_notifier (struct notifier_block *nb);/* CPU notifier constants for slave processors */
@@ -240,6 +241,10 @@ static inline int register_slave_cpu_notifier(struct notifier_block *nb) {return 0 ; } +static inline int register_slave_cpu_timer_notifier (struct notifier_block *nb) +{ +return 0 ; +}static inline void unregister_slave_cpu_notifier (struct notifier_block *nb) {}#endif
kernel/hrtimer.c +1 -1
@@ -1773,7 +1773,7 @@ void __init hrtimers_init(void)hrtimer_cpu_notify (&hrtimers_nb, (unsigned long )CPU_UP_PREPARE , (void *)(long )smp_processor_id ());register_cpu_notifier (&hrtimers_nb); -register_slave_cpu_notifier (&hrtimers_slave_nb); +register_slave_cpu_timer_notifier (&hrtimers_slave_nb);#ifdef CONFIG_HIGH_RES_TIMERS open_softirq (HRTIMER_SOFTIRQ , run_hrtimer_softirq);#endif
RFC PATCH v2 21/21 x86: request TLB flush to slave CPU using NMI
In reply to Tomoki SekiyamaFor slave CPUs, it is inapropriate to request TLB flush using IPI. because the IPI may be sent to a KVM guest when the slave CPU is running the guest with direct interrupt routing.
Instead, it registers a TLB flush request in per-cpu bitmask and send a NMI to interrupt execution of the guest. Then, NMI handler will check the requests and handles the requests.
This implementation has an issue in scalability, and is just for PoC.
---
arch/x86/include/asm/tlbflush.h | 5 ++ arch/x86/kernel/smpboot.c | 3 + arch/x86/kvm/x86.c | 5 ++ arch/x86/mm/tlb.c | 94 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 106 insertions(+), 1 deletions(-)
arch/x86/include/asm/tlbflush.h +5 -0
@@ -168,6 +168,11 @@ static inline void reset_lazy_tlbstate(void) this_cpu_write(cpu_tlbstate.active_mm, &init_mm); } +#ifdef CONFIG_SLAVE_CPU +DECLARE_PER_CPU (bool, slave_idle); +void handle_slave_tlb_flush (unsigned int cpu); +#endif /* SLAVE_CPU */ +#endif /* SMP */ #ifndef CONFIG_PARAVIRT
arch/x86/kernel/smpboot.c +3 -0
@@ -395,7 +395,10 @@ notrace static void __cpuinit start_slave_cpu(void *unused)rcu_note_context_switch (cpu);if (!f.func) { +__this_cpu_write (slave_idle,1 ); +handle_slave_tlb_flush (cpu);native_safe_halt (); +__this_cpu_write (slave_idle,0 );continue ; }
arch/x86/kvm/x86.c +5 -0
@@ -65,6 +65,7 @@#include <asm/cpu.h> #include <asm/nmi.h> #include <asm/mmu.h> +#include <asm/tlbflush.h> #define MAX_IO_MSRS 256#define KVM_MAX_MCE_BANKS 32
@@ -5529,6 +5530,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu, struct task_struct *task)srcu_read_unlock (&vcpu->kvm->srcu, vcpu->srcu_idx); +handle_slave_tlb_flush (vcpu->cpu); +if (req_immediate_exit)smp_send_reschedule (vcpu->cpu);
@@ -5631,6 +5634,8 @@ static void __vcpu_enter_guest_slave(void *_arg) r =vcpu_enter_guest (vcpu, arg->task); +handle_slave_tlb_flush (cpu); +if (r <=0 )break ;
arch/x86/mm/tlb.c +93 -1
126 lines not on this page · raw message
Quoted 11 lines · Tomoki Sekiyama
This RFC patch series provides facility to dedicate CPUs to KVM guests and enable the guests to handle interrupts from passed-through PCI devices directly (without VM exit and relay by the host).
With this feature, we can improve throughput and response time of the device and the host's CPU usage by reducing the overhead of interrupt handling. This is good for the application using very high throughput/frequent interrupt device (e.g. 10GbE NIC).
Real-time applicatoins also gets benefit from CPU isolation feature, which reduces interfare from host kernel tasks and scheduling delay.
Quoted 3 lines
The overview of this patch series is presented in CloudOpen 2012. The slides are available at:
http://events.linuxfoundation.org/images/stories/pdf/lcna_co2012_sekiyama.pdf
During Plumbers 2012, both Intel and AMD disclosed upcoming features to their processors (APIC-V and AVIC) that allow directing device interrupts to guest vcpus without host kernel involvement. This works without pinning, dedicating a core to a guest, or any special measures beyond support for the feature.
CPU isolation is still useful to improve real-time latency further, but this is really independent of kvm.
I am inclined to reject this feature in favour of the new hardware support. Sorry, I know this isn't nice to hear, but the extra maintenance burden cannot be justified for a niche use case with special limitations when generally useful feature exploiting proper hardware support provides the same functionality.
Signature · 1 line
error compiling committee.c: too many arguments to function
Quoted 11 lines · Tomoki Sekiyama
This RFC patch series provides facility to dedicate CPUs to KVM guests and enable the guests to handle interrupts from passed-through PCI devices directly (without VM exit and relay by the host).
With this feature, we can improve throughput and response time of the device and the host's CPU usage by reducing the overhead of interrupt handling. This is good for the application using very high throughput/frequent interrupt device (e.g. 10GbE NIC).
Real-time applicatoins also gets benefit from CPU isolation feature, which reduces interfare from host kernel tasks and scheduling delay.
Quoted 3 lines
The overview of this patch series is presented in CloudOpen 2012. The slides are available at:
http://events.linuxfoundation.org/images/stories/pdf/lcna_co2012_sekiyama.pdf
One question regarding your benchmarks: If you measured against standard KVM, were the vCPU thread running on an isolcpus core of its own as well? If not, your numbers about the impact of these patches on maximum, maybe also average latencies are likely too good. Also, using a non-RT host and possibly a non-prioritized vCPU thread for the standard scenario looks like an unfair comparison.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SDP-DE Corporate Competence Center Embedded Linux
Hi Jan,
Quoted 16 lines · Jan Kiszka
Quoted 11 lines · Tomoki Sekiyama
This RFC patch series provides facility to dedicate CPUs to KVM guests and enable the guests to handle interrupts from passed-through PCI devices directly (without VM exit and relay by the host).
With this feature, we can improve throughput and response time of the device and the host's CPU usage by reducing the overhead of interrupt handling. This is good for the application using very high throughput/frequent interrupt device (e.g. 10GbE NIC).
Real-time applicatoins also gets benefit from CPU isolation feature, which reduces interfare from host kernel tasks and scheduling delay.Quoted 3 lines
The overview of this patch series is presented in CloudOpen 2012. The slides are available at:
http://events.linuxfoundation.org/images/stories/pdf/lcna_co2012_sekiyama.pdf
Quoted 6 lines
One question regarding your benchmarks: If you measured against standard KVM, were the vCPU thread running on an isolcpus core of its own as well? If not, your numbers about the impact of these patches on maximum, maybe also average latencies are likely too good. Also, using a non-RT host and possibly a non-prioritized vCPU thread for the standard scenario looks like an unfair comparison.
In the standard KVM benchmark, the vCPU thread is pinned down to its own CPU core. In addition, the vCPU thread and irq/*-kvm threads are both set to the max priority with SCHED_RR policy.
As you said, RT-host may result in better max latencies as below. (But min/average latencies became worse, however, this might be our setup
issue.)
Min / Avg / Max latencies
Normal KVM
RT-host (3.4.4-rt14) 15us / 21us / 117us
non RT-host (3.5.0-rc6) 6us / 11us / 152us
KVM + Direct IRQ
non RT-host (3.5.0-rc6 +patch) 1us / 2us / 14us
Thanks,
Signature · 3 lines
Tomoki Sekiyama <[email protected]> Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory