Skip to contentSkip to conversation

KVM over Xenomai and I-pipe

RFC In review Jan Kiszka 2 participants 8 messages Mar 12, 2010 – Apr 26, 2010

Subsystems (2)
lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

RFC 1/1 KVM over Xenomai and I-pipe

Hi,

this is still in the state "study", but it is working fairly nicely so far:

These two patches harden latest KVM for use over I-pipe kernels and make Xenomai aware of the lazy host state restoring that KVM uses for performance reasons. The latter basically means calling the sched-out notifier that KVM registers with the kernel when switching from a Linux task to some shadow. This is safe in all recent versions of KVM and still gives nice KVM performance (that of KVM before 2.6.32) without significant impact on the RT latency (Note: if you have an old VT-x CPU, guest-issued wbinvd will ruin RT as it is not intercepted by the hardware!).

To test it, you need to apply the kernel patch on top of current kvm.git master [1], obtain kvm-kmod.git [2], run configure on it (assuming your host kernel is a Xenomai one, otherwise use --kerneldir) and then "make sync-kmod LINUX=/path/to/kvm.git". After a final make && make install, you will have recent kvm modules that are I-pipe aware. The Xenomai patch simply appies to the 2.5 tree. This has been tested with ipipe-2.6.32-x86-2.6-01 + [3] and Xenomai-2.5 git.

Feedback welcome, specifically if you think it's worth integrating both patches into upstream. The kernel bits would make sense over some 2.6.33-x86, but additional work will be required to account for the user-return notifiers introduced with that release (kvm-kmod currently wraps them away for older kernels).

Jan

[1]git://git.kernel.org/pub/scm/virt/kvm/kvm.git [2]git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git [3]http://git.kiszka.org/?p=ipipe-2.6.git;a=commitdiff;h=0bddff1716aba6dd5ca11627ee377a5a25fa3dae

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
Quoted 1 line

From 55480e98b8f35818a838bb2bd1f24764276a9b17 Mon Sep 17 00:00:00 2001

From: Jan Kiszka <[email protected]>
Date: Wed, 10 Mar 2010 08:32:02 +0100
Subject: [PATCH] Harden KVM for use over I-pipe

This allows to use KVM on I-pipe-enabled kernels. I-pipe domains that preempt a VCPU task and let the preempting task return to user space additionally have to fire the sched-out notifiers of VCPU task with IRQs disable. Those will restore host states that are lazily switched for performance reasons.

Tested with modfied Xenomai on Intel, should work with AMD hosts as well.

---
 arch/x86/kvm/svm.c |    4 ++--
 arch/x86/kvm/vmx.c |   10 ++++++----
 arch/x86/kvm/x86.c |   11 +++++++++--
 3 files changed, 17 insertions(+), 8 deletions(-)
arch/x86/kvm/svm.c +2 -2
index def4877..7e81639 100644
@@ -3025,7 +3025,7 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
 
 	clgi();
 
-	local_irq_enable();
+	local_irq_enable_hw();
 
 	asm volatile (
 		"push %%"R"bp; \n\t"
@@ -3110,7 +3110,7 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
 
 	reload_tss(vcpu);
 
-	local_irq_disable();
+	local_irq_disable_hw();
 
 	stgi();
 
arch/x86/kvm/vmx.c +6 -4
index ae3217d..741e2a1 100644
@@ -736,12 +736,12 @@ static void __vmx_load_host_state(struct vcpu_vmx *vmx)
 		 * If we have to reload gs, we must take care to
 		 * preserve our gs base.
 		 */
-		local_irq_save(flags);
+		local_irq_save_hw(flags);
 		kvm_load_gs(vmx->host_state.gs_sel);
 #ifdef CONFIG_X86_64
 		wrmsrl(MSR_GS_BASE, vmcs_readl(HOST_GS_BASE));
 #endif
-		local_irq_restore(flags);
+		local_irq_restore_hw(flags);
 	}
 	reload_tss();
 #ifdef CONFIG_X86_64
@@ -754,9 +754,11 @@ static void __vmx_load_host_state(struct vcpu_vmx *vmx)
 
 static void vmx_load_host_state(struct vcpu_vmx *vmx)
 {
-	preempt_disable();
+	unsigned long flags;
+
+	ipipe_preempt_disable(flags);
 	__vmx_load_host_state(vmx);
-	preempt_enable();
+	ipipe_preempt_enable(flags);
 }
 
 /*
arch/x86/kvm/x86.c +9 -2
index 703f637..713a392 100644
@@ -1657,8 +1657,12 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 
 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 {
+	unsigned long flags;
+
+	local_irq_save_hw_cond(flags);
 	kvm_put_guest_fpu(vcpu);
 	kvm_x86_ops->vcpu_put(vcpu);
+	local_irq_restore_hw_cond(flags);
 }
 
 static int is_efer_nx(void)
@@ -4332,17 +4336,19 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 
 	preempt_disable();
 
+	local_irq_disable();
+	local_irq_disable_hw();
+
 	kvm_x86_ops->prepare_guest_switch(vcpu);
 	if (vcpu->fpu_active)
 		kvm_load_guest_fpu(vcpu);
 
-	local_irq_disable();
-
 	clear_bit(KVM_REQ_KICK, &vcpu->requests);
 	smp_mb__after_clear_bit();
 
 	if (vcpu->requests || need_resched() || signal_pending(current)) {
 		set_bit(KVM_REQ_KICK, &vcpu->requests);
+		local_irq_enable_hw();
 		local_irq_enable();
 		preempt_enable();
 		r = 1;
@@ -4388,6 +4394,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 		hw_breakpoint_restore();
 
 	set_bit(KVM_REQ_KICK, &vcpu->requests);
+	local_irq_enable_hw();
 	local_irq_enable();
 
 	++vcpu->stat.exits;
Signature · 1 line
1.6.0.2
Quoted 1 line

From 618e445548d38f712c4d5d108da627fd30207631 Mon Sep 17 00:00:00 2001

From: Jan Kiszka <[email protected]>
Date: Wed, 10 Mar 2010 10:35:36 +0100
Subject: [PATCH] Enable KVM on Xenomai kernels

Call the sched-out notifier (so far this can only be kvm_sched_out) when switching Linux tasks. This restores the complete host state after a VM exit and allows shadow threads to safely preempt VCPU threads.

---
 include/asm-x86/bits/pod_64.h |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
include/asm-x86/bits/pod_64.h +9 -0
index 88e049d..96793f7 100644
@@ -68,6 +68,15 @@ static inline void xnarch_switch_to(xnarchtcb_t *out_tcb, xnarchtcb_t *in_tcb)
 	struct task_struct *next = in_tcb->user_task;
 
 	if (likely(next != NULL)) {
+#ifdef CONFIG_PREEMPT_NOTIFIERS
+		struct preempt_notifier *notifier;
+		struct hlist_node *node;
+
+		hlist_for_each_entry(notifier, node, &prev->preempt_notifiers,
+				     link)
+			notifier->ops->sched_out(notifier, next);
+#endif
+
 		if (task_thread_info(prev)->status & TS_USEDFPU)
 			/*
 			 * __switch_to will try and use __unlazy_fpu,
Signature · 1 line
1.6.0.2
#1 lore raw
Quoted 21 lines · Jan Kiszka

Hi,

this is still in the state "study", but it is working fairly nicely so far:

These two patches harden latest KVM for use over I-pipe kernels and make Xenomai aware of the lazy host state restoring that KVM uses for performance reasons. The latter basically means calling the sched-out notifier that KVM registers with the kernel when switching from a Linux task to some shadow. This is safe in all recent versions of KVM and still gives nice KVM performance (that of KVM before 2.6.32) without significant impact on the RT latency (Note: if you have an old VT-x CPU, guest-issued wbinvd will ruin RT as it is not intercepted by the hardware!).

To test it, you need to apply the kernel patch on top of current kvm.git master [1], obtain kvm-kmod.git [2], run configure on it (assuming your host kernel is a Xenomai one, otherwise use --kerneldir) and then "make sync-kmod LINUX=/path/to/kvm.git". After a final make && make install, you will have recent kvm modules that are I-pipe aware. The Xenomai patch simply appies to the 2.5 tree. This has been tested with ipipe-2.6.32-x86-2.6-01 + [3] and Xenomai-2.5 git.

Quoted 5 lines

Feedback welcome, specifically if you think it's worth integrating both patches into upstream. The kernel bits would make sense over some 2.6.33-x86, but additional work will be required to account for the user-return notifiers introduced with that release (kvm-kmod currently wraps them away for older kernels).

No concern on the final goal, running a Xenomai-enabled kernel rock-solid over KVM is a must.

The KVM code ironing from the 1st patch looks fine to me, no big deal to maintain AFAICS. I would be only concerned by the 2nd patch, specifically how the KVM callout is invoked from the Xenomai context switching code:

- depending on CONFIG_PREEMPT_NOTIFIERS is much broader than required; I guess that CONFIG_KVM would be enough.

- calling the KVM callout directly instead of going through the notifier list would be more acceptable, so that we don't assume anything from the non-KVM hooks (whether they exist or not), albeit we may assume that we have complete information about which KVM callout has to be run for a particular kernel version.

But again, rock-solid KVM+Xenomai is a much desirable feature.

Quoted 11 lines

Jan

[1]git://git.kernel.org/pub/scm/virt/kvm/kvm.git [2]git://git.kernel.org/pub/scm/virt/kvm/kvm-kmod.git [3]http://git.kiszka.org/?p=ipipe-2.6.git;a=commitdiff;h=0bddff1716aba6dd5ca11627ee377a5a25fa3dae

_______________________________________________
Xenomai-core mailing list
[email protected]
https://mail.gna.org/listinfo/xenomai-core

Signature · 1 line
Philippe.
author
#2 lore raw

[ dropping xenomai-help before going into details ]

Quoted 31 lines · Philippe Gerum
Quoted 21 lines · Jan Kiszka

Hi,

this is still in the state "study", but it is working fairly nicely so far:

These two patches harden latest KVM for use over I-pipe kernels and make Xenomai aware of the lazy host state restoring that KVM uses for performance reasons. The latter basically means calling the sched-out notifier that KVM registers with the kernel when switching from a Linux task to some shadow. This is safe in all recent versions of KVM and still gives nice KVM performance (that of KVM before 2.6.32) without significant impact on the RT latency (Note: if you have an old VT-x CPU, guest-issued wbinvd will ruin RT as it is not intercepted by the hardware!).

To test it, you need to apply the kernel patch on top of current kvm.git master [1], obtain kvm-kmod.git [2], run configure on it (assuming your host kernel is a Xenomai one, otherwise use --kerneldir) and then "make sync-kmod LINUX=/path/to/kvm.git". After a final make && make install, you will have recent kvm modules that are I-pipe aware. The Xenomai patch simply appies to the 2.5 tree. This has been tested with ipipe-2.6.32-x86-2.6-01 + [3] and Xenomai-2.5 git.

Quoted 5 lines

Feedback welcome, specifically if you think it's worth integrating both patches into upstream. The kernel bits would make sense over some 2.6.33-x86, but additional work will be required to account for the user-return notifiers introduced with that release (kvm-kmod currently wraps them away for older kernels).

No concern on the final goal, running a Xenomai-enabled kernel rock-solid over KVM is a must.

Quoted 7 lines

The KVM code ironing from the 1st patch looks fine to me, no big deal to maintain AFAICS. I would be only concerned by the 2nd patch, specifically how the KVM callout is invoked from the Xenomai context switching code:

- depending on CONFIG_PREEMPT_NOTIFIERS is much broader than required; I guess that CONFIG_KVM would be enough.

So far, only CONFIG_KVM enables CONFIG_PREEMPT_NOTIFIERS. Granted, this could change in the future. But letting our invocation depend on CONFIG_KVM would not automatically remove the need to review those new notifiers (BTW, there would be a fairly high probability that those will be of some use for Xenomai as well).

Quoted 6 lines

- calling the KVM callout directly instead of going through the notifier list would be more acceptable, so that we don't assume anything from the non-KVM hooks (whether they exist or not), albeit we may assume that we have complete information about which KVM callout has to be run for a particular kernel version.

Possible, but hacky. We would have to

- export the callback from the KVM module
(this will also mean the nucleus will depend on CONFIG_KVM if the latter is on)
- somehow get hold of the notifier entry (I have no clue how as they are per-vcpu)
- invoke the callback directly, passing that notifier entry

or

- identify the KVM callback in the notifier chain and only call that one when walking the list

The latter could be achieved by somehow tagging KVM notifiers in order to find them when walking the chain. Still quite some patching, and I'm not yet sure it's worth the safety gain.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
#3 lore raw
Quoted 55 lines · Jan Kiszka

[ dropping xenomai-help before going into details ]

Quoted 31 lines · Philippe Gerum
Quoted 21 lines · Jan Kiszka

Hi,

this is still in the state "study", but it is working fairly nicely so far:

These two patches harden latest KVM for use over I-pipe kernels and make Xenomai aware of the lazy host state restoring that KVM uses for performance reasons. The latter basically means calling the sched-out notifier that KVM registers with the kernel when switching from a Linux task to some shadow. This is safe in all recent versions of KVM and still gives nice KVM performance (that of KVM before 2.6.32) without significant impact on the RT latency (Note: if you have an old VT-x CPU, guest-issued wbinvd will ruin RT as it is not intercepted by the hardware!).

To test it, you need to apply the kernel patch on top of current kvm.git master [1], obtain kvm-kmod.git [2], run configure on it (assuming your host kernel is a Xenomai one, otherwise use --kerneldir) and then "make sync-kmod LINUX=/path/to/kvm.git". After a final make && make install, you will have recent kvm modules that are I-pipe aware. The Xenomai patch simply appies to the 2.5 tree. This has been tested with ipipe-2.6.32-x86-2.6-01 + [3] and Xenomai-2.5 git.

Quoted 5 lines

Feedback welcome, specifically if you think it's worth integrating both patches into upstream. The kernel bits would make sense over some 2.6.33-x86, but additional work will be required to account for the user-return notifiers introduced with that release (kvm-kmod currently wraps them away for older kernels).

No concern on the final goal, running a Xenomai-enabled kernel rock-solid over KVM is a must.

Quoted 7 lines

The KVM code ironing from the 1st patch looks fine to me, no big deal to maintain AFAICS. I would be only concerned by the 2nd patch, specifically how the KVM callout is invoked from the Xenomai context switching code:

- depending on CONFIG_PREEMPT_NOTIFIERS is much broader than required; I guess that CONFIG_KVM would be enough.

So far, only CONFIG_KVM enables CONFIG_PREEMPT_NOTIFIERS. Granted, this could change in the future. But letting our invocation depend on CONFIG_KVM would not automatically remove the need to review those new notifiers (BTW, there would be a fairly high probability that those will be of some use for Xenomai as well).

Quoted 6 lines

- calling the KVM callout directly instead of going through the notifier list would be more acceptable, so that we don't assume anything from the non-KVM hooks (whether they exist or not), albeit we may assume that we have complete information about which KVM callout has to be run for a particular kernel version.

Quoted 5 lines

Possible, but hacky. We would have to

- export the callback from the KVM module
(this will also mean the nucleus will depend on CONFIG_KVM if the latter is on)

Which is already the case for a number of knobs anyway (particularly on x86*).

Quoted 4 lines

- somehow get hold of the notifier entry (I have no clue how as they are per-vcpu)
- invoke the callback directly, passing that notifier entry

This is what I had in mind in my post.

Quoted 4 lines

or

- identify the KVM callback in the notifier chain and only call that one when walking the list

I don't see any upside to this yet. If this is about context preparation that would be done by the notification system, then we'd better off mimicking it, instead of introducing kludges to reuse it.

Quoted 4 lines

The latter could be achieved by somehow tagging KVM notifiers in order to find them when walking the chain. Still quite some patching, and I'm not yet sure it's worth the safety gain.

The point is that we shall check whether our coupling to the KVM system is correct, for each kernel version we want to support anyway. This means that some preparation work has to be done, whether it is by inspecting the possibly NMI-unsafe notifier hooks or the interface rules to the KVM hook is not the most important thing here.

If you definition of "hacky" here means "ad hoc", in any case, any implementation you could find would be hacky, because Xenomai introduces a context switching spot in a kernel that does not expect it, and as such, we do bypass the normal paths for this. Therefore, I see no way to do this without exactly knowing the kernel/KVM context, on a per-release basis.

Quoted 3 lines

Jan

Signature · 1 line
Philippe.
#4 lore raw
Quoted 65 lines · Philippe Gerum
Quoted 55 lines · Jan Kiszka

[ dropping xenomai-help before going into details ]

Quoted 31 lines · Philippe Gerum
Quoted 21 lines · Jan Kiszka

Hi,

this is still in the state "study", but it is working fairly nicely so far:

These two patches harden latest KVM for use over I-pipe kernels and make Xenomai aware of the lazy host state restoring that KVM uses for performance reasons. The latter basically means calling the sched-out notifier that KVM registers with the kernel when switching from a Linux task to some shadow. This is safe in all recent versions of KVM and still gives nice KVM performance (that of KVM before 2.6.32) without significant impact on the RT latency (Note: if you have an old VT-x CPU, guest-issued wbinvd will ruin RT as it is not intercepted by the hardware!).

To test it, you need to apply the kernel patch on top of current kvm.git master [1], obtain kvm-kmod.git [2], run configure on it (assuming your host kernel is a Xenomai one, otherwise use --kerneldir) and then "make sync-kmod LINUX=/path/to/kvm.git". After a final make && make install, you will have recent kvm modules that are I-pipe aware. The Xenomai patch simply appies to the 2.5 tree. This has been tested with ipipe-2.6.32-x86-2.6-01 + [3] and Xenomai-2.5 git.

Quoted 5 lines

Feedback welcome, specifically if you think it's worth integrating both patches into upstream. The kernel bits would make sense over some 2.6.33-x86, but additional work will be required to account for the user-return notifiers introduced with that release (kvm-kmod currently wraps them away for older kernels).

No concern on the final goal, running a Xenomai-enabled kernel rock-solid over KVM is a must.

Quoted 7 lines

The KVM code ironing from the 1st patch looks fine to me, no big deal to maintain AFAICS. I would be only concerned by the 2nd patch, specifically how the KVM callout is invoked from the Xenomai context switching code:

- depending on CONFIG_PREEMPT_NOTIFIERS is much broader than required; I guess that CONFIG_KVM would be enough.

So far, only CONFIG_KVM enables CONFIG_PREEMPT_NOTIFIERS. Granted, this could change in the future. But letting our invocation depend on CONFIG_KVM would not automatically remove the need to review those new notifiers (BTW, there would be a fairly high probability that those will be of some use for Xenomai as well).

Quoted 6 lines

- calling the KVM callout directly instead of going through the notifier list would be more acceptable, so that we don't assume anything from the non-KVM hooks (whether they exist or not), albeit we may assume that we have complete information about which KVM callout has to be run for a particular kernel version.

Quoted 5 lines

Possible, but hacky. We would have to

- export the callback from the KVM module
(this will also mean the nucleus will depend on CONFIG_KVM if the latter is on)

Which is already the case for a number of knobs anyway (particularly on x86*).

Quoted 6 lines
Quoted 4 lines

- somehow get hold of the notifier entry (I have no clue how as they are per-vcpu)
- invoke the callback directly, passing that notifier entry

This is what I had in mind in my post.

Sorry, wrong read: what I had in mind, was simply to identify the KVM hook within the code, and forge a correct call interface, whatever this means (i.e. with the original notifier entry, or by providing a second hook entry point which would not require such notifier entry).

Quoted 33 lines
Quoted 5 lines

or

- identify the KVM callback in the notifier chain and only call that one when walking the list

I don't see any upside to this yet. If this is about context preparation that would be done by the notification system, then we'd better off mimicking it, instead of introducing kludges to reuse it.

Quoted 4 lines

The latter could be achieved by somehow tagging KVM notifiers in order to find them when walking the chain. Still quite some patching, and I'm not yet sure it's worth the safety gain.

The point is that we shall check whether our coupling to the KVM system is correct, for each kernel version we want to support anyway. This means that some preparation work has to be done, whether it is by inspecting the possibly NMI-unsafe notifier hooks or the interface rules to the KVM hook is not the most important thing here.

If you definition of "hacky" here means "ad hoc", in any case, any implementation you could find would be hacky, because Xenomai introduces a context switching spot in a kernel that does not expect it, and as such, we do bypass the normal paths for this. Therefore, I see no way to do this without exactly knowing the kernel/KVM context, on a per-release basis.

Quoted 3 lines

Jan

Signature · 1 line
Philippe.
author
#5 lore raw
Quoted 53 lines · Philippe Gerum
Quoted 52 lines · Philippe Gerum
Quoted 51 lines · Jan Kiszka

[ dropping xenomai-help before going into details ]

Quoted 30 lines · Philippe Gerum
Quoted 21 lines · Jan Kiszka

Hi,

this is still in the state "study", but it is working fairly nicely so far:

These two patches harden latest KVM for use over I-pipe kernels and make Xenomai aware of the lazy host state restoring that KVM uses for performance reasons. The latter basically means calling the sched-out notifier that KVM registers with the kernel when switching from a Linux task to some shadow. This is safe in all recent versions of KVM and still gives nice KVM performance (that of KVM before 2.6.32) without significant impact on the RT latency (Note: if you have an old VT-x CPU, guest-issued wbinvd will ruin RT as it is not intercepted by the hardware!).

To test it, you need to apply the kernel patch on top of current kvm.git master [1], obtain kvm-kmod.git [2], run configure on it (assuming your host kernel is a Xenomai one, otherwise use --kerneldir) and then "make sync-kmod LINUX=/path/to/kvm.git". After a final make && make install, you will have recent kvm modules that are I-pipe aware. The Xenomai patch simply appies to the 2.5 tree. This has been tested with ipipe-2.6.32-x86-2.6-01 + [3] and Xenomai-2.5 git.

Quoted 5 lines

Feedback welcome, specifically if you think it's worth integrating both patches into upstream. The kernel bits would make sense over some 2.6.33-x86, but additional work will be required to account for the user-return notifiers introduced with that release (kvm-kmod currently wraps them away for older kernels).

No concern on the final goal, running a Xenomai-enabled kernel rock-solid over KVM is a must.

Quoted 7 lines

The KVM code ironing from the 1st patch looks fine to me, no big deal to maintain AFAICS. I would be only concerned by the 2nd patch, specifically how the KVM callout is invoked from the Xenomai context switching code:

- depending on CONFIG_PREEMPT_NOTIFIERS is much broader than required; I guess that CONFIG_KVM would be enough.

So far, only CONFIG_KVM enables CONFIG_PREEMPT_NOTIFIERS. Granted, this could change in the future. But letting our invocation depend on CONFIG_KVM would not automatically remove the need to review those new notifiers (BTW, there would be a fairly high probability that those will be of some use for Xenomai as well).

Quoted 5 lines

- calling the KVM callout directly instead of going through the notifier list would be more acceptable, so that we don't assume anything from the non-KVM hooks (whether they exist or not), albeit we may assume that we have complete information about which KVM callout has to be run for a particular kernel version.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Possible, but hacky. We would have to

- export the callback from the KVM module
(this will also mean the nucleus will depend on CONFIG_KVM if the latter is on)

Which is already the case for a number of knobs anyway (particularly on x86*).

The difference is that kvm can be configured as _module_. Simply exporting won't be enough.

Quoted 11 lines
Quoted 6 lines
Quoted 5 lines

- somehow get hold of the notifier entry (I have no clue how as they are per-vcpu)
- invoke the callback directly, passing that notifier entry

This is what I had in mind in my post.

Sorry, wrong read: what I had in mind, was simply to identify the KVM hook within the code, and forge a correct call interface, whatever this means (i.e. with the original notifier entry, or by providing a second hook entry point which would not require such notifier entry).

As KVM registers dynamically with the notifier chain (when the corresponding VCPU is scheduled in an out), getting the right context is tricky unless you reuse the notifier chain or let I-pipe provide another callback interface.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

or

- identify the KVM callback in the notifier chain and only call that one when walking the list

I don't see any upside to this yet. If this is about context preparation that would be done by the notification system, then we'd better off mimicking it, instead of introducing kludges to reuse it.

Mimicking will mean (almost) 1:1 copying.

Quoted 10 lines
Quoted 10 lines
Quoted 4 lines

The latter could be achieved by somehow tagging KVM notifiers in order to find them when walking the chain. Still quite some patching, and I'm not yet sure it's worth the safety gain.

The point is that we shall check whether our coupling to the KVM system is correct, for each kernel version we want to support anyway. This means that some preparation work has to be done, whether it is by inspecting the possibly NMI-unsafe notifier hooks or the interface rules to the KVM hook is not the most important thing here.

Quoted 6 lines
Quoted 6 lines

If you definition of "hacky" here means "ad hoc", in any case, any implementation you could find would be hacky, because Xenomai introduces a context switching spot in a kernel that does not expect it, and as such, we do bypass the normal paths for this. Therefore, I see no way to do this without exactly knowing the kernel/KVM context, on a per-release basis.

Right, that's what we already have to know in order to reuse e.g. switch_mm safely. The preempt notifier plays in the same league as they are there to inform subsystems about such kind of switches.

So we have two basic options:
- patch KVM to additionally register callbacks with I-pipe (ipipe_preempt_notifiers)
- reuse the existing sched_out notifier, keeping an eye on potential new users (they exist since 2.6.23 - without anyone else showing interest so far)

In both case, we will pull some tricky parts of KVM on our review list, that's unavoidable. But as long as we reuse well-established interfaces for this, I'm not too concerned about this.

Jan

#6 lore raw
Quoted 56 lines · Jan Kiszka
Quoted 55 lines · Philippe Gerum
Quoted 54 lines · Philippe Gerum
Quoted 53 lines · Jan Kiszka

[ dropping xenomai-help before going into details ]

Quoted 30 lines · Philippe Gerum
Quoted 21 lines · Jan Kiszka

Hi,

this is still in the state "study", but it is working fairly nicely so far:

These two patches harden latest KVM for use over I-pipe kernels and make Xenomai aware of the lazy host state restoring that KVM uses for performance reasons. The latter basically means calling the sched-out notifier that KVM registers with the kernel when switching from a Linux task to some shadow. This is safe in all recent versions of KVM and still gives nice KVM performance (that of KVM before 2.6.32) without significant impact on the RT latency (Note: if you have an old VT-x CPU, guest-issued wbinvd will ruin RT as it is not intercepted by the hardware!).

To test it, you need to apply the kernel patch on top of current kvm.git master [1], obtain kvm-kmod.git [2], run configure on it (assuming your host kernel is a Xenomai one, otherwise use --kerneldir) and then "make sync-kmod LINUX=/path/to/kvm.git". After a final make && make install, you will have recent kvm modules that are I-pipe aware. The Xenomai patch simply appies to the 2.5 tree. This has been tested with ipipe-2.6.32-x86-2.6-01 + [3] and Xenomai-2.5 git.

Quoted 5 lines

Feedback welcome, specifically if you think it's worth integrating both patches into upstream. The kernel bits would make sense over some 2.6.33-x86, but additional work will be required to account for the user-return notifiers introduced with that release (kvm-kmod currently wraps them away for older kernels).

No concern on the final goal, running a Xenomai-enabled kernel rock-solid over KVM is a must.

Quoted 7 lines

The KVM code ironing from the 1st patch looks fine to me, no big deal to maintain AFAICS. I would be only concerned by the 2nd patch, specifically how the KVM callout is invoked from the Xenomai context switching code:

- depending on CONFIG_PREEMPT_NOTIFIERS is much broader than required; I guess that CONFIG_KVM would be enough.

So far, only CONFIG_KVM enables CONFIG_PREEMPT_NOTIFIERS. Granted, this could change in the future. But letting our invocation depend on CONFIG_KVM would not automatically remove the need to review those new notifiers (BTW, there would be a fairly high probability that those will be of some use for Xenomai as well).

Quoted 5 lines

- calling the KVM callout directly instead of going through the notifier list would be more acceptable, so that we don't assume anything from the non-KVM hooks (whether they exist or not), albeit we may assume that we have complete information about which KVM callout has to be run for a particular kernel version.

Possible, but hacky. We would have to

Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines

- export the callback from the KVM module
(this will also mean the nucleus will depend on CONFIG_KVM if the latter is on)

Which is already the case for a number of knobs anyway (particularly on x86*).

The difference is that kvm can be configured as _module_. Simply exporting won't be enough.

Quite frankly, I see no showstopper in forcing a statically built KVM whenever Xenomai is enabled, provided we do that onmy when say, CONFIG_XENOMAI_VMCLIENT is switched on. Would you see a significant feature loss in removing modular support for KVM in this context?

Quoted 20 lines
Quoted 11 lines
Quoted 6 lines
Quoted 5 lines

- somehow get hold of the notifier entry (I have no clue how as they are per-vcpu)
- invoke the callback directly, passing that notifier entry

This is what I had in mind in my post.

Sorry, wrong read: what I had in mind, was simply to identify the KVM hook within the code, and forge a correct call interface, whatever this means (i.e. with the original notifier entry, or by providing a second hook entry point which would not require such notifier entry).

As KVM registers dynamically with the notifier chain (when the corresponding VCPU is scheduled in an out), getting the right context is tricky unless you reuse the notifier chain or let I-pipe provide another callback interface.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

or

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

- identify the KVM callback in the notifier chain and only call that one when walking the list

I don't see any upside to this yet. If this is about context preparation that would be done by the notification system, then we'd better off mimicking it, instead of introducing kludges to reuse it.

Mimicking will mean (almost) 1:1 copying.

Yes, for sure. That's the price to pay, I guess, like for anything we must reuse from the innards.

Quoted 29 lines
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines

The latter could be achieved by somehow tagging KVM notifiers in order to find them when walking the chain. Still quite some patching, and I'm not yet sure it's worth the safety gain.

The point is that we shall check whether our coupling to the KVM system is correct, for each kernel version we want to support anyway. This means that some preparation work has to be done, whether it is by inspecting the possibly NMI-unsafe notifier hooks or the interface rules to the KVM hook is not the most important thing here.

Quoted 6 lines
Quoted 6 lines

If you definition of "hacky" here means "ad hoc", in any case, any implementation you could find would be hacky, because Xenomai introduces a context switching spot in a kernel that does not expect it, and as such, we do bypass the normal paths for this. Therefore, I see no way to do this without exactly knowing the kernel/KVM context, on a per-release basis.

Right, that's what we already have to know in order to reuse e.g. switch_mm safely. The preempt notifier plays in the same league as they are there to inform subsystems about such kind of switches.

So we have two basic options:
- patch KVM to additionally register callbacks with I-pipe (ipipe_preempt_notifiers)
- reuse the existing sched_out notifier, keeping an eye on potential new users (they exist since 2.6.23 - without anyone else showing interest so far)

Quoted 4 lines

In both case, we will pull some tricky parts of KVM on our review list, that's unavoidable. But as long as we reuse well-established interfaces for this, I'm not too concerned about this.

Ack.

Quoted 2 lines

Jan

Signature · 1 line
Philippe.
author
#7 lore raw
Quoted 62 lines · Philippe Gerum
Quoted 61 lines · Jan Kiszka
Quoted 60 lines · Philippe Gerum
Quoted 59 lines · Philippe Gerum
Quoted 51 lines · Jan Kiszka

[ dropping xenomai-help before going into details ]

Quoted 30 lines · Philippe Gerum
Quoted 21 lines · Jan Kiszka

Hi,

this is still in the state "study", but it is working fairly nicely so far:

These two patches harden latest KVM for use over I-pipe kernels and make Xenomai aware of the lazy host state restoring that KVM uses for performance reasons. The latter basically means calling the sched-out notifier that KVM registers with the kernel when switching from a Linux task to some shadow. This is safe in all recent versions of KVM and still gives nice KVM performance (that of KVM before 2.6.32) without significant impact on the RT latency (Note: if you have an old VT-x CPU, guest-issued wbinvd will ruin RT as it is not intercepted by the hardware!).

To test it, you need to apply the kernel patch on top of current kvm.git master [1], obtain kvm-kmod.git [2], run configure on it (assuming your host kernel is a Xenomai one, otherwise use --kerneldir) and then "make sync-kmod LINUX=/path/to/kvm.git". After a final make && make install, you will have recent kvm modules that are I-pipe aware. The Xenomai patch simply appies to the 2.5 tree. This has been tested with ipipe-2.6.32-x86-2.6-01 + [3] and Xenomai-2.5 git.

Quoted 5 lines

Feedback welcome, specifically if you think it's worth integrating both patches into upstream. The kernel bits would make sense over some 2.6.33-x86, but additional work will be required to account for the user-return notifiers introduced with that release (kvm-kmod currently wraps them away for older kernels).

No concern on the final goal, running a Xenomai-enabled kernel rock-solid over KVM is a must.

Quoted 7 lines

The KVM code ironing from the 1st patch looks fine to me, no big deal to maintain AFAICS. I would be only concerned by the 2nd patch, specifically how the KVM callout is invoked from the Xenomai context switching code:

- depending on CONFIG_PREEMPT_NOTIFIERS is much broader than required; I guess that CONFIG_KVM would be enough.

So far, only CONFIG_KVM enables CONFIG_PREEMPT_NOTIFIERS. Granted, this could change in the future. But letting our invocation depend on CONFIG_KVM would not automatically remove the need to review those new notifiers (BTW, there would be a fairly high probability that those will be of some use for Xenomai as well).

Quoted 5 lines

- calling the KVM callout directly instead of going through the notifier list would be more acceptable, so that we don't assume anything from the non-KVM hooks (whether they exist or not), albeit we may assume that we have complete information about which KVM callout has to be run for a particular kernel version.

Quoted 5 lines

Possible, but hacky. We would have to

- export the callback from the KVM module
(this will also mean the nucleus will depend on CONFIG_KVM if the latter is on)

Which is already the case for a number of knobs anyway (particularly on x86*).

Quoted 8 lines
Quoted 3 lines

The difference is that kvm can be configured as _module_. Simply exporting won't be enough.

Quite frankly, I see no showstopper in forcing a statically built KVM whenever Xenomai is enabled, provided we do that onmy when say, CONFIG_XENOMAI_VMCLIENT is switched on. Would you see a significant feature loss in removing modular support for KVM in this context?

I finally realized that, due to the dynamic nature of KVM's registration, this would not work at all. We need a callback that KVM actively registers (along with the right cookie).

Moreover, it makes sense to do the deregistration atomically along voluntary sched_out calls of KVM. So establishing some tiny ipipe_preempt_notifier has its benefits (or is even mandatory). Will hack up something like that for 2.6.34 and come back once it works.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux