X86: Deliver Hyper-V interrupts on a seperate IDT vector
Subsystems (3)
cover X86: Deliver Hyper-V interrupts on a seperate IDT vector
This patch-set implements the functionality to deliver Hyper-V VMBUS interrupts via a special IDT entry. Xen emulates Hyper-V and we have added code in this patch-set to properly manage this emulation when Linux is running on Hyper-V.
K. Y. Srinivasan (2):
X86: Add a check to catch Xen emulation of Hyper-V
X86: Handle Hyper-V vmbus interrupts as special hypervisor interrupts
Olaf Hering (1):
x86: Hyper-V: register clocksource only if its advertised
arch/x86/include/asm/irq_vectors.h | 4 +- arch/x86/include/asm/mshyperv.h | 4 +++ arch/x86/kernel/cpu/mshyperv.c | 48 +++++++++++++++++++++++++++++++++++- arch/x86/kernel/entry_32.S | 9 ++++++- arch/x86/kernel/entry_64.S | 7 ++++- drivers/xen/events.c | 7 +++-- 6 files changed, 71 insertions(+), 8 deletions(-)
Signature · 1 line
1.7.4.1
PATCH 1/3 x86: Hyper-V: register clocksource only if its advertised
From: Olaf Hering <[mailto:[email protected]]>
Enable hyperv_clocksource only if its advertised as a feature. XenServer 6 returns the signature which is checked in ms_hyperv_platform(), but it does not offer all features. Currently the clocksource is enabled unconditionally in ms_hyperv_init_platform(), and the result is a hanging guest.
Hyper-V spec Bit 1 indicates the availability of Partition Reference Counter. Register the clocksource only if this bit is set.
The guest in question prints this in dmesg:
[ 0.000000] Hypervisor detected: Microsoft HyperV [ 0.000000] HyperV: features 0x70, hints 0x0
This bug can be reproduced easily be setting 'viridian=1' in a HVM domU .cfg file. A workaround without this patch is to boot the HVM guest with 'clocksource=jiffies'.
--- arch/x86/kernel/cpu/mshyperv.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
arch/x86/kernel/cpu/mshyperv.c +2 -1
@@ -68,7 +68,8 @@ static void __init ms_hyperv_init_platform(void)printk (KERN_INFO "HyperV: features 0x%x, hints 0x%x\n" , ms_hyperv.features, ms_hyperv.hints); -clocksource_register_hz (&hyperv_cs,NSEC_PER_SEC /100 ); +if (ms_hyperv.features &HV_X64_MSR_TIME_REF_COUNT_AVAILABLE ) +clocksource_register_hz (&hyperv_cs,NSEC_PER_SEC /100 ); }const __refconst struct hypervisor_x86 x86_hyper_ms_hyperv = {
Signature · 1 line
1.7.4.1
PATCH 2/3 X86: Add a check to catch Xen emulation of Hyper-V
Xen emulates Hyper-V to host enlightened Windows. Looks like this emulation may be turned on by default even for Linux guests. Check and fail Hyper-V detection if we are on Xen.
--- arch/x86/kernel/cpu/mshyperv.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)
arch/x86/kernel/cpu/mshyperv.c +7 -0
@@ -30,6 +30,13 @@ static bool __init ms_hyperv_platform(void)if (!boot_cpu_has (X86_FEATURE_HYPERVISOR ))return false; +/* +* Xen emulates Hyper-V to support enlightened Windows. +* Check to see first if we are on a Xen Hypervisor. +*/ +if (xen_cpuid_base ()) +return false; +cpuid (HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS , &eax, &hyp_signature[0 ], &hyp_signature[1 ], &hyp_signature[2 ]);
Signature · 1 line
1.7.4.1
Re: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
Quoted 17 lines
Quoted 1 line
Quoted 1 line · K. Y. Srinivasan
Xen emulates Hyper-V to host enlightened Windows. Looks like this emulation may be turned on by default even for Linux guests. Check and fail Hyper-V detection if we are on Xen.
Signed-off-by K. Y. Srinivasan <[email protected]>--- arch/x86/kernel/cpu/mshyperv.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)arch/x86/kernel/cpu/mshyperv.c +0
-0@@ -30,6 +30,13 @@ static bool __init ms_hyperv_platform(void)if (!boot_cpu_has (X86_FEATURE_HYPERVISOR ))return false;
arch/x86/kernel/cpu/mshyperv.c static bool __init ms_hyperv_platform(void) · line 41
Quoted 10 lines
arch/x86/kernel/cpu/mshyperv.c +7
-0+/* +* Xen emulates Hyper-V to support enlightened Windows. +* Check to see first if we are on a Xen Hypervisor. +*/ +if (xen_cpuid_base ()) +return false; +cpuid (HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS , &eax, &hyp_signature[0 ], &hyp_signature[1 ], &hyp_signature[2 ]);
I'm not convinced that's the right approach - any hypervisor could do similar emulation, and hence you either want to make sure you run on Hyper-V (by excluding all others), or you tolerate using the emulation (which may require syncing up with the other guest implementations so that shared resources don't get used by two parties).
I also wonder whether using the Hyper-V emulation (where useful, there might not be anything right now, but this may change going forward) when no Xen support is configured wouldn't be better than not using anything...
Jan
RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
Quoted 47 lines
-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 3:53 AM
To: KY Srinivasan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-VReview
Quoted 11 lines
Quoted 1 line
Quoted 1 line · K. Y. Srinivasan
Xen emulates Hyper-V to host enlightened Windows. Looks like this emulation may be turned on by default even for Linux guests. Check and fail Hyper-V detection if we are on Xen.
Signed-off-by K. Y. Srinivasan <[email protected]>--- arch/x86/kernel/cpu/mshyperv.c | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-)(quoted diff) +0
-0
b/arch/x86/kernel/cpu/mshyperv.cQuoted 6 lines
index 646d192..4dab317 100644
arch/x86/kernel/cpu/mshyperv.c +0
-0@@ -30,6 +30,13 @@ static bool __init ms_hyperv_platform(void)if (!boot_cpu_has (X86_FEATURE_HYPERVISOR ))return false;arch/x86/kernel/cpu/mshyperv.c static bool __init ms_hyperv_platform(void) · line 41
Quoted 10 lines
arch/x86/kernel/cpu/mshyperv.c +7
-0+/* +* Xen emulates Hyper-V to support enlightened Windows. +* Check to see first if we are on a Xen Hypervisor. +*/ +if (xen_cpuid_base ()) +return false; +cpuid (HYPERV_CPUID_VENDOR_AND_MAX_FUNCTIONS , &eax, &hyp_signature[0 ], &hyp_signature[1 ], &hyp_signature[2 ]);I'm not convinced that's the right approach - any hypervisor could do similar emulation, and hence you either want to make sure you run on Hyper-V (by excluding all others), or you tolerate using the emulation (which may require syncing up with the other guest implementations so that shared resources don't get used by two parties).
Quoted 4 lines
I also wonder whether using the Hyper-V emulation (where useful, there might not be anything right now, but this may change going forward) when no Xen support is configured wouldn't be better than not using anything...
Jan,
Presumably, Hyper-V emulation is only to run enlightened Windows. The issue with Xen is not that it emulates Hyper-V, but this emulation is turned on while running Linux. That is the reason I chose to check for Xen. Would you prefer a DMI check for the Hyper-V platform.
Regards,
K. Y
H. Peter Anvin
reviewerRe: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
Quoted 13 lines · KY Srinivasan
Quoted 8 lines
I'm not convinced that's the right approach - any hypervisor could do similar emulation, and hence you either want to make sure you run on Hyper-V (by excluding all others), or you tolerate using the emulation (which may require syncing up with the other guest implementations so that shared resources don't get used by two parties).
Quoted 4 lines
I also wonder whether using the Hyper-V emulation (where useful, there might not be anything right now, but this may change going forward) when no Xen support is configured wouldn't be better than not using anything...
Quoted 7 lines
Jan,
Presumably, Hyper-V emulation is only to run enlightened Windows. The issue with Xen is not that it emulates Hyper-V, but this emulation is turned on while running Linux. That is the reason I chose to check for Xen. Would you prefer a DMI check for the Hyper-V platform.
The real issue here is anyone going to run a kernel:
1. which has Xen support *disabled*,
2. which has HyperV support *enabled*,
3. on top of Xen,
4. with Xen HyperV support enabled.
The fundamental problem seems to be that Xen HyperV support is "just good enough to run Windows", but in fact has some serious shortcomings which breaks Linux guests that don't have Xen support (if they do, the Xen support kicks in instead.)
Am I right?
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
In reply to KY SrinivasanQuoted 8 lines
Quoted 1 line
Quoted 1 line · KY Srinivasan
Presumably, Hyper-V emulation is only to run enlightened Windows. The issue with
Xen is not that it emulates Hyper-V, but this emulation is turned on while running Linux.
That is the reason I chose to check for Xen. Would you prefer a DMI check for the Hyper-V
platform.
I consider DMI checks to be too fragile here - in particular with the eventual passing through of host DMI attributes to guests, this sets you up for mistakes. Instead, I would envision you scanning the whole CPUID range "reserved" for virtualization (starting at 0x40000000) and see whether you get back anything other than the Hyper-V identification (much like the way xen_cpuid_base() scans for the Xen range). Of course that's under the premise that you're right in assuming Hyper-V would never emulate any other hypervisor's interface.
Jan
RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
Quoted 20 lines
-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Thursday, January 31, 2013 2:38 AM
To: KY Srinivasan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-VQuoted 8 lines
Quoted 1 line
Quoted 1 line · KY Srinivasan
Presumably, Hyper-V emulation is only to run enlightened Windows. The issue with
Xen is not that it emulates Hyper-V, but this emulation is turned on while running Linux.
That is the reason I chose to check for Xen. Would you prefer a DMI check for the Hyper-V
platform.
Quoted 9 lines
I consider DMI checks to be too fragile here - in particular with the eventual passing through of host DMI attributes to guests, this sets you up for mistakes. Instead, I would envision you scanning the whole CPUID range "reserved" for virtualization (starting at
0x40000000) and see whether you get back anything other than the Hyper-V identification (much like the wayxen_cpuid_base()scans for the Xen range). Of course that's under the premise that you're right in assuming Hyper-V would never emulate any other hypervisor's interface.
Agreed; I will make the appropriate changes as you have recommended.
Regards,
K. Y
RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
Quoted 44 lines
-----Original Message-----
From: devel [mailto:[email protected]] On Behalf Of KY Srinivasan
Sent: Thursday, January 31, 2013 9:46 AM
To: Jan Beulich
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-VQuoted 6 lines
-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Thursday, January 31, 2013 2:38 AM
To: KY Srinivasan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected];Quoted 5 lines
[email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-VQuoted 2 lines
Quoted 1 line
Quoted 1 line · KY Srinivasan
Presumably, Hyper-V emulation is only to run enlightened Windows. The
issue
Quoted 7 lines
Quoted 6 lines
with
Xen is not that it emulates Hyper-V, but this emulation is turned on while running Linux.
That is the reason I chose to check for Xen. Would you prefer a DMI check for the Hyper-V
platform.Quoted 9 lines
I consider DMI checks to be too fragile here - in particular with the eventual passing through of host DMI attributes to guests, this sets you up for mistakes. Instead, I would envision you scanning the whole CPUID range "reserved" for virtualization (starting at
0x40000000) and see whether you get back anything other than the Hyper-V identification (much like the wayxen_cpuid_base()scans for the Xen range). Of course that's under the premise that you're right in assuming Hyper-V would never emulate any other hypervisor's interface.
Quoted 1 line
Agreed; I will make the appropriate changes as you have recommended.
Jan,
Are there any published standards in terms of how the CPUID space should be populated in the range from 0x40000000 to 0x40010000. Specifically, unless the standard mandates that all ranges unused by a given hypervisor would return a known value, how can this code be used to detect the presence of an unknown hypervisor. Hyper-V is going to return the Hyper-V string at 0x40000000. So, I was planning to scan starting at 0x40000100. Clearly, I can check for a specific hypervisor that I know causes a problem for Hyper-V (as I have currently done by checking for Xen). How can I check for the presence of yet to be created Hypervisors that may emulate Hyper-V by scanning the CPUID space. I am almost tempted to say that Xen is the special case and the patch I have submitted addresses that. If a new (or existing hypervisor) plans to do what Xen is doing, perhaps we can dissuade them from doing that or we can fix that within the general framework we have here.
Regards,
K. Y
Quoted 5 lines
_______________________________________________
devel mailing list
[email protected]
http://driverdev.linuxdriverproject.org/mailman/listinfo/devel
RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
Quoted 3 lines
Quoted 1 line
Quoted 1 line · KY Srinivasan
Are there any published standards in terms of how the CPUID space should be populated in the range from
0x40000000to0x40010000. Specifically, unless
I recall having seen this range being marked as reserved for hypervisor use somewhere, but I don't remember where that was.
Quoted 11 lines
the standard mandates that all ranges unused by a given hypervisor would return a known value, how can this code be used to detect the presence of an unknown hypervisor. Hyper-V is going to return the Hyper-V string at
0x40000000. So, I was planning to scan starting at0x40000100. Clearly, I can check for a specific hypervisor that I know causes a problem for Hyper-V (as I have currently done by checking for Xen). How can I check for the presence of yet to be created Hypervisors that may emulate Hyper-V by scanning the CPUID space. I am almost tempted to say that Xen is the special case and the patch I have submitted addresses that. If a new (or existing hypervisor) plans to do what Xen is doing, perhaps we can dissuade them from doing that or we can fix that within the general framework we have here.
I'd simply preset ECX, EDX, and EBX to zero, and check whether they change on any EAX input divisible by 256, skipping the one range you know Hyper-V sits in.
Jan
H. Peter Anvin
reviewerRe: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
Quoted 7 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line · KY Srinivasan
Are there any published standards in terms of how the CPUID space should be populated in the range from
0x40000000to0x40010000. Specifically, unlessI recall having seen this range being marked as reserved for hypervisor use somewhere, but I don't remember where that was.
AFAIK Intel reserves 0x40000000 to 0x4FFFFFFF for hypervisor use. Typically multivendor CPUID utilities expect "blocks" of 64K but that is not in any spec that I know of.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
H. Peter Anvin
reviewerRe: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
In reply to Jan BeulichQuoted 8 lines · Jan Beulich
I'm not convinced that's the right approach - any hypervisor could do similar emulation, and hence you either want to make sure you run on Hyper-V (by excluding all others), or you tolerate using the emulation (which may require syncing up with the other guest implementations so that shared resources don't get used by two parties).
Quoted 5 lines
I also wonder whether using the Hyper-V emulation (where useful, there might not be anything right now, but this may change going forward) when no Xen support is configured wouldn't be better than not using anything...
I'm confused about "the right approach" here is. As far as I understand, this only can affect a Xen guest where HyperV guest support is enabled but not Xen support, and only because Xen emulates HyperV but does so incorrectly.
This is a Xen bug, and as such it makes sense to reject Xen specifically. If another hypervisor emulates HyperV and does so correctly there seems to be no reason to reject it.
-hpa
Re: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
Quoted 15 lines · H. Peter Anvin
Quoted 8 lines · Jan Beulich
I'm not convinced that's the right approach - any hypervisor could do similar emulation, and hence you either want to make sure you run on Hyper-V (by excluding all others), or you tolerate using the emulation (which may require syncing up with the other guest implementations so that shared resources don't get used by two parties).
Quoted 5 lines
I also wonder whether using the Hyper-V emulation (where useful, there might not be anything right now, but this may change going forward) when no Xen support is configured wouldn't be better than not using anything...
Quoted 8 lines
I'm confused about "the right approach" here is. As far as I understand, this only can affect a Xen guest where HyperV guest support is enabled but not Xen support, and only because Xen emulates HyperV but does so incorrectly.
This is a Xen bug, and as such it makes sense to reject Xen specifically. If another hypervisor emulates HyperV and does so correctly there seems to be no reason to reject it.
I don't think so.
AFAIK originally there were features exported as flags and Xen doesn't turn on the flags that correspond to features that are not implemented. The problem here is that Hyper-V is about to introduce a feature without a flag that is not implemented by Xen (see "X86: Deliver Hyper-V interrupts on a separate IDT vector").
K.Y. please confirm if I got this right.
If I were the Microsoft engineer implementing this feature, no matter what Xen does or does not, I would also make sure that there is a corresponding flag for it, because in my experience they avoid future headaches.
I wonder what happens if you run Linux with Hyper-V support on an old Hyper-V host that doesn't support vector injection.
RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
Quoted 37 lines
-----Original Message-----
From: Stefano Stabellini [mailto:[email protected]]
Sent: Friday, February 01, 2013 8:20 AM
To: H. Peter Anvin
Cc: Jan Beulich; KY Srinivasan; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-VQuoted 15 lines · H. Peter Anvin
Quoted 8 lines · Jan Beulich
I'm not convinced that's the right approach - any hypervisor could do similar emulation, and hence you either want to make sure you run on Hyper-V (by excluding all others), or you tolerate using the emulation (which may require syncing up with the other guest implementations so that shared resources don't get used by two parties).
Quoted 5 lines
I also wonder whether using the Hyper-V emulation (where useful, there might not be anything right now, but this may change going forward) when no Xen support is configured wouldn't be better than not using anything...
Quoted 8 lines
I'm confused about "the right approach" here is. As far as I understand, this only can affect a Xen guest where HyperV guest support is enabled but not Xen support, and only because Xen emulates HyperV but does so incorrectly.
This is a Xen bug, and as such it makes sense to reject Xen specifically. If another hypervisor emulates HyperV and does so correctly there seems to be no reason to reject it.
Quoted 8 lines
I don't think so.
AFAIK originally there were features exported as flags and Xen doesn't turn on the flags that correspond to features that are not implemented. The problem here is that Hyper-V is about to introduce a feature without a flag that is not implemented by Xen (see "X86: Deliver Hyper-V interrupts on a separate IDT vector").
K.Y. please confirm if I got this right.
I am not sure I can agree with you here. There are two discriminating factors
here: (a) Hypervisor check and (b) Feature check. Not every feature of the hypervisor can be surfaced as feature bit and furthermore, just because a feature bit is turned on, it does not necessarily mean that the feature is to be used. For instance, let us say that Windows guests begin to use the "partition counter" and Xen chooses to implement that to better support Windows. This does not mean that while hosting Linux on Xen, you want to plug in a clock source based on the emulated "partition counter". This is what would happen in the code we have today.
Other Hypervisors emulating Hyper-V do not have this problem and Xen would not either
if the emulation bit is selectively turned on (only while running Windows) or if Xen were allowed to check first ahead of Hyper-V (unconditionally) in the hypervisor detection code. As Peter pointed out, we have this problem because of the unique situation with Xen.
In any event, I am not going to further argue this issue; this last round of patches I sent out, fixes the issue for Xen. Jan wants me to make this check more general. While I don't think we need to do that, I will see if I can do it. I am checking to see if MSFT guarantees that Hyper-V would initialize the unused CPUID space to 0. If this is the case, I will implement the check Jan has suggested; if not, we have to live with the Xen specific check that I currently have.
Quoted 8 lines
If I were the Microsoft engineer implementing this feature, no matter what Xen does or does not, I would also make sure that there is a corresponding flag for it, because in my experience they avoid future headaches.
I wonder what happens if you run Linux with Hyper-V support on an old Hyper-V host that doesn't support vector injection.
To answer your specific question, this feature of being able to distribute vmbus interrupt load across all VCPUs in the guest is a win8 and beyond feature. On prior hosts, all interrupts will be delivered on the boot CPU. VMBUS, as part of connecting with the hosts determines host supported protocol version and decides how it wants to program the hypervisor with regards to interrupt delivery. Even though we might setup an IDT entry for delivering the hypervisor interrupt, if the host is a pre-win8 host, the vmbus driver will program the hypervisor to deliver the interrupt on the boot CPU via a legacy interrupt vector.
Regards,
K. Y
RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-V
Quoted 79 lines
-----Original Message-----
From: devel [mailto:[email protected]] On Behalf Of KY Srinivasan
Sent: Friday, February 01, 2013 10:11 AM
To: Stefano Stabellini; H. Peter Anvin
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Jan Beulich; [email protected]; [email protected]; [email protected]
Subject: RE: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-VQuoted 35 lines
-----Original Message-----
From: Stefano Stabellini [mailto:[email protected]]
Sent: Friday, February 01, 2013 8:20 AM
To: H. Peter Anvin
Cc: Jan Beulich; KY Srinivasan; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 2/3] X86: Add a check to catch Xen emulation of Hyper-VQuoted 15 lines · H. Peter Anvin
Quoted 8 lines · Jan Beulich
I'm not convinced that's the right approach - any hypervisor could do similar emulation, and hence you either want to make sure you run on Hyper-V (by excluding all others), or you tolerate using the emulation (which may require syncing up with the other guest implementations so that shared resources don't get used by two parties).
Quoted 5 lines
I also wonder whether using the Hyper-V emulation (where useful, there might not be anything right now, but this may change going forward) when no Xen support is configured wouldn't be better than not using anything...
Quoted 8 lines
I'm confused about "the right approach" here is. As far as I understand, this only can affect a Xen guest where HyperV guest support is enabled but not Xen support, and only because Xen emulates HyperV but does so incorrectly.
This is a Xen bug, and as such it makes sense to reject Xen specifically. If another hypervisor emulates HyperV and does so correctly there seems to be no reason to reject it.
Quoted 8 lines
I don't think so.
AFAIK originally there were features exported as flags and Xen doesn't turn on the flags that correspond to features that are not implemented. The problem here is that Hyper-V is about to introduce a feature without a flag that is not implemented by Xen (see "X86: Deliver Hyper-V interrupts on a separate IDT vector").
K.Y. please confirm if I got this right.I am not sure I can agree with you here. There are two discriminating factors
here: (a) Hypervisor check and (b) Feature check. Not every feature of the hypervisor can be surfaced as feature bit and furthermore, just because a feature
bit is turned on, it does not necessarily mean that the feature is to be used. For instance,
let us say that Windows guests begin to use the "partition counter" and Xen chooses
to implement that to better support Windows. This does not mean that while hosting
Linux on Xen, you want to plug in a clock source based on the emulated "partition counter". This is what would happen in the code we have today.Other Hypervisors emulating Hyper-V do not have this problem and Xen would not either
if the emulation bit is selectively turned on (only while running Windows) or if Xen were allowed
to check first ahead of Hyper-V (unconditionally) in the hypervisor detection code. As Peter pointed out, we
have this problem because of the unique situation with Xen.
Quoted 10 lines
In any event, I am not going to further argue this issue; this last round of patches I sent out,
fixes the issue for Xen. Jan wants me to make this check more general. While I don't think
we need to do that, I will see if I can do it. I am checking to see if MSFT guarantees that Hyper-V
would initialize the unused CPUID space to 0. If this is the case, I will implement the check
Jan has suggested; if not, we have to live with the Xen specific check that I currently have.
I checked with the Hyper-V guys and I am told that there is no guarantee that Hyper-V would not use some other range in the CPUID space in the future. So, I will keep the Xen specific check that I had in this version. I will add the appropriate compilation switches to take care of the "dead code" and resubmit the patches.
Regards,
K. Y
Quoted 33 lines
Quoted 9 lines
If I were the Microsoft engineer implementing this feature, no matter what Xen does or does not, I would also make sure that there is a corresponding flag for it, because in my experience they avoid future headaches.
I wonder what happens if you run Linux with Hyper-V support on an old Hyper-V host that doesn't support vector injection.To answer your specific question, this feature of being able to distribute vmbus interrupt load across all VCPUs in the guest is a win8 and beyond feature. On prior hosts, all interrupts will be delivered on the boot CPU. VMBUS, as part of connecting with
the hosts determines host supported protocol version and decides how it wants to
program the hypervisor with regards to interrupt delivery. Even though we might setup
an IDT entry for delivering the hypervisor interrupt, if the host is a pre-win8 host, the vmbus
driver will program the hypervisor to deliver the interrupt on the boot CPU via a legacy interrupt
vector.Regards,
K. Y
_______________________________________________
devel mailing list
[email protected]
http://driverdev.linuxdriverproject.org/mailman/listinfo/devel
PATCH 3/3 X86: Handle Hyper-V vmbus interrupts as special hypervisor interrupts
In reply to K. Y. SrinivasanStarting with win8, vmbus interrupts can be delivered on any VCPU in the guest and furthermore can be concurrently active on multiple VCPUs. Support this interrupt delivery model by setting up a separate IDT entry for Hyper-V vmbus. interrupts. I would like to thank Jan Beulich <[email protected]> and Thomas Gleixner <[email protected]>, for their help.
In this version of the patch, based on the feedback, I have merged the IDT vector for Xen and Hyper-V and made the necessary adjustments.
--- arch/x86/include/asm/irq_vectors.h | 4 +- arch/x86/include/asm/mshyperv.h | 4 +++ arch/x86/kernel/cpu/mshyperv.c | 38 ++++++++++++++++++++++++++++++++++++ arch/x86/kernel/entry_32.S | 9 +++++++- arch/x86/kernel/entry_64.S | 7 +++++- drivers/xen/events.c | 7 +++-- 6 files changed, 62 insertions(+), 7 deletions(-)
arch/x86/include/asm/irq_vectors.h +2 -2
@@ -109,8 +109,8 @@#define UV_BAU_MESSAGE 0xf5 -/* Xen vector callback to receive events in a HVM domain */ -#define XEN_HVM_EVTCHN_CALLBACK 0xf3 +/* Vector on which hypervisor callbacks will be delivered */ +#define HYPERVISOR_CALLBACK_VECTOR 0xf3/* * Local APIC timer IRQ vector is on a different priority level,
arch/x86/include/asm/mshyperv.h +4 -0
@@ -11,4 +11,8 @@ struct ms_hyperv_info {extern struct ms_hyperv_info ms_hyperv; +void hyperv_callback_vector (void ); +void hyperv_vector_handler (struct pt_regs *regs); +void hv_register_vmbus_handler (int irq,irq_handler_t handler); +#endif
arch/x86/kernel/cpu/mshyperv.c +38 -0
@@ -14,10 +14,15 @@#include <linux/time.h> #include <linux/clocksource.h> #include <linux/module.h> +#include <linux/hardirq.h> +#include <linux/interrupt.h> #include <asm/processor.h> #include <asm/hypervisor.h> #include <asm/hyperv.h> #include <asm/mshyperv.h> +#include <asm/desc.h> +#include <asm/idle.h> +#include <asm/irq_regs.h> struct ms_hyperv_info ms_hyperv;EXPORT_SYMBOL_GPL (ms_hyperv);
@@ -77,6 +82,12 @@ static void __init ms_hyperv_init_platform(void)if (ms_hyperv.features &HV_X64_MSR_TIME_REF_COUNT_AVAILABLE )clocksource_register_hz (&hyperv_cs,NSEC_PER_SEC /100 ); +#if IS_ENABLED (CONFIG_HYPERV ) +/* +* Setup the IDT for hypervisor callback. +*/ +alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , hyperv_callback_vector); +#endif }const __refconst struct hypervisor_x86 x86_hyper_ms_hyperv = {
@@ -85,3 +96,30 @@ const __refconst struct hypervisor_x86 x86_hyper_ms_hyperv = { .init_platform = ms_hyperv_init_platform, };EXPORT_SYMBOL (x86_hyper_ms_hyperv); + +static int vmbus_irq =-1 ; +static irq_handler_t vmbus_isr; + +void hv_register_vmbus_handler (int irq,irq_handler_t handler) +{ + vmbus_irq = irq; + vmbus_isr = handler; +} +EXPORT_SYMBOL_GPL (hv_register_vmbus_handler); + +void hyperv_vector_handler (struct pt_regs *regs) +{ +struct pt_regs *old_regs =set_irq_regs (regs); +struct irq_desc *desc; + +irq_enter (); +exit_idle (); + + desc =irq_to_desc (vmbus_irq); + +if (desc) +generic_handle_irq_desc (vmbus_irq, desc); + +irq_exit (); +set_irq_regs (old_regs); +}
arch/x86/kernel/entry_32.S +8 -1
@@ -1046,11 +1046,18 @@ ENTRY(xen_failsafe_callback) _ASM_EXTABLE(4b,9b) ENDPROC(xen_failsafe_callback) -BUILD_INTERRUPT3(xen_hvm_callback_vector, XEN_HVM_EVTCHN_CALLBACK, +BUILD_INTERRUPT3(xen_hvm_callback_vector, HYPERVISOR_CALLBACK_VECTOR, xen_evtchn_do_upcall) #endif /* CONFIG_XEN */ +#if IS_ENABLED(CONFIG_HYPERV) + +BUILD_INTERRUPT3(hyperv_callback_vector, HYPERVISOR_CALLBACK_VECTOR, + hyperv_vector_handler) + +#endif /* CONFIG_HYPERV */ + #ifdef CONFIG_FUNCTION_TRACER #ifdef CONFIG_DYNAMIC_FTRACE
arch/x86/kernel/entry_64.S +6 -1
@@ -1446,11 +1446,16 @@ ENTRY(xen_failsafe_callback) CFI_ENDPROC END(xen_failsafe_callback) -apicinterrupt XEN_HVM_EVTCHN_CALLBACK \ +apicinterrupt HYPERVISOR_CALLBACK_VECTOR \ xen_hvm_callback_vector xen_evtchn_do_upcall #endif /* CONFIG_XEN */ +#if IS_ENABLED(CONFIG_HYPERV) +apicinterrupt HYPERVISOR_CALLBACK_VECTOR \ + hyperv_callback_vector hyperv_vector_handler +#endif /* CONFIG_HYPERV */ + /* * Some functions should be protected against kprobes */
drivers/xen/events.c +4 -3
@@ -1787,7 +1787,7 @@ void xen_callback_vector(void)int rc;uint64_t callback_via;if (xen_have_vector_callback) { - callback_via =HVM_CALLBACK_VECTOR (XEN_HVM_EVTCHN_CALLBACK ); + callback_via =HVM_CALLBACK_VECTOR (HYPERVISOR_CALLBACK_VECTOR ); rc =xen_set_callback_via (callback_via);if (rc) { printk(KERN_ERR "Request for Xen HVM callback vector"
@@ -1798,8 +1798,9 @@ void xen_callback_vector(void)printk (KERN_INFO "Xen HVM callback vector for event delivery is " "enabled\n" );/* in the restore case the vector has already been allocated */ -if (!test_bit (XEN_HVM_EVTCHN_CALLBACK , used_vectors)) - alloc_intr_gate(XEN_HVM_EVTCHN_CALLBACK, xen_hvm_callback_vector); +if (!test_bit (HYPERVISOR_CALLBACK_VECTOR , used_vectors)) +alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , + xen_hvm_callback_vector); } }#else
Signature · 1 line
1.7.4.1
Re: [PATCH 3/3] X86: Handle Hyper-V vmbus interrupts as special hypervisor interrupts
Quoted 53 lines
Quoted 1 line
Quoted 1 line · K. Y. Srinivasan
arch/x86/kernel/cpu/mshyperv.c +28
-0@@ -14,10 +14,15 @@#include <linux/time.h> #include <linux/clocksource.h> #include <linux/module.h> +#include <linux/hardirq.h> +#include <linux/interrupt.h> #include <asm/processor.h> #include <asm/hypervisor.h> #include <asm/hyperv.h> #include <asm/mshyperv.h> +#include <asm/desc.h> +#include <asm/idle.h> +#include <asm/irq_regs.h> struct ms_hyperv_info ms_hyperv;EXPORT_SYMBOL_GPL (ms_hyperv);@@ -77,6 +82,12 @@ static void __init ms_hyperv_init_platform(void)if (ms_hyperv.features &HV_X64_MSR_TIME_REF_COUNT_AVAILABLE )clocksource_register_hz (&hyperv_cs,NSEC_PER_SEC /100 ); +#if IS_ENABLED (CONFIG_HYPERV ) +/* +* Setup the IDT for hypervisor callback. +*/ +alloc_intr_gate (HYPERVISOR_CALLBACK_VECTOR , hyperv_callback_vector); +#endif }const __refconst struct hypervisor_x86 x86_hyper_ms_hyperv = {@@ -85,3 +96,30 @@ const __refconst struct hypervisor_x86 x86_hyper_ms_hyperv = { .init_platform = ms_hyperv_init_platform, };EXPORT_SYMBOL (x86_hyper_ms_hyperv); + +static int vmbus_irq =-1 ; +static irq_handler_t vmbus_isr; + +void hv_register_vmbus_handler (int irq,irq_handler_t handler) +{ + vmbus_irq = irq; + vmbus_isr = handler; +} +EXPORT_SYMBOL_GPL (hv_register_vmbus_handler); + +void hyperv_vector_handler (struct pt_regs *regs) +{ +struct pt_regs *old_regs =set_irq_regs (regs); +struct irq_desc *desc; + +irq_enter ();
arch/x86/kernel/cpu/mshyperv.c const __refconst struct hypervisor_x86 x86_hyper_ms_hyperv = { · line 125
Quoted 10 lines
arch/x86/kernel/cpu/mshyperv.c +10
-0+exit_idle (); + + desc =irq_to_desc (vmbus_irq); + +if (desc) +generic_handle_irq_desc (vmbus_irq, desc); + +irq_exit (); +set_irq_regs (old_regs); +}
This function appears to be dead code when !CONFIG_HYPERV, because ...
Quoted 12 lines
arch/x86/kernel/entry_32.S +1
-1@@ -1046,11 +1046,18 @@ ENTRY(xen_failsafe_callback) _ASM_EXTABLE(4b,9b) ENDPROC(xen_failsafe_callback) -BUILD_INTERRUPT3(xen_hvm_callback_vector, XEN_HVM_EVTCHN_CALLBACK, +BUILD_INTERRUPT3(xen_hvm_callback_vector, HYPERVISOR_CALLBACK_VECTOR, xen_evtchn_do_upcall) #endif /* CONFIG_XEN */
arch/x86/kernel/entry_32.S ENTRY(xen_failsafe_callback) · line 1063
Quoted 10 lines
arch/x86/kernel/entry_32.S +7
-0+#if IS_ENABLED(CONFIG_HYPERV) + +BUILD_INTERRUPT3(hyperv_callback_vector, HYPERVISOR_CALLBACK_VECTOR, + hyperv_vector_handler) + +#endif /* CONFIG_HYPERV */ + #ifdef CONFIG_FUNCTION_TRACER #ifdef CONFIG_DYNAMIC_FTRACE
... the consumers here and below are conditional.
I also wonder why arch/x86/kernel/cpu/mshyperv.c is being built at all when !CONFIG_HYPERV (which would eliminate the need for the conditional the patch adds to ms_hyperv_init_platform()).
Jan
Quoted 22 lines
arch/x86/kernel/entry_64.S +6
-1@@ -1446,11 +1446,16 @@ ENTRY(xen_failsafe_callback) CFI_ENDPROC END(xen_failsafe_callback) -apicinterrupt XEN_HVM_EVTCHN_CALLBACK \ +apicinterrupt HYPERVISOR_CALLBACK_VECTOR \ xen_hvm_callback_vector xen_evtchn_do_upcall #endif /* CONFIG_XEN */ +#if IS_ENABLED(CONFIG_HYPERV) +apicinterrupt HYPERVISOR_CALLBACK_VECTOR \ + hyperv_callback_vector hyperv_vector_handler +#endif /* CONFIG_HYPERV */ + /* * Some functions should be protected against kprobes */
RE: [PATCH 3/3] X86: Handle Hyper-V vmbus interrupts as special hypervisor interrupts
Quoted 72 lines
-----Original Message-----
From: Jan Beulich [mailto:[email protected]]
Sent: Wednesday, January 30, 2013 3:59 AM
To: KY Srinivasan
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH 3/3] X86: Handle Hyper-V vmbus interrupts as special hypervisor interruptsQuoted 20 lines
Quoted 1 line
Quoted 1 line · K. Y. Srinivasan
arch/x86/kernel/cpu/mshyperv.c +5
-0@@ -14,10 +14,15 @@#include <linux/time.h> #include <linux/clocksource.h> #include <linux/module.h> +#include <linux/hardirq.h> +#include <linux/interrupt.h> #include <asm/processor.h> #include <asm/hypervisor.h> #include <asm/hyperv.h> #include <asm/mshyperv.h> +#include <asm/desc.h> +#include <asm/idle.h> +#include <asm/irq_regs.h> struct ms_hyperv_info ms_hyperv;EXPORT_SYMBOL_GPL (ms_hyperv);arch/x86/kernel/cpu/mshyperv.c static void __init ms_hyperv_init_platform(void) · line 89
Quoted 8 lines
arch/x86/kernel/cpu/mshyperv.c +5
-0@@ -77,6 +82,12 @@ static void __init ms_hyperv_init_platform(void)if (ms_hyperv.features &HV_X64_MSR_TIME_REF_COUNT_AVAILABLE )clocksource_register_hz (&hyperv_cs,NSEC_PER_SEC /100 ); +#if IS_ENABLED (CONFIG_HYPERV ) +/* +* Setup the IDT for hypervisor callback. +*/ + alloc_intr_gate(HYPERVISOR_CALLBACK_VECTOR,
hyperv_callback_vector);arch/x86/kernel/cpu/mshyperv.c const __refconst struct hypervisor_x86
Quoted 5 lines
arch/x86/kernel/cpu/mshyperv.c +1
-0+#endif }const __refconst struct hypervisor_x86 x86_hyper_ms_hyperv = {@@ -85,3 +96,30 @@ const __refconst struct hypervisor_x86
x86_hyper_ms_hyperv= {Quoted 25 lines
arch/x86/kernel/cpu/mshyperv.c +22
-0.init_platform = ms_hyperv_init_platform, };EXPORT_SYMBOL (x86_hyper_ms_hyperv); + +static int vmbus_irq =-1 ; +static irq_handler_t vmbus_isr; + +void hv_register_vmbus_handler (int irq,irq_handler_t handler) +{ + vmbus_irq = irq; + vmbus_isr = handler; +} +EXPORT_SYMBOL_GPL (hv_register_vmbus_handler); + +void hyperv_vector_handler (struct pt_regs *regs) +{ +struct pt_regs *old_regs =set_irq_regs (regs); +struct irq_desc *desc; + +irq_enter (); +exit_idle (); + + desc =irq_to_desc (vmbus_irq); + +if (desc)
Quoted 8 lines
arch/x86/kernel/cpu/mshyperv.c const __refconst struct hypervisor_x86 x86_hyper_ms_hyperv = { · line 125
Quoted 5 lines
arch/x86/kernel/cpu/mshyperv.c +5
-0+generic_handle_irq_desc (vmbus_irq, desc); + +irq_exit (); +set_irq_regs (old_regs); +}This function appears to be dead code when !CONFIG_HYPERV, because ...
I will make the necessary adjustments to deal with this.
Quoted 29 lines
arch/x86/kernel/entry_32.S ENTRY(xen_failsafe_callback) · line 1049
Quoted 8 lines
arch/x86/kernel/entry_32.S +0
-1@@ -1046,11 +1046,18 @@ ENTRY(xen_failsafe_callback) _ASM_EXTABLE(4b,9b) ENDPROC(xen_failsafe_callback) -BUILD_INTERRUPT3(xen_hvm_callback_vector,
XEN_HVM_EVTCHN_CALLBACK,arch/x86/kernel/entry_32.S ENTRY(xen_failsafe_callback) · line 1049
Quoted 1 line
arch/x86/kernel/entry_32.S +1
-0+BUILD_INTERRUPT3(xen_hvm_callback_vector,
HYPERVISOR_CALLBACK_VECTOR,arch/x86/kernel/entry_32.S ENTRY(xen_failsafe_callback)
Quoted 7 lines
arch/x86/kernel/entry_32.S +3
-0xen_evtchn_do_upcall) #endif /* CONFIG_XEN */ +#if IS_ENABLED(CONFIG_HYPERV) + +BUILD_INTERRUPT3(hyperv_callback_vector,
HYPERVISOR_CALLBACK_VECTOR,arch/x86/kernel/entry_32.S ENTRY(xen_failsafe_callback) · line 1063
Quoted 7 lines
arch/x86/kernel/entry_32.S +4
-0+ hyperv_vector_handler) + +#endif /* CONFIG_HYPERV */ + #ifdef CONFIG_FUNCTION_TRACER #ifdef CONFIG_DYNAMIC_FTRACE... the consumers here and below are conditional.
Quoted 3 lines
I also wonder why
arch/x86/kernel/cpu/mshyperv.cis being built at all when !CONFIG_HYPERV (which would eliminate the need for the conditional the patch adds toms_hyperv_init_platform()).
Linux is usable on Hyper-V in full emulation mode with the Hyper-V specific clocksource plugged in. The current code permits that.
K. Y