Skip to contentSkip to conversation

Port Xen to Hyper-V

RFC PATCH Superseded Wei Liu 4 participants 40 messages Sep 23, 2019 – Sep 27, 2019

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Wei Liu

author
#0 lore raw

cover Port Xen to Hyper-V

Hi all

In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network.

This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon.

There are two major areas to be worked on:
* Make Dom0 able to use Hyper-V's synthetic devices.
* Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces.

They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready.

Questions and comments are welcome.

Thanks,
Wei.

Cc Jan Beulich <[email protected]>
Cc Andrew Cooper <[email protected]>
Cc Wei Liu <[email protected]>
Cc Roger Pau Monné <[email protected]>

Wei Liu (8):
x86: introduce CONFIG_GUEST and move code
x86: include asm_defns.h directly in hypercall.h
x86: drop hypervisor_cpuid_base
x86: factor out xen variants for hypervisor setup code
x86: factor out hypervisor agnostic code
x86: make probe_xen return boolean value
x86: introduce CONFIG_HYPERV and hyperv directory
x86: be more verbose when running nested

 xen/arch/x86/Kconfig                          |  13 ++
 xen/arch/x86/Makefile                         |   2 +-
 xen/arch/x86/guest/Makefile                   |   6 +-
 xen/arch/x86/guest/hyperv/Makefile            |   1 +
 xen/arch/x86/guest/hyperv/hyperv.c            |  63 ++++++++++
 xen/arch/x86/guest/hypervisor.c               | 116 ++++++++++++++++++
 xen/arch/x86/guest/xen/Makefile               |   4 +
 xen/arch/x86/guest/{ => xen}/hypercall_page.S |   0
 xen/arch/x86/guest/{ => xen}/pvh-boot.c       |   0
 xen/arch/x86/guest/{ => xen}/xen.c            |  73 ++---------
 xen/arch/x86/pv/shim.c                        |   2 +-
 xen/arch/x86/setup.c                          |  11 ++
 xen/include/asm-x86/guest.h                   |   2 +
 xen/include/asm-x86/guest/hypercall.h         |   2 +
 xen/include/asm-x86/guest/hyperv.h            |  48 ++++++++
 xen/include/asm-x86/guest/hypervisor.h        |  58 +++++++++
 xen/include/asm-x86/guest/xen.h               |  23 +---
 17 files changed, 337 insertions(+), 87 deletions(-)
 create mode 100644 xen/arch/x86/guest/hyperv/Makefile
 create mode 100644 xen/arch/x86/guest/hyperv/hyperv.c
 create mode 100644 xen/arch/x86/guest/hypervisor.c
 create mode 100644 xen/arch/x86/guest/xen/Makefile
 rename xen/arch/x86/guest/{ => xen}/hypercall_page.S (100%)
 rename xen/arch/x86/guest/{ => xen}/pvh-boot.c (100%)
 rename xen/arch/x86/guest/{ => xen}/xen.c (80%)
 create mode 100644 xen/include/asm-x86/guest/hyperv.h
 create mode 100644 xen/include/asm-x86/guest/hypervisor.h
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#1 lore raw

RFC PATCH 1/8 x86: introduce CONFIG_GUEST and move code

Xen is able to run as a guest on Xen. We plan to make it able to run on Hyper-V as well.

Introduce CONFIG_GUEST which is set to true if either running on Xen or Hyper-V is desired. Restructure code hierarchy for new code to come.

No functional change intended.

---
 xen/arch/x86/Kconfig                          | 4 ++++
 xen/arch/x86/Makefile                         | 2 +-
 xen/arch/x86/guest/Makefile                   | 5 +----
 xen/arch/x86/guest/xen/Makefile               | 4 ++++
 xen/arch/x86/guest/{ => xen}/hypercall_page.S | 0
 xen/arch/x86/guest/{ => xen}/pvh-boot.c       | 0
 xen/arch/x86/guest/{ => xen}/xen.c            | 0
 7 files changed, 10 insertions(+), 5 deletions(-)
 create mode 100644 xen/arch/x86/guest/xen/Makefile
 rename xen/arch/x86/guest/{ => xen}/hypercall_page.S (100%)
 rename xen/arch/x86/guest/{ => xen}/pvh-boot.c (100%)
 rename xen/arch/x86/guest/{ => xen}/xen.c (100%)
xen/arch/x86/Kconfig +4 -0
index 288dc6c042..584bdc1bb8 100644
@@ -160,8 +160,12 @@ config XEN_ALIGN_2M
 
 endchoice
 
+config GUEST
+	bool
+
 config XEN_GUEST
 	def_bool n
+	select GUEST
 	prompt "Xen Guest"
 	---help---
 	  Support for Xen detecting when it is running under Xen.
xen/arch/x86/Makefile +1 -1
index 2443fd2cc5..99a12d0090 100644
@@ -1,7 +1,7 @@
 subdir-y += acpi
 subdir-y += cpu
 subdir-y += genapic
-subdir-$(CONFIG_XEN_GUEST) += guest
+subdir-$(CONFIG_GUEST) += guest
 subdir-$(CONFIG_HVM) += hvm
 subdir-y += mm
 subdir-$(CONFIG_XENOPROF) += oprofile
xen/arch/x86/guest/Makefile +1 -4
index 26fb4b1007..6806f04947 100644
@@ -1,4 +1 @@
-obj-y += hypercall_page.o
-obj-y += xen.o
-
-obj-bin-$(CONFIG_PVH_GUEST) += pvh-boot.init.o
+subdir-$(CONFIG_XEN_GUEST) += xen
xen/arch/x86/guest/xen/Makefile new file +4 -0
new file mode 100644
index 0000000000..26fb4b1007
@@ -0,0 +1,4 @@
+obj-y += hypercall_page.o
+obj-y += xen.o
+
+obj-bin-$(CONFIG_PVH_GUEST) += pvh-boot.init.o
xen/arch/x86/guest/xen/hypercall_page.S renamed from xen/arch/x86/guest/hypercall_page.S +0 -0
similarity index 100%
rename from xen/arch/x86/guest/hypercall_page.S
rename to xen/arch/x86/guest/xen/hypercall_page.S
xen/arch/x86/guest/xen/pvh-boot.c renamed from xen/arch/x86/guest/pvh-boot.c +0 -0
similarity index 100%
rename from xen/arch/x86/guest/pvh-boot.c
rename to xen/arch/x86/guest/xen/pvh-boot.c
xen/arch/x86/guest/xen/xen.c renamed from xen/arch/x86/guest/xen.c +0 -0
similarity index 100%
rename from xen/arch/x86/guest/xen.c
rename to xen/arch/x86/guest/xen/xen.c
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 1/8
#2 lore raw

Re: [Xen-devel] [PATCH for-next RFC 1/8] x86: introduce CONFIG_GUEST and move code

Quoted 10 lines · Wei Liu

Xen is able to run as a guest on Xen. We plan to make it able to run on Hyper-V as well.

Introduce CONFIG_GUEST which is set to true if either running on Xen or Hyper-V is desired. Restructure code hierarchy for new code to come.

No functional change intended.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#3 lore raw

RFC PATCH 2/8 x86: include asm_defns.h directly in hypercall.h

In reply to Wei Liu

ASM_CALL_CONSTRAINT is defined there.

No functional change.

---
 xen/include/asm-x86/guest/hypercall.h | 2 ++
 1 file changed, 2 insertions(+)
xen/include/asm-x86/guest/hypercall.h +2 -0
index d548816b30..c9deca6ffc 100644
@@ -23,6 +23,8 @@
 
 #include <xen/types.h>
 
+#include <asm/asm_defns.h>
+
 #include <public/xen.h>
 #include <public/sched.h>
 #include <public/hvm/hvm_op.h>
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#5 lore raw

RFC PATCH 3/8 x86: drop hypervisor_cpuid_base

In reply to Wei Liu

The only user is Xen specific code in PV shim. We can therefore export the variable directly.

---
 xen/arch/x86/guest/xen/xen.c    | 7 +------
 xen/arch/x86/pv/shim.c          | 2 +-
 xen/include/asm-x86/guest/xen.h | 2 +-
 3 files changed, 3 insertions(+), 8 deletions(-)
xen/arch/x86/guest/xen/xen.c +1 -6
index 7b7a5badab..78fc603996 100644
@@ -37,7 +37,7 @@
 
 bool __read_mostly xen_guest;
 
-static __read_mostly uint32_t xen_cpuid_base;
+__read_mostly uint32_t xen_cpuid_base;
 extern char hypercall_page[];
 static struct rangeset *mem;
 
@@ -301,11 +301,6 @@ int hypervisor_free_unused_page(mfn_t mfn)
     return rangeset_remove_range(mem, mfn_x(mfn), mfn_x(mfn));
 }
 
-uint32_t hypervisor_cpuid_base(void)
-{
-    return xen_cpuid_base;
-}
-
 static void ap_resume(void *unused)
 {
     map_vcpuinfo();
xen/arch/x86/pv/shim.c +1 -1
index 324ca27f93..1b174c24df 100644
@@ -992,7 +992,7 @@ domid_t get_initial_domain_id(void)
     if ( !pv_shim )
         return 0;
 
-    cpuid(hypervisor_cpuid_base() + 4, &eax, &ebx, &ecx, &edx);
+    cpuid(xen_cpuid_base + 4, &eax, &ebx, &ecx, &edx);
 
     return (eax & XEN_HVM_CPUID_DOMID_PRESENT) ? ecx : 1;
 }
xen/include/asm-x86/guest/xen.h +1 -1
index 7e04e4a7ab..b015ed1883 100644
@@ -30,13 +30,13 @@
 
 extern bool xen_guest;
 extern bool pv_console;
+extern uint32_t xen_cpuid_base;
 
 void probe_hypervisor(void);
 void hypervisor_setup(void);
 void hypervisor_ap_setup(void);
 int hypervisor_alloc_unused_page(mfn_t *mfn);
 int hypervisor_free_unused_page(mfn_t mfn);
-uint32_t hypervisor_cpuid_base(void);
 void hypervisor_resume(void);
 
 DECLARE_PER_CPU(unsigned int, vcpu_id);
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 3/8
#6 lore raw

Re: [Xen-devel] [PATCH for-next RFC 3/8] x86: drop hypervisor_cpuid_base

Quoted 4 lines · Wei Liu

The only user is Xen specific code in PV shim. We can therefore export the variable directly.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#7 lore raw

RFC PATCH 4/8 x86: factor out xen variants for hypervisor setup code

In reply to Wei Liu

We will add Hyper-V specific implementations in the future.

No functional change.

---
 xen/arch/x86/guest/xen/xen.c | 32 ++++++++++++++++++++++++++------
 1 file changed, 26 insertions(+), 6 deletions(-)
xen/arch/x86/guest/xen/xen.c +26 -6
index 78fc603996..f93c8fbd1c 100644
@@ -67,7 +67,7 @@ static void __init find_xen_leaves(void)
     }
 }
 
-void __init probe_hypervisor(void)
+static void __init probe_xen(void)
 {
     if ( xen_guest )
         return;
@@ -87,6 +87,11 @@ void __init probe_hypervisor(void)
     xen_guest = true;
 }
 
+void __init probe_hypervisor(void)
+{
+    probe_xen();
+}
+
 static void map_shared_info(void)
 {
     mfn_t mfn;
@@ -249,10 +254,8 @@ static void init_evtchn(void)
     }
 }
 
-void __init hypervisor_setup(void)
+static void __init xen_setup(void)
 {
-    init_memmap();
-
     map_shared_info();
 
     set_vcpu_id();
@@ -277,13 +280,25 @@ void __init hypervisor_setup(void)
     init_evtchn();
 }
 
-void hypervisor_ap_setup(void)
+void __init hypervisor_setup(void)
+{
+    init_memmap();
+
+    xen_setup();
+}
+
+static void xen_ap_setup(void)
 {
     set_vcpu_id();
     map_vcpuinfo();
     init_evtchn();
 }
 
+void hypervisor_ap_setup(void)
+{
+    xen_ap_setup();
+}
+
 int hypervisor_alloc_unused_page(mfn_t *mfn)
 {
     unsigned long m;
@@ -307,7 +322,7 @@ static void ap_resume(void *unused)
     init_evtchn();
 }
 
-void hypervisor_resume(void)
+static void xen_resume(void)
 {
     /* Reset shared info page. */
     map_shared_info();
@@ -330,6 +345,11 @@ void hypervisor_resume(void)
         pv_console_init();
 }
 
+void hypervisor_resume(void)
+{
+    xen_resume();
+}
+
 /*
  * Local variables:
  * mode: C
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#8 lore raw

Re: [Xen-devel] [PATCH for-next RFC 4/8] x86: factor out xen variants for hypervisor setup code

Quoted 14 lines · Wei Liu

We will add Hyper-V specific implementations in the future.

No functional change.

---
 xen/arch/x86/guest/xen/xen.c | 32 ++++++++++++++++++++++++++------
 1 file changed, 26 insertions(+), 6 deletions(-)
xen/arch/x86/guest/xen/xen.c +0 -0
index 78fc603996..f93c8fbd1c 100644

xen/arch/x86/guest/xen/xen.c static void __init find_xen_leaves(void) · line 70

Quoted 5 lines
xen/arch/x86/guest/xen/xen.c +1 -1
@@ -67,7 +67,7 @@ static void __init find_xen_leaves(void)
     }
 }
 
-void __init probe_hypervisor(void)
+static void __init probe_xen(void)

While here I would rename to xen_probe, to match with the other functions (ie: xen_setup below for example).

xen/arch/x86/guest/xen/xen.c void __init probe_hypervisor(void) · line 90

Quoted 8 lines
xen/arch/x86/guest/xen/xen.c +1 -0
 {
     if ( xen_guest )
         return;
@@ -87,6 +87,11 @@ void __init probe_hypervisor(void)
     xen_guest = true;
 }
 
+void __init probe_hypervisor(void)

Shouldn't this live in a separate file, like guest/guest.c or some such?

Also it might be nice to introduce something like:

enum guest_type {
XEN_GUEST,
} guest_type;

So that you can add a switch here, even if the only case is Xen ATM. I guess this will be done in a later patch, or introduce an hypervisor_ops struct that contain pointers to functions to allow for different implementations.

Quoted 20 lines
xen/arch/x86/guest/xen/xen.c +5 -3
+{
+    probe_xen();
+}
+
 static void map_shared_info(void)
 {
     mfn_t mfn;
@@ -249,10 +254,8 @@ static void init_evtchn(void)
     }
 }
 
-void __init hypervisor_setup(void)
+static void __init xen_setup(void)
 {
-    init_memmap();
-
     map_shared_info();
 
     set_vcpu_id();

xen/arch/x86/guest/xen/xen.c void __init hypervisor_setup(void) · line 285

Quoted 7 lines
xen/arch/x86/guest/xen/xen.c +3 -1
@@ -277,13 +280,25 @@ void __init hypervisor_setup(void)
     init_evtchn();
 }
 
-void hypervisor_ap_setup(void)
+void __init hypervisor_setup(void)
+{
+    init_memmap();

I wonder, do you also require to map hypervisor data into the guest physmap when running on HyperV?

Is there a way when running on HyperV to request unused physical address space ranges? What Xen currently does in init_memmap is quite crappy.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#9 lore raw

Re: [Xen-devel] [PATCH for-next RFC 4/8] x86: factor out xen variants for hypervisor setup code

Quoted 5 lines · Roger Pau Monné

xen/arch/x86/guest/xen/xen.c static void __init find_xen_leaves(void) · line 70

Quoted 2 lines
xen/arch/x86/guest/xen/xen.c +1 -1
-void __init probe_hypervisor(void)
+static void __init probe_xen(void)

While here I would rename to xen_probe, to match with the other functions (ie: xen_setup below for example).

Sure. I can do that. I always thought that strange too.

Quoted 8 lines
Quoted 8 lines

{
if ( xen_guest )
return;

xen/arch/x86/guest/xen/xen.c +0 -0
@@ -87,6 +87,11 @@ void __init probe_hypervisor(void)
     xen_guest = true;
 }
 
Quoted 5 lines

xen/arch/x86/guest/xen/xen.c void __init probe_hypervisor(void) · line 90

Quoted 1 line
xen/arch/x86/guest/xen/xen.c +1 -0
+void __init probe_hypervisor(void)

Shouldn't this live in a separate file, like guest/guest.c or some such?

It's done in a later patch. I believe you've already seen it.

Quoted 6 lines

Also it might be nice to introduce something like:

enum guest_type {
XEN_GUEST,
} guest_type;

This works for me.

Quoted 5 lines

So that you can add a switch here, even if the only case is Xen ATM. I guess this will be done in a later patch, or introduce an hypervisor_ops struct that contain pointers to functions to allow for different implementations.

I debated this. These changes require more code churn with no apparent benefit, but if people agree this is better I can make those changes.

Quoted 23 lines
Quoted 23 lines
xen/arch/x86/guest/xen/xen.c +5 -3
+{
+    probe_xen();
+}
+
 static void map_shared_info(void)
 {
     mfn_t mfn;
@@ -249,10 +254,8 @@ static void init_evtchn(void)
     }
 }
 
-void __init hypervisor_setup(void)
+static void __init xen_setup(void)
 {
-    init_memmap();
-
     map_shared_info();
 
     set_vcpu_id();
@@ -277,13 +280,25 @@ void __init hypervisor_setup(void)
     init_evtchn();
 }
 
Quoted 8 lines

xen/arch/x86/guest/xen/xen.c void __init hypervisor_setup(void) · line 285

Quoted 4 lines
xen/arch/x86/guest/xen/xen.c +3 -1
-void hypervisor_ap_setup(void)
+void __init hypervisor_setup(void)
+{
+    init_memmap();

I wonder, do you also require to map hypervisor data into the guest physmap when running on HyperV?

Yes. There are a lot of comparable concepts in Hyper-V. For example, there is a page called VP assist page which is more or less the same as Xen's vcpuinfo. Its format, content and interfaces may be different, but conceptually it is the same as vcpuinfo.

Quoted 4 lines

Is there a way when running on HyperV to request unused physical address space ranges? What Xen currently does in init_memmap is quite crappy.

Xen itself still needs to manage the address space, no?

I agree the rangeset code in xen.c isn't pretty, but it does the job and is not too intrusive.

Wei.

Quoted 1 line

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

Re: [Xen-devel] [PATCH for-next RFC 4/8] x86: factor out xen variants for hypervisor setup code

Quoted 10 lines · Wei Liu
Quoted 6 lines · Roger Pau Monné

Also it might be nice to introduce something like:

enum guest_type {
XEN_GUEST,
} guest_type;

This works for me.

Quoted 8 lines
Quoted 5 lines

So that you can add a switch here, even if the only case is Xen ATM. I guess this will be done in a later patch, or introduce an hypervisor_ops struct that contain pointers to functions to allow for different implementations.

I debated this. These changes require more code churn with no apparent benefit, but if people agree this is better I can make those changes.

Well, if the expectation is for the enum to grow (even just by one further entry), then yes, I'd agree that a switch() would be nice.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#11 lore raw

Re: [Xen-devel] [PATCH for-next RFC 4/8] x86: factor out xen variants for hypervisor setup code

Quoted 17 lines · Jan Beulich
Quoted 16 lines · Wei Liu
Quoted 6 lines · Roger Pau Monné

Also it might be nice to introduce something like:

enum guest_type {
XEN_GUEST,
} guest_type;

This works for me.

Quoted 5 lines

So that you can add a switch here, even if the only case is Xen ATM. I guess this will be done in a later patch, or introduce an hypervisor_ops struct that contain pointers to functions to allow for different implementations.

Quoted 5 lines
Quoted 2 lines

I debated this. These changes require more code churn with no apparent benefit, but if people agree this is better I can make those changes.

Well, if the expectation is for the enum to grow (even just by one further entry), then yes, I'd agree that a switch() would be nice.

Not sure if you notice comments in a later email.

Do you prefer enum+switch solution or hypervisor_op struct?

Wei.

Quoted 2 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw

Re: [Xen-devel] [PATCH for-next RFC 4/8] x86: factor out xen variants for hypervisor setup code

Quoted 21 lines · Wei Liu
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Wei Liu
Quoted 6 lines · Roger Pau Monné

Also it might be nice to introduce something like:

enum guest_type {
XEN_GUEST,
} guest_type;

This works for me.

Quoted 5 lines

So that you can add a switch here, even if the only case is Xen ATM. I guess this will be done in a later patch, or introduce an hypervisor_ops struct that contain pointers to functions to allow for different implementations.

I debated this. These changes require more code churn with no apparent benefit, but if people agree this is better I can make those changes.

Quoted 6 lines
Quoted 2 lines

Well, if the expectation is for the enum to grow (even just by one further entry), then yes, I'd agree that a switch() would be nice.

Not sure if you notice comments in a later email.

Do you prefer enum+switch solution or hypervisor_op struct?

Hard to tell without knowing how many switch() there would end up being. The more of them, the better I'd like the ops structure variant.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#13 lore raw

Re: [Xen-devel] [PATCH for-next RFC 4/8] x86: factor out xen variants for hypervisor setup code

In reply to Wei Liu
Quoted 85 lines · Wei Liu
Quoted 5 lines · Roger Pau Monné

xen/arch/x86/guest/xen/xen.c static void __init find_xen_leaves(void) · line 70

Quoted 2 lines
xen/arch/x86/guest/xen/xen.c +1 -1
-void __init probe_hypervisor(void)
+static void __init probe_xen(void)

While here I would rename to xen_probe, to match with the other functions (ie: xen_setup below for example).

Sure. I can do that. I always thought that strange too.

Quoted 8 lines
Quoted 8 lines

{
if ( xen_guest )
return;

xen/arch/x86/guest/xen/xen.c +0 -0
@@ -87,6 +87,11 @@ void __init probe_hypervisor(void)
     xen_guest = true;
 }
 
Quoted 5 lines

xen/arch/x86/guest/xen/xen.c void __init probe_hypervisor(void) · line 90

Quoted 1 line
xen/arch/x86/guest/xen/xen.c +1 -0
+void __init probe_hypervisor(void)

Shouldn't this live in a separate file, like guest/guest.c or some such?

It's done in a later patch. I believe you've already seen it.

Quoted 6 lines

Also it might be nice to introduce something like:

enum guest_type {
XEN_GUEST,
} guest_type;

This works for me.

Quoted 5 lines

So that you can add a switch here, even if the only case is Xen ATM. I guess this will be done in a later patch, or introduce an hypervisor_ops struct that contain pointers to functions to allow for different implementations.

I debated this. These changes require more code churn with no apparent benefit, but if people agree this is better I can make those changes.

Quoted 23 lines
Quoted 23 lines
xen/arch/x86/guest/xen/xen.c +5 -3
+{
+    probe_xen();
+}
+
 static void map_shared_info(void)
 {
     mfn_t mfn;
@@ -249,10 +254,8 @@ static void init_evtchn(void)
     }
 }
 
-void __init hypervisor_setup(void)
+static void __init xen_setup(void)
 {
-    init_memmap();
-
     map_shared_info();
 
     set_vcpu_id();
@@ -277,13 +280,25 @@ void __init hypervisor_setup(void)
     init_evtchn();
 }
 
Quoted 8 lines

xen/arch/x86/guest/xen/xen.c void __init hypervisor_setup(void) · line 285

Quoted 4 lines
xen/arch/x86/guest/xen/xen.c +3 -1
-void hypervisor_ap_setup(void)
+void __init hypervisor_setup(void)
+{
+    init_memmap();

I wonder, do you also require to map hypervisor data into the guest physmap when running on HyperV?

Yes. There are a lot of comparable concepts in Hyper-V. For example, there is a page called VP assist page which is more or less the same as Xen's vcpuinfo. Its format, content and interfaces may be different, but conceptually it is the same as vcpuinfo.

Quoted 4 lines

Is there a way when running on HyperV to request unused physical address space ranges? What Xen currently does in init_memmap is quite crappy.

Quoted 4 lines

Xen itself still needs to manage the address space, no?

I agree the rangeset code in xen.c isn't pretty, but it does the job and is not too intrusive.

The problem with the current approach is that the nested Xen has no way of knowing whether those holes are actually unused, or are MMIO regions used by devices for example.

Hence I wondered if HyperV had a way to signal to guests that a physical address range is safe to be used as scratch mapping space. We had spoken avoid introducing something in Xen to be able to report to guests safe ranges in the physmap to map stuff.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#14 lore raw

Re: [Xen-devel] [PATCH for-next RFC 4/8] x86: factor out xen variants for hypervisor setup code

Quoted 20 lines · Roger Pau Monné
Quoted 15 lines
Quoted 4 lines

I wonder, do you also require to map hypervisor data into the guest physmap when running on HyperV?

Yes. There are a lot of comparable concepts in Hyper-V. For example, there is a page called VP assist page which is more or less the same as Xen's vcpuinfo. Its format, content and interfaces may be different, but conceptually it is the same as vcpuinfo.

Quoted 4 lines

Is there a way when running on HyperV to request unused physical address space ranges? What Xen currently does in init_memmap is quite crappy.

Quoted 4 lines

Xen itself still needs to manage the address space, no?

I agree the rangeset code in xen.c isn't pretty, but it does the job and is not too intrusive.

Quoted 8 lines

The problem with the current approach is that the nested Xen has no way of knowing whether those holes are actually unused, or are MMIO regions used by devices for example.

Hence I wondered if HyperV had a way to signal to guests that a physical address range is safe to be used as scratch mapping space. We had spoken avoid introducing something in Xen to be able to report to guests safe ranges in the physmap to map stuff.

AFAICT Hyper-V TLFS doesn't describe such functionality.

On the other hand, Hyper-V may not need this infrastructure at all because it doesn't have grant table frame or shared info page. The most likely outcome is in the next version the memmap stuff will be left to Xen only until I find a use case for it.

Wei.

Quoted 2 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#15 lore raw

RFC PATCH 5/8 x86: factor out hypervisor agnostic code

In reply to Wei Liu

The only implementation there is Xen.

No functional change.

---
 xen/arch/x86/guest/Makefile            |   2 +
 xen/arch/x86/guest/hypervisor.c        | 112 +++++++++++++++++++++++++
 xen/arch/x86/guest/xen/xen.c           |  81 +-----------------
 xen/include/asm-x86/guest.h            |   1 +
 xen/include/asm-x86/guest/hypervisor.h |  58 +++++++++++++
 xen/include/asm-x86/guest/xen.h        |  21 ++---
 6 files changed, 182 insertions(+), 93 deletions(-)
 create mode 100644 xen/arch/x86/guest/hypervisor.c
 create mode 100644 xen/include/asm-x86/guest/hypervisor.h
xen/arch/x86/guest/Makefile +2 -0
index 6806f04947..f63d64bbee 100644
@@ -1 +1,3 @@
+obj-y += hypervisor.o
+
 subdir-$(CONFIG_XEN_GUEST) += xen
xen/arch/x86/guest/hypervisor.c new file +112 -0

112 lines not on this page · raw message

xen/arch/x86/guest/xen/xen.c +4 -77

139 lines not on this page · raw message

xen/include/asm-x86/guest.h +1 -0
index a38c6b5b3f..8e167165ae 100644
@@ -20,6 +20,7 @@
 #define __X86_GUEST_H__
 
 #include <asm/guest/hypercall.h>
+#include <asm/guest/hypervisor.h>
 #include <asm/guest/pvh-boot.h>
 #include <asm/guest/xen.h>
 #include <asm/pv/shim.h>
xen/include/asm-x86/guest/hypervisor.h new file +58 -0

58 lines not on this page · raw message

xen/include/asm-x86/guest/xen.h +5 -16
index b015ed1883..d031f1f70d 100644
@@ -32,12 +32,10 @@ extern bool xen_guest;
 extern bool pv_console;
 extern uint32_t xen_cpuid_base;
 
-void probe_hypervisor(void);
-void hypervisor_setup(void);
-void hypervisor_ap_setup(void);
-int hypervisor_alloc_unused_page(mfn_t *mfn);
-int hypervisor_free_unused_page(mfn_t mfn);
-void hypervisor_resume(void);
+void probe_xen(void);
+void xen_setup(void);
+void xen_ap_setup(void);
+void xen_resume(void);
 
 DECLARE_PER_CPU(unsigned int, vcpu_id);
 DECLARE_PER_CPU(struct vcpu_info *, vcpu_info);
@@ -47,16 +45,7 @@ DECLARE_PER_CPU(struct vcpu_info *, vcpu_info);
 #define xen_guest 0
 #define pv_console 0
 
-static inline void probe_hypervisor(void) {}
-
-static inline void hypervisor_setup(void)
-{
-    ASSERT_UNREACHABLE();
-}
-static inline void hypervisor_ap_setup(void)
-{
-    ASSERT_UNREACHABLE();
-}
+static inline void probe_xen(void) {}
 
 #endif /* CONFIG_XEN_GUEST */
 #endif /* __X86_GUEST_XEN_H__ */
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#16 lore raw

Re: [Xen-devel] [PATCH for-next RFC 5/8] x86: factor out hypervisor agnostic code

Quoted 51 lines · Wei Liu

The only implementation there is Xen.

No functional change.

---
 xen/arch/x86/guest/Makefile            |   2 +
 xen/arch/x86/guest/hypervisor.c        | 112 +++++++++++++++++++++++++
 xen/arch/x86/guest/xen/xen.c           |  81 +-----------------
 xen/include/asm-x86/guest.h            |   1 +
 xen/include/asm-x86/guest/hypervisor.h |  58 +++++++++++++
 xen/include/asm-x86/guest/xen.h        |  21 ++---
 6 files changed, 182 insertions(+), 93 deletions(-)
 create mode 100644 xen/arch/x86/guest/hypervisor.c
 create mode 100644 xen/include/asm-x86/guest/hypervisor.h
xen/arch/x86/guest/Makefile +2 -0
index 6806f04947..f63d64bbee 100644
@@ -1 +1,3 @@
+obj-y += hypervisor.o
+
 subdir-$(CONFIG_XEN_GUEST) += xen
xen/arch/x86/guest/hypervisor.c new file +21 -0
new file mode 100644
index 0000000000..b0a724bf13
@@ -0,0 +1,112 @@
+/******************************************************************************
+ * arch/x86/guest/hypervisor.c
+ *
+ * Support for detecting and running under a hypervisor.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Copyright (c) 2017 Citrix Systems Ltd.
+ */
+

xen/arch/x86/guest/hypervisor.c line 31

Quoted 10 lines
xen/arch/x86/guest/hypervisor.c new file +10 -0
+#include <xen/init.h>
+#include <xen/mm.h>
+#include <xen/rangeset.h>
+
+#include <asm/guest.h>
+#include <asm/processor.h>
+
+static struct rangeset *mem;
+
+void __init probe_hypervisor(void)

IMO would be nice to take the opportunity to rename this to hypervisor_probe, to match with the rest of the hypervisor_ functions.

Quoted 26 lines
xen/include/asm-x86/guest/xen.h +4 -7
index b015ed1883..d031f1f70d 100644
@@ -32,12 +32,10 @@ extern bool xen_guest;
 extern bool pv_console;
 extern uint32_t xen_cpuid_base;
 
-void probe_hypervisor(void);
-void hypervisor_setup(void);
-void hypervisor_ap_setup(void);
-int hypervisor_alloc_unused_page(mfn_t *mfn);
-int hypervisor_free_unused_page(mfn_t mfn);
-void hypervisor_resume(void);
+void probe_xen(void);
+void xen_setup(void);
+void xen_ap_setup(void);
+void xen_resume(void);
 
 DECLARE_PER_CPU(unsigned int, vcpu_id);
 DECLARE_PER_CPU(struct vcpu_info *, vcpu_info);
@@ -47,16 +45,7 @@ DECLARE_PER_CPU(struct vcpu_info *, vcpu_info);
 #define xen_guest 0
 #define pv_console 0
 
-static inline void probe_hypervisor(void) {}

xen/include/asm-x86/guest/xen.h DECLARE_PER_CPU(struct vcpu_info *, vcpu_info); · line 48

Quoted 10 lines
xen/include/asm-x86/guest/xen.h +1 -9
-
-static inline void hypervisor_setup(void)
-{
-    ASSERT_UNREACHABLE();
-}
-static inline void hypervisor_ap_setup(void)
-{
-    ASSERT_UNREACHABLE();
-}
+static inline void probe_xen(void) {}

Why do you need this?

AFAICT probe_xen is used in the same way as the rest of the xen_* functions, and hence I wonder why you need a stub for it?

I guess this is a forward-looking change for when probe_xen will be called unconditionally to check for Xen support?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#17 lore raw

Re: [Xen-devel] [PATCH for-next RFC 5/8] x86: factor out hypervisor agnostic code

Quoted 57 lines · Roger Pau Monné
Quoted 56 lines · Wei Liu

The only implementation there is Xen.

No functional change.

---
 xen/arch/x86/guest/Makefile            |   2 +
 xen/arch/x86/guest/hypervisor.c        | 112 +++++++++++++++++++++++++
 xen/arch/x86/guest/xen/xen.c           |  81 +-----------------
 xen/include/asm-x86/guest.h            |   1 +
 xen/include/asm-x86/guest/hypervisor.h |  58 +++++++++++++
 xen/include/asm-x86/guest/xen.h        |  21 ++---
 6 files changed, 182 insertions(+), 93 deletions(-)
 create mode 100644 xen/arch/x86/guest/hypervisor.c
 create mode 100644 xen/include/asm-x86/guest/hypervisor.h
xen/arch/x86/guest/Makefile +2 -0
index 6806f04947..f63d64bbee 100644
@@ -1 +1,3 @@
+obj-y += hypervisor.o
+
 subdir-$(CONFIG_XEN_GUEST) += xen
xen/arch/x86/guest/hypervisor.c new file +26 -0
new file mode 100644
index 0000000000..b0a724bf13
@@ -0,0 +1,112 @@
+/******************************************************************************
+ * arch/x86/guest/hypervisor.c
+ *
+ * Support for detecting and running under a hypervisor.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Copyright (c) 2017 Citrix Systems Ltd.
+ */
+
+#include <xen/init.h>
+#include <xen/mm.h>
+#include <xen/rangeset.h>
+
+#include <asm/guest.h>
Quoted 8 lines

xen/arch/x86/guest/hypervisor.c line 31

Quoted 5 lines
xen/arch/x86/guest/hypervisor.c new file +5 -0
+#include <asm/processor.h>
+
+static struct rangeset *mem;
+
+void __init probe_hypervisor(void)

IMO would be nice to take the opportunity to rename this to hypervisor_probe, to match with the rest of the hypervisor_ functions.

No problem.

Quoted 40 lines
Quoted 27 lines
xen/include/asm-x86/guest/xen.h +4 -7
index b015ed1883..d031f1f70d 100644
@@ -32,12 +32,10 @@ extern bool xen_guest;
 extern bool pv_console;
 extern uint32_t xen_cpuid_base;
 
-void probe_hypervisor(void);
-void hypervisor_setup(void);
-void hypervisor_ap_setup(void);
-int hypervisor_alloc_unused_page(mfn_t *mfn);
-int hypervisor_free_unused_page(mfn_t mfn);
-void hypervisor_resume(void);
+void probe_xen(void);
+void xen_setup(void);
+void xen_ap_setup(void);
+void xen_resume(void);
 
 DECLARE_PER_CPU(unsigned int, vcpu_id);
 DECLARE_PER_CPU(struct vcpu_info *, vcpu_info);
@@ -47,16 +45,7 @@ DECLARE_PER_CPU(struct vcpu_info *, vcpu_info);
 #define xen_guest 0
 #define pv_console 0
 
-static inline void probe_hypervisor(void) {}

xen/include/asm-x86/guest/xen.h DECLARE_PER_CPU(struct vcpu_info *, vcpu_info); · line 48

Quoted 10 lines
xen/include/asm-x86/guest/xen.h +1 -9
-
-static inline void hypervisor_setup(void)
-{
-    ASSERT_UNREACHABLE();
-}
-static inline void hypervisor_ap_setup(void)
-{
-    ASSERT_UNREACHABLE();
-}
+static inline void probe_xen(void) {}

Why do you need this?

Quoted 5 lines

AFAICT probe_xen is used in the same way as the rest of the xen_* functions, and hence I wonder why you need a stub for it?

I guess this is a forward-looking change for when probe_xen will be called unconditionally to check for Xen support?

Yes when probing hypervisor support you will need to unconditionally call these functions.

Wei.

Quoted 2 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#18 lore raw

RFC PATCH 6/8 x86: make probe_xen return boolean value

In reply to Wei Liu

We need indication whether it has succeeded or not.

---
 xen/arch/x86/guest/hypervisor.c | 5 ++++-
 xen/arch/x86/guest/xen/xen.c    | 7 ++++---
 xen/include/asm-x86/guest/xen.h | 4 ++--
 3 files changed, 10 insertions(+), 6 deletions(-)
xen/arch/x86/guest/hypervisor.c +4 -1
index b0a724bf13..fb572b0402 100644
@@ -34,7 +34,10 @@ void __init probe_hypervisor(void)
     if ( !(cpuid_ecx(1) & cpufeat_mask(X86_FEATURE_HYPERVISOR)) )
         return;
 
-    probe_xen();
+    if ( probe_xen() )
+        return;
+
+    /* Hyper-V probing to follow. */
 }
 
 static void __init init_memmap(void)
xen/arch/x86/guest/xen/xen.c +4 -3
index a730e6ad1b..8390b045f0 100644
@@ -66,20 +66,21 @@ static void __init find_xen_leaves(void)
     }
 }
 
-void __init probe_xen(void)
+bool __init probe_xen(void)
 {
     if ( xen_guest )
-        return;
+        return true;
 
     find_xen_leaves();
 
     if ( !xen_cpuid_base )
-        return;
+        return false;
 
     /* Fill the hypercall page. */
     wrmsrl(cpuid_ebx(xen_cpuid_base + 2), __pa(hypercall_page));
 
     xen_guest = true;
+    return true;
 }
 
 static void map_shared_info(void)
xen/include/asm-x86/guest/xen.h +2 -2
index d031f1f70d..7eda3d4956 100644
@@ -32,7 +32,7 @@ extern bool xen_guest;
 extern bool pv_console;
 extern uint32_t xen_cpuid_base;
 
-void probe_xen(void);
+bool probe_xen(void);
 void xen_setup(void);
 void xen_ap_setup(void);
 void xen_resume(void);
@@ -45,7 +45,7 @@ DECLARE_PER_CPU(struct vcpu_info *, vcpu_info);
 #define xen_guest 0
 #define pv_console 0
 
-static inline void probe_xen(void) {}
+static inline bool probe_xen(void) { return false; }
 
 #endif /* CONFIG_XEN_GUEST */
 #endif /* __X86_GUEST_XEN_H__ */
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#19 lore raw

Re: [Xen-devel] [PATCH for-next RFC 6/8] x86: make probe_xen return boolean value

Quoted 3 lines · Wei Liu

We need indication whether it has succeeded or not.

The code LGTM, I have just a suggestion on the approach.

Quoted 11 lines
---
 xen/arch/x86/guest/hypervisor.c | 5 ++++-
 xen/arch/x86/guest/xen/xen.c    | 7 ++++---
 xen/include/asm-x86/guest/xen.h | 4 ++--
 3 files changed, 10 insertions(+), 6 deletions(-)
xen/arch/x86/guest/hypervisor.c +0 -0
index b0a724bf13..fb572b0402 100644

xen/arch/x86/guest/hypervisor.c void __init probe_hypervisor(void) · line 40

Quoted 8 lines
xen/arch/x86/guest/hypervisor.c +4 -1
@@ -34,7 +34,10 @@ void __init probe_hypervisor(void)
     if ( !(cpuid_ecx(1) & cpufeat_mask(X86_FEATURE_HYPERVISOR)) )
         return;
 
-    probe_xen();
+    if ( probe_xen() )
+        return;
+
+    /* Hyper-V probing to follow. */

Instead of having to append a new probe_foo here every time support for running on a new hypervisor is added, you could do something similar to what's done in REGISTER_VPCI_INIT, where each hypervisor would register it's own set of helpers and probe function in a specific section, and then you would just iterate over all the guest support that's compiled in Xen.

That would also prevent you from having to export a dummy probe_xen helper if CONFIG_XEN_GUEST is not defined.

Anyway, maybe that's overkill...

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#20 lore raw

Re: [Xen-devel] [PATCH for-next RFC 6/8] x86: make probe_xen return boolean value

Quoted 34 lines · Roger Pau Monné
Quoted 3 lines · Wei Liu

We need indication whether it has succeeded or not.

The code LGTM, I have just a suggestion on the approach.

Quoted 11 lines
---
 xen/arch/x86/guest/hypervisor.c | 5 ++++-
 xen/arch/x86/guest/xen/xen.c    | 7 ++++---
 xen/include/asm-x86/guest/xen.h | 4 ++--
 3 files changed, 10 insertions(+), 6 deletions(-)
xen/arch/x86/guest/hypervisor.c +0 -0
index b0a724bf13..fb572b0402 100644

xen/arch/x86/guest/hypervisor.c void __init probe_hypervisor(void) · line 40

Quoted 8 lines
xen/arch/x86/guest/hypervisor.c +4 -1
@@ -34,7 +34,10 @@ void __init probe_hypervisor(void)
     if ( !(cpuid_ecx(1) & cpufeat_mask(X86_FEATURE_HYPERVISOR)) )
         return;
 
-    probe_xen();
+    if ( probe_xen() )
+        return;
+
+    /* Hyper-V probing to follow. */

Instead of having to append a new probe_foo here every time support for running on a new hypervisor is added, you could do something similar to what's done in REGISTER_VPCI_INIT, where each hypervisor would register it's own set of helpers and probe function in a specific section, and then you would just iterate over all the guest support that's compiled in Xen.

Quoted 4 lines

That would also prevent you from having to export a dummy probe_xen helper if CONFIG_XEN_GUEST is not defined.

Anyway, maybe that's overkill...

Let's see what Andrew and Jan think.

Either works for me -- it is just a matter of writing code, but obviously the less work the better. :-)

Wei.

Quoted 2 lines

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#21 lore raw

RFC PATCH 7/8 x86: introduce CONFIG_HYPERV and hyperv directory

In reply to Wei Liu

We use the same code structure as we did for Xen code.

As starters, detect Hyper-V in probe_hyperv. More complex functionality will be added later.

---
 xen/arch/x86/Kconfig               |  9 +++++
 xen/arch/x86/guest/Makefile        |  1 +
 xen/arch/x86/guest/hyperv/Makefile |  1 +
 xen/arch/x86/guest/hyperv/hyperv.c | 63 ++++++++++++++++++++++++++++++
 xen/arch/x86/guest/hypervisor.c    |  3 +-
 xen/include/asm-x86/guest.h        |  1 +
 xen/include/asm-x86/guest/hyperv.h | 48 +++++++++++++++++++++++
 7 files changed, 125 insertions(+), 1 deletion(-)
 create mode 100644 xen/arch/x86/guest/hyperv/Makefile
 create mode 100644 xen/arch/x86/guest/hyperv/hyperv.c
 create mode 100644 xen/include/asm-x86/guest/hyperv.h
xen/arch/x86/Kconfig +9 -0
index 584bdc1bb8..c5a93babfe 100644
@@ -163,6 +163,15 @@ endchoice
 config GUEST
 	bool
 
+config HYPERV_GUEST
+	def_bool n
+	select GUEST
+	prompt "Hyper-V Guest"
+	---help---
+	  Support for Xen detecting when it is running under Hyper-V.
+
+	  If unsure, say N.
+
 config XEN_GUEST
 	def_bool n
 	select GUEST
xen/arch/x86/guest/Makefile +1 -0
index f63d64bbee..f164196772 100644
@@ -1,3 +1,4 @@
 obj-y += hypervisor.o
 
+subdir-$(CONFIG_HYPERV_GUEST) += hyperv
 subdir-$(CONFIG_XEN_GUEST) += xen
xen/arch/x86/guest/hyperv/Makefile new file +1 -0
new file mode 100644
index 0000000000..68170109a9
@@ -0,0 +1 @@
+obj-y += hyperv.o
xen/arch/x86/guest/hyperv/hyperv.c new file +63 -0

63 lines not on this page · raw message

xen/arch/x86/guest/hypervisor.c +2 -1
index fb572b0402..17d56cd05e 100644
@@ -37,7 +37,8 @@ void __init probe_hypervisor(void)
     if ( probe_xen() )
         return;
 
-    /* Hyper-V probing to follow. */
+    if ( probe_hyperv() )
+        return;
 }
 
 static void __init init_memmap(void)
xen/include/asm-x86/guest.h +1 -0
index 8e167165ae..94448606d4 100644
@@ -20,6 +20,7 @@
 #define __X86_GUEST_H__
 
 #include <asm/guest/hypercall.h>
+#include <asm/guest/hyperv.h>
 #include <asm/guest/hypervisor.h>
 #include <asm/guest/pvh-boot.h>
 #include <asm/guest/xen.h>
xen/include/asm-x86/guest/hyperv.h new file +48 -0

48 lines not on this page · raw message

Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#22 lore raw

Re: [Xen-devel] [PATCH for-next RFC 7/8] x86: introduce CONFIG_HYPERV and hyperv directory

Quoted 76 lines · Wei Liu

We use the same code structure as we did for Xen code.

As starters, detect Hyper-V in probe_hyperv. More complex functionality will be added later.

---
 xen/arch/x86/Kconfig               |  9 +++++
 xen/arch/x86/guest/Makefile        |  1 +
 xen/arch/x86/guest/hyperv/Makefile |  1 +
 xen/arch/x86/guest/hyperv/hyperv.c | 63 ++++++++++++++++++++++++++++++
 xen/arch/x86/guest/hypervisor.c    |  3 +-
 xen/include/asm-x86/guest.h        |  1 +
 xen/include/asm-x86/guest/hyperv.h | 48 +++++++++++++++++++++++
 7 files changed, 125 insertions(+), 1 deletion(-)
 create mode 100644 xen/arch/x86/guest/hyperv/Makefile
 create mode 100644 xen/arch/x86/guest/hyperv/hyperv.c
 create mode 100644 xen/include/asm-x86/guest/hyperv.h
xen/arch/x86/Kconfig +9 -0
index 584bdc1bb8..c5a93babfe 100644
@@ -163,6 +163,15 @@ endchoice
 config GUEST
 	bool
 
+config HYPERV_GUEST
+	def_bool n
+	select GUEST
+	prompt "Hyper-V Guest"
+	---help---
+	  Support for Xen detecting when it is running under Hyper-V.
+
+	  If unsure, say N.
+
 config XEN_GUEST
 	def_bool n
 	select GUEST
xen/arch/x86/guest/Makefile +1 -0
index f63d64bbee..f164196772 100644
@@ -1,3 +1,4 @@
 obj-y += hypervisor.o
 
+subdir-$(CONFIG_HYPERV_GUEST) += hyperv
 subdir-$(CONFIG_XEN_GUEST) += xen
xen/arch/x86/guest/hyperv/Makefile new file +1 -0
new file mode 100644
index 0000000000..68170109a9
@@ -0,0 +1 @@
+obj-y += hyperv.o
xen/arch/x86/guest/hyperv/hyperv.c new file +15 -0
new file mode 100644
index 0000000000..488e9c14a3
@@ -0,0 +1,63 @@
+/******************************************************************************
+ * arch/x86/guest/hyperv/hyperv.c
+ *
+ * Support for detecting and running under Hyper-V.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *

xen/arch/x86/guest/hyperv/hyperv.c line 25

Quoted 10 lines
xen/arch/x86/guest/hyperv/hyperv.c new file +10 -0
+ * You should have received a copy of the GNU General Public License
+ * along with this program; If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Copyright (c) 2019 Microsoft.
+ */
+#include <xen/init.h>
+
+#include <asm/guest.h>
+
+bool __read_mostly hyperv_guest;

static?

Quoted 18 lines
xen/arch/x86/guest/hyperv/hyperv.c new file +18 -0
+
+bool __init probe_hyperv(void)
+{
+    uint32_t eax, ebx, ecx, edx;
+
+    if ( hyperv_guest )
+        return true;
+
+    cpuid(0x40000000, &eax, &ebx, &ecx, &edx);
+    if ( (ebx == 0x7263694d) && /* "Micr" */
+         (ecx == 0x666f736f) && /* "osof" */
+         (edx == 0x76482074) )  /* "t Hv" */
+        hyperv_guest = true;
+
+    return hyperv_guest;
+}
+
+void __init hyperv_setup(void)

xen/arch/x86/guest/hyperv/hyperv.c line 53

Quoted 10 lines
xen/arch/x86/guest/hyperv/hyperv.c new file +10 -0
+{
+}
+
+void hyperv_ap_setup(void)
+{
+}
+
+void hyperv_resume(void)
+{
+}

Seems kind of pointless to add those dummy hyperv_ functions, AFAICT they don't have any callers yet?

I would prefer that you introduce such dummy callers when they are used.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#23 lore raw

Re: [Xen-devel] [PATCH for-next RFC 7/8] x86: introduce CONFIG_HYPERV and hyperv directory

Quoted 81 lines · Roger Pau Monné
Quoted 80 lines · Wei Liu

We use the same code structure as we did for Xen code.

As starters, detect Hyper-V in probe_hyperv. More complex functionality will be added later.

---
 xen/arch/x86/Kconfig               |  9 +++++
 xen/arch/x86/guest/Makefile        |  1 +
 xen/arch/x86/guest/hyperv/Makefile |  1 +
 xen/arch/x86/guest/hyperv/hyperv.c | 63 ++++++++++++++++++++++++++++++
 xen/arch/x86/guest/hypervisor.c    |  3 +-
 xen/include/asm-x86/guest.h        |  1 +
 xen/include/asm-x86/guest/hyperv.h | 48 +++++++++++++++++++++++
 7 files changed, 125 insertions(+), 1 deletion(-)
 create mode 100644 xen/arch/x86/guest/hyperv/Makefile
 create mode 100644 xen/arch/x86/guest/hyperv/hyperv.c
 create mode 100644 xen/include/asm-x86/guest/hyperv.h
xen/arch/x86/Kconfig +9 -0
index 584bdc1bb8..c5a93babfe 100644
@@ -163,6 +163,15 @@ endchoice
 config GUEST
 	bool
 
+config HYPERV_GUEST
+	def_bool n
+	select GUEST
+	prompt "Hyper-V Guest"
+	---help---
+	  Support for Xen detecting when it is running under Hyper-V.
+
+	  If unsure, say N.
+
 config XEN_GUEST
 	def_bool n
 	select GUEST
xen/arch/x86/guest/Makefile +1 -0
index f63d64bbee..f164196772 100644
@@ -1,3 +1,4 @@
 obj-y += hypervisor.o
 
+subdir-$(CONFIG_HYPERV_GUEST) += hyperv
 subdir-$(CONFIG_XEN_GUEST) += xen
xen/arch/x86/guest/hyperv/Makefile new file +1 -0
new file mode 100644
index 0000000000..68170109a9
@@ -0,0 +1 @@
+obj-y += hyperv.o
xen/arch/x86/guest/hyperv/hyperv.c new file +19 -0
new file mode 100644
index 0000000000..488e9c14a3
@@ -0,0 +1,63 @@
+/******************************************************************************
+ * arch/x86/guest/hyperv/hyperv.c
+ *
+ * Support for detecting and running under Hyper-V.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Copyright (c) 2019 Microsoft.
Quoted 9 lines

xen/arch/x86/guest/hyperv/hyperv.c line 25

Quoted 6 lines
xen/arch/x86/guest/hyperv/hyperv.c new file +6 -0
+ */
+#include <xen/init.h>
+
+#include <asm/guest.h>
+
+bool __read_mostly hyperv_guest;

static?

I expected this to be referenced elsewhere, hence I made it non-static.

Quoted 32 lines
Quoted 18 lines
xen/arch/x86/guest/hyperv/hyperv.c new file +18 -0
+
+bool __init probe_hyperv(void)
+{
+    uint32_t eax, ebx, ecx, edx;
+
+    if ( hyperv_guest )
+        return true;
+
+    cpuid(0x40000000, &eax, &ebx, &ecx, &edx);
+    if ( (ebx == 0x7263694d) && /* "Micr" */
+         (ecx == 0x666f736f) && /* "osof" */
+         (edx == 0x76482074) )  /* "t Hv" */
+        hyperv_guest = true;
+
+    return hyperv_guest;
+}
+
+void __init hyperv_setup(void)

xen/arch/x86/guest/hyperv/hyperv.c line 53

Quoted 10 lines
xen/arch/x86/guest/hyperv/hyperv.c new file +10 -0
+{
+}
+
+void hyperv_ap_setup(void)
+{
+}
+
+void hyperv_resume(void)
+{
+}

Seems kind of pointless to add those dummy hyperv_ functions, AFAICT they don't have any callers yet?

Quoted 3 lines

I would prefer that you introduce such dummy callers when they are used.

No problem.

Wei.

Quoted 1 line

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#24 lore raw

RFC PATCH 8/8 x86: be more verbose when running nested

In reply to Wei Liu
---
 xen/arch/x86/setup.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
xen/arch/x86/setup.c +11 -0
index dec60d0301..bbcc5a503d 100644
@@ -789,6 +789,17 @@ void __init noreturn __start_xen(unsigned long mbi_p)
 
     printk("Xen image load base address: %#lx\n", xen_phys_start);
 
+#ifdef CONFIG_GUEST
+#ifdef CONFIG_XEN_GUEST
+    if ( xen_guest )
+        printk("Running on Xen hypervisor\n");
+#endif
+#ifdef CONFIG_HYPERV_GUEST
+    if ( hyperv_guest )
+        printk("Running on Hyper-V\n");
+#endif
+#endif
+
 #ifdef CONFIG_VIDEO
     printk("Video information:\n");
 
Signature · 7 lines
2.20.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#25 lore raw

Re: [Xen-devel] [PATCH for-next RFC 8/8] x86: be more verbose when running nested

Quoted 13 lines · Wei Liu
---
 xen/arch/x86/setup.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
xen/arch/x86/setup.c +0 -0
index dec60d0301..bbcc5a503d 100644
@@ -789,6 +789,17 @@ void __init noreturn __start_xen(unsigned long mbi_p)
 
     printk("Xen image load base address: %#lx\n", xen_phys_start);
 

xen/arch/x86/setup.c void __init noreturn __start_xen(unsigned long mbi_p) · line 801

Quoted 10 lines
xen/arch/x86/setup.c +10 -0
+#ifdef CONFIG_GUEST
+#ifdef CONFIG_XEN_GUEST
+    if ( xen_guest )
+        printk("Running on Xen hypervisor\n");
+#endif
+#ifdef CONFIG_HYPERV_GUEST
+    if ( hyperv_guest )
+        printk("Running on Hyper-V\n");
+#endif
+#endif

Urg, that's kind of ugly. Why not introduce a hypervisor_name helper or some such?

You could even have a structure like:

struct hypervisor_ops {
    const char *name;
    bool (*probe)(void);
    void (*setup)(void);
    ...
}

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#26 lore raw
In reply to Wei Liu
Quoted 24 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Liu
Sent: 23 September 2019 11:09
To: Xen Development List <[email protected]>
Cc: Wei Liu <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

Hi all

In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network.

This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon.

There are two major areas to be worked on:
* Make Dom0 able to use Hyper-V's synthetic devices.
* Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces.

Quoted 4 lines

They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready.

Questions and comments are welcome.

Yay! Glad you found the time to do this :-) Making use of the synic, stimer, will be useful. To get the benefit for nested guests though, I assume we'll need to exposed enlightened VMCS, EPT flush, etc. through?

Cheers,

Paul

Quoted 47 lines

Thanks,
Wei.

Cc Jan Beulich <[email protected]>
Cc Andrew Cooper <[email protected]>
Cc Wei Liu <[email protected]>
Cc Roger Pau Monné <[email protected]>

Wei Liu (8):
x86: introduce CONFIG_GUEST and move code
x86: include asm_defns.h directly in hypercall.h
x86: drop hypervisor_cpuid_base
x86: factor out xen variants for hypervisor setup code
x86: factor out hypervisor agnostic code
x86: make probe_xen return boolean value
x86: introduce CONFIG_HYPERV and hyperv directory
x86: be more verbose when running nested

 xen/arch/x86/Kconfig                          |  13 ++
 xen/arch/x86/Makefile                         |   2 +-
 xen/arch/x86/guest/Makefile                   |   6 +-
 xen/arch/x86/guest/hyperv/Makefile            |   1 +
 xen/arch/x86/guest/hyperv/hyperv.c            |  63 ++++++++++
 xen/arch/x86/guest/hypervisor.c               | 116 ++++++++++++++++++
 xen/arch/x86/guest/xen/Makefile               |   4 +
 xen/arch/x86/guest/{ => xen}/hypercall_page.S |   0
 xen/arch/x86/guest/{ => xen}/pvh-boot.c       |   0
 xen/arch/x86/guest/{ => xen}/xen.c            |  73 ++---------
 xen/arch/x86/pv/shim.c                        |   2 +-
 xen/arch/x86/setup.c                          |  11 ++
 xen/include/asm-x86/guest.h                   |   2 +
 xen/include/asm-x86/guest/hypercall.h         |   2 +
 xen/include/asm-x86/guest/hyperv.h            |  48 ++++++++
 xen/include/asm-x86/guest/hypervisor.h        |  58 +++++++++
 xen/include/asm-x86/guest/xen.h               |  23 +---
 17 files changed, 337 insertions(+), 87 deletions(-)
 create mode 100644 xen/arch/x86/guest/hyperv/Makefile
 create mode 100644 xen/arch/x86/guest/hyperv/hyperv.c
 create mode 100644 xen/arch/x86/guest/hypervisor.c
 create mode 100644 xen/arch/x86/guest/xen/Makefile
 rename xen/arch/x86/guest/{ => xen}/hypercall_page.S (100%)
 rename xen/arch/x86/guest/{ => xen}/pvh-boot.c (100%)
 rename xen/arch/x86/guest/{ => xen}/xen.c (80%)
 create mode 100644 xen/include/asm-x86/guest/hyperv.h
 create mode 100644 xen/include/asm-x86/guest/hypervisor.h
Quoted 8 lines

--
2.20.1

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#27 lore raw
Quoted 27 lines · Paul Durrant
Quoted 27 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Liu
Sent: 23 September 2019 11:09
To: Xen Development List <[email protected]>
Cc: Wei Liu <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

Hi all

In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network.

This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon.

There are two major areas to be worked on:
* Make Dom0 able to use Hyper-V's synthetic devices.
* Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces.

They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready.

Quoted 6 lines
Quoted 1 line

Questions and comments are welcome.

Yay! Glad you found the time to do this :-) Making use of the synic, stimer, will be useful. To get the benefit for nested guests though, I assume we'll need to exposed enlightened VMCS, EPT flush, etc. through?

Per TLFS, eVMCS should be used by L1 Xen.

Not sure what you meant by EPT flush. If you mean things like HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen only as well. I'm not sure whether they should be expose to L2 guests.

There are hypercalls for virtual TLB management. We may choose to expose them to L2 guests for better performance.

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#28 lore raw
Quoted 38 lines

-----Original Message-----
From: Wei Liu <[email protected]>
Sent: 23 September 2019 12:27
To: Paul Durrant <[email protected]>
Cc: 'Wei Liu' <[email protected]>; Xen Development List <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

Quoted 5 lines · Paul Durrant
Quoted 5 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Liu
Sent: 23 September 2019 11:09
To: Xen Development List <[email protected]>
Cc: Wei Liu <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper

<[email protected]>;

Quoted 22 lines
Quoted 22 lines

Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

Hi all

In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network.

This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon.

There are two major areas to be worked on:
* Make Dom0 able to use Hyper-V's synthetic devices.
* Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces.

They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Questions and comments are welcome.

Yay! Glad you found the time to do this :-) Making use of the synic, stimer, will be useful. To get the benefit for nested guests though, I assume we'll need to exposed enlightened VMCS, EPT flush, etc. through?

Per TLFS, eVMCS should be used by L1 Xen.

Yes, I guess it only needs to be used by L1, but Windows is using an increasing number of VMs for various purposes so I think making it stack well would be useful.

Quoted 5 lines

Not sure what you meant by EPT flush. If you mean things like HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen only as well. I'm not sure whether they should be expose to L2 guests.

Yes, that's what I meant and, again, I was thinking about the nested Windows guest using VMs itself.

Quoted 3 lines

There are hypercalls for virtual TLB management. We may choose to expose them to L2 guests for better performance.

Do you mean 'Direct Virtual Flush'? The hypercalls in question are already handled in viridian for the purposes of remote TLB flush, so I guess it would be a question of wiring them through.

Paul

Quoted 1 line

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#29 lore raw
Quoted 52 lines · Paul Durrant
Quoted 38 lines

-----Original Message-----
From: Wei Liu <[email protected]>
Sent: 23 September 2019 12:27
To: Paul Durrant <[email protected]>
Cc: 'Wei Liu' <[email protected]>; Xen Development List <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

Quoted 5 lines · Paul Durrant
Quoted 5 lines

-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Wei Liu
Sent: 23 September 2019 11:09
To: Xen Development List <[email protected]>
Cc: Wei Liu <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper

<[email protected]>;

Quoted 22 lines
Quoted 22 lines

Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

Hi all

In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network.

This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon.

There are two major areas to be worked on:
* Make Dom0 able to use Hyper-V's synthetic devices.
* Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces.

They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Questions and comments are welcome.

Yay! Glad you found the time to do this :-) Making use of the synic, stimer, will be useful. To get the benefit for nested guests though, I assume we'll need to exposed enlightened VMCS, EPT flush, etc. through?

Per TLFS, eVMCS should be used by L1 Xen.

Yes, I guess it only needs to be used by L1, but Windows is using an increasing number of VMs for various purposes so I think making it stack well would be useful.

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

Not sure what you meant by EPT flush. If you mean things like HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen only as well. I'm not sure whether they should be expose to L2 guests.

Yes, that's what I meant and, again, I was thinking about the nested Windows guest using VMs itself.

So this is basically nested on nested. I'm not sure how to make that work yet. I'm also not sure how well these interfaces stack. TLFS only describes one level of nesting. I will need to ask if we can make nested on nested work.

Quoted 6 lines
Quoted 4 lines

There are hypercalls for virtual TLB management. We may choose to expose them to L2 guests for better performance.

Do you mean 'Direct Virtual Flush'? The hypercalls in question are already handled in viridian for the purposes of remote TLB flush, so I guess it would be a question of wiring them through.

Yes. I meant that.

Wei.

Quoted 4 lines

Paul

Quoted 1 line

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#30 lore raw

And a bit more thought.

On Mon, Sep 23, 2019 at 01:54:31PM +0100, Wei Liu wrote:
[...]

Quoted 13 lines
Quoted 13 lines
Quoted 2 lines

Per TLFS, eVMCS should be used by L1 Xen.

Yes, I guess it only needs to be used by L1, but Windows is using an increasing number of VMs for various purposes so I think making it stack well would be useful.

Quoted 5 lines

Not sure what you meant by EPT flush. If you mean things like HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen only as well. I'm not sure whether they should be expose to L2 guests.

Quoted 8 lines
Quoted 2 lines

Yes, that's what I meant and, again, I was thinking about the nested Windows guest using VMs itself.

So this is basically nested on nested. I'm not sure how to make that work yet. I'm also not sure how well these interfaces stack. TLFS only describes one level of nesting. I will need to ask if we can make nested on nested work.

Since L2 Root is Windows in a Hyper-V setup, chances are what you want already works. Of course this depends on whether Windows, when running as Root or Child, uses VM related features.

I'm not versed in Windows features. If you have specific things in mind, I can ask for more details and think about how to implement them in Xen.

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#31 lore raw
Quoted 35 lines

-----Original Message-----
From: Wei Liu <[email protected]>
Sent: 23 September 2019 14:34
To: Paul Durrant <[email protected]>
Cc: 'Wei Liu' <[email protected]>; Xen Development List <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

And a bit more thought.

On Mon, Sep 23, 2019 at 01:54:31PM +0100, Wei Liu wrote:
[...]

Quoted 13 lines
Quoted 13 lines
Quoted 2 lines

Per TLFS, eVMCS should be used by L1 Xen.

Yes, I guess it only needs to be used by L1, but Windows is using an increasing number of VMs for various purposes so I think making it stack well would be useful.

Quoted 5 lines

Not sure what you meant by EPT flush. If you mean things like HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen only as well. I'm not sure whether they should be expose to L2 guests.

Quoted 8 lines
Quoted 2 lines

Yes, that's what I meant and, again, I was thinking about the nested Windows guest using VMs itself.

So this is basically nested on nested. I'm not sure how to make that work yet. I'm also not sure how well these interfaces stack. TLFS only describes one level of nesting. I will need to ask if we can make nested on nested work.

Quoted 6 lines

Since L2 Root is Windows in a Hyper-V setup, chances are what you want already works. Of course this depends on whether Windows, when running as Root or Child, uses VM related features.

I'm not versed in Windows features. If you have specific things in mind, I can ask for more details and think about how to implement them in Xen.

Well, eVMCS would be a major one. If the L2 Windows could be told to use that rather than L0 have to bounce individual VMREAD/VMWRITEs up to L1 then that would be a win.

Paul

Quoted 2 lines

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#32 lore raw
Quoted 39 lines · Paul Durrant
Quoted 39 lines

-----Original Message-----
From: Wei Liu <[email protected]>
Sent: 23 September 2019 14:34
To: Paul Durrant <[email protected]>
Cc: 'Wei Liu' <[email protected]>; Xen Development List <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

And a bit more thought.

On Mon, Sep 23, 2019 at 01:54:31PM +0100, Wei Liu wrote:
[...]

Quoted 13 lines
Quoted 13 lines
Quoted 2 lines

Per TLFS, eVMCS should be used by L1 Xen.

Yes, I guess it only needs to be used by L1, but Windows is using an increasing number of VMs for various purposes so I think making it stack well would be useful.

Quoted 5 lines

Not sure what you meant by EPT flush. If you mean things like HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen only as well. I'm not sure whether they should be expose to L2 guests.

Quoted 8 lines
Quoted 2 lines

Yes, that's what I meant and, again, I was thinking about the nested Windows guest using VMs itself.

So this is basically nested on nested. I'm not sure how to make that work yet. I'm also not sure how well these interfaces stack. TLFS only describes one level of nesting. I will need to ask if we can make nested on nested work.

Since L2 Root is Windows in a Hyper-V setup, chances are what you want already works. Of course this depends on whether Windows, when running as Root or Child, uses VM related features.

Quoted 6 lines
Quoted 2 lines

I'm not versed in Windows features. If you have specific things in mind, I can ask for more details and think about how to implement them in Xen.

Well, eVMCS would be a major one. If the L2 Windows could be told to use that rather than L0 have to bounce individual VMREAD/VMWRITEs up to L1 then that would be a win.

Oh, I was thinking more about names of Windows specific high-level features. Yes I can definitely ask whether eVMCS is exposed to L2 Dom0 / Root.

Wei.

Quoted 5 lines

Paul

Quoted 2 lines

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#33 lore raw
Quoted 52 lines

-----Original Message-----
From: Wei Liu <[email protected]>
Sent: 23 September 2019 15:21
To: Paul Durrant <[email protected]>
Cc: 'Wei Liu' <[email protected]>; Xen Development List <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

Quoted 42 lines · Paul Durrant
Quoted 41 lines

-----Original Message-----
From: Wei Liu <[email protected]>
Sent: 23 September 2019 14:34
To: Paul Durrant <[email protected]>
Cc: 'Wei Liu' <[email protected]>; Xen Development List <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

And a bit more thought.

On Mon, Sep 23, 2019 at 01:54:31PM +0100, Wei Liu wrote:
[...]

Quoted 13 lines
Quoted 13 lines
Quoted 2 lines

Per TLFS, eVMCS should be used by L1 Xen.

Yes, I guess it only needs to be used by L1, but Windows is using an increasing number of VMs for various purposes so I think making it stack well would be useful.

Quoted 5 lines

Not sure what you meant by EPT flush. If you mean things like HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen only as well. I'm not sure whether they should be expose to L2 guests.

Quoted 8 lines
Quoted 2 lines

Yes, that's what I meant and, again, I was thinking about the nested Windows guest using VMs itself.

So this is basically nested on nested. I'm not sure how to make that work yet. I'm also not sure how well these interfaces stack. TLFS only describes one level of nesting. I will need to ask if we can make nested on nested work.

Since L2 Root is Windows in a Hyper-V setup, chances are what you want already works. Of course this depends on whether Windows, when running as Root or Child, uses VM related features.

I'm not versed in Windows features. If you have specific things in mind, I can ask for more details and think about how to implement them in Xen.

Quoted 7 lines
Quoted 3 lines

Well, eVMCS would be a major one. If the L2 Windows could be told to use that rather than L0 have to bounce individual VMREAD/VMWRITEs up to L1 then that would be a win.

Oh, I was thinking more about names of Windows specific high-level features. Yes I can definitely ask whether eVMCS is exposed to L2 Dom0 / Root.

Oh, I see. The main ones are 'credential guard' and the more recent 'sandbox'.

Paul

Quoted 8 lines

Wei.

Quoted 5 lines

Paul

Quoted 2 lines

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#34 lore raw
Quoted 56 lines · Paul Durrant
Quoted 56 lines

-----Original Message-----
From: Wei Liu <[email protected]>
Sent: 23 September 2019 15:21
To: Paul Durrant <[email protected]>
Cc: 'Wei Liu' <[email protected]>; Xen Development List <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

Quoted 45 lines · Paul Durrant
Quoted 35 lines

-----Original Message-----
From: Wei Liu <[email protected]>
Sent: 23 September 2019 14:34
To: Paul Durrant <[email protected]>
Cc: 'Wei Liu' <[email protected]>; Xen Development List <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; Michael Kelley <[email protected]>; Jan Beulich <[email protected]>; Roger Pau Monne <[email protected]>
Subject: Re: [Xen-devel] [PATCH for-next RFC 0/8] Port Xen to Hyper-V

And a bit more thought.

On Mon, Sep 23, 2019 at 01:54:31PM +0100, Wei Liu wrote:
[...]

Quoted 13 lines
Quoted 13 lines
Quoted 2 lines

Per TLFS, eVMCS should be used by L1 Xen.

Yes, I guess it only needs to be used by L1, but Windows is using an increasing number of VMs for various purposes so I think making it stack well would be useful.

Quoted 5 lines

Not sure what you meant by EPT flush. If you mean things like HvFlushGuestPhysicalAddressSpace, I think they should be used by Xen only as well. I'm not sure whether they should be expose to L2 guests.

Quoted 8 lines
Quoted 2 lines

Yes, that's what I meant and, again, I was thinking about the nested Windows guest using VMs itself.

So this is basically nested on nested. I'm not sure how to make that work yet. I'm also not sure how well these interfaces stack. TLFS only describes one level of nesting. I will need to ask if we can make nested on nested work.

Quoted 6 lines

Since L2 Root is Windows in a Hyper-V setup, chances are what you want already works. Of course this depends on whether Windows, when running as Root or Child, uses VM related features.

I'm not versed in Windows features. If you have specific things in mind, I can ask for more details and think about how to implement them in Xen.

Well, eVMCS would be a major one. If the L2 Windows could be told to use that rather than L0 have to bounce individual VMREAD/VMWRITEs up to L1 then that would be a win.

Quoted 6 lines
Quoted 3 lines

Oh, I was thinking more about names of Windows specific high-level features. Yes I can definitely ask whether eVMCS is exposed to L2 Dom0 / Root.

Oh, I see. The main ones are 'credential guard' and the more recent 'sandbox'.

Got it.

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#35 lore raw
In reply to Wei Liu
Quoted 15 lines · Wei Liu

Hi all

In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network.

This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon.

There are two major areas to be worked on:
* Make Dom0 able to use Hyper-V's synthetic devices.
* Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces.

Quoted 4 lines

They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready.

Questions and comments are welcome.

Thanks for doing this!

In the past I've played with trying to get Xen to boot as a guest on HyperV gen2 instances, and I did manage to get it booting, Linux however would get extremely confused because gen2 lacks emulated devices and instead the guest is forced to use the HyperV PV devices AFAIK.

Anyway, I had the following patch on my backlog which allowed me to boot on gen2 instances, posting it here in case it helps you if you plan to go that route.

Roger.
---8<---
From 7c9fad735fdc3633ed401befda26c21dc7145af8 Mon Sep 17 00:00:00 2001
From: Roger Pau Monne <[email protected]>
Date: Tue, 18 Sep 2018 11:36:42 +0200
Subject: [PATCH] x86/lapic: remove the PIT usage to calibrate the lapic timer
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

And instead use NOW which is based on the TSC. This was already used when running in shim mode, since there's likely no PIT in that environment.

Remove printing the CPU frequency, since it's already printed earlier at boot, and getting the CPU frequency against the TSC without any external reference timer is pointless.

The motivation behind this change is to allow Xen to boot on HyperV gen2 instances, which lack a PIT.

---
I'm not sure about the reason behind the usage of the PIT instead of the TSC, maybe this was done because the TSC wasn't available until the Pentium? Xen certainly doesn't care about such hardware anymore, and the TSC is already used unconditionally in Xen.

Linux seems to prefer to calibrate the lapic timer against the PM timer and has already dropped PIT usage for that.

My early tests on a single box show no differences between the TSC or the PIT for lapic timer calibration, but that's a single box.

The RFC is because I'm not sure I understand the motivation for using the PIT in the first place, so I might be missing something relevant that could make this patch moot.
---
Cc: Jan Beulich <[email protected]>
Cc: Andrew Cooper <[email protected]>
Cc: Wei Liu <[email protected]>

---
 xen/arch/x86/apic.c | 67 ++-------------------------------------------
 1 file changed, 2 insertions(+), 65 deletions(-)
xen/arch/x86/apic.c +2 -65
index 88ada9d0ec..21dbeb2298 100644
@@ -1010,46 +1010,6 @@ __next:
 /* used for system time scaling */
 static u32 __read_mostly bus_scale; /* scaling factor: ns -> bus cycles */
 
-/*
- * The timer chip is already set up at HZ interrupts per second here,
- * but we do not accept timer interrupts yet. We only allow the BP
- * to calibrate.
- */
-static unsigned int __init get_8254_timer_count(void)
-{
-    /*extern spinlock_t i8253_lock;*/
-    /*unsigned long flags;*/
-
-    unsigned int count;
-
-    /*spin_lock_irqsave(&i8253_lock, flags);*/
-
-    outb_p(0x00, PIT_MODE);
-    count = inb_p(PIT_CH0);
-    count |= inb_p(PIT_CH0) << 8;
-
-    /*spin_unlock_irqrestore(&i8253_lock, flags);*/
-
-    return count;
-}
-
-/* next tick in 8254 can be caught by catching timer wraparound */
-static void __init wait_8254_wraparound(void)
-{
-    unsigned int curr_count, prev_count;
-    
-    curr_count = get_8254_timer_count();
-    do {
-        prev_count = curr_count;
-        curr_count = get_8254_timer_count();
-
-        /* workaround for broken Mercury/Neptune */
-        if (prev_count >= curr_count + 0x100)
-            curr_count = get_8254_timer_count();
-        
-    } while (prev_count >= curr_count);
-}
-
 /*
  * This function sets up the local APIC timer, with a timeout of
  * 'clocks' APIC bus clock. During calibration we actually call
@@ -1092,7 +1052,7 @@ static void setup_APIC_timer(void)
     local_irq_restore(flags);
 }
 
-static void wait_tick_pvh(void)
+static void wait_tick(void)
 {
     u64 lapse_ns = 1000000000ULL / HZ;
     s_time_t start, curr_time;
@@ -1121,7 +1081,6 @@ static void wait_tick_pvh(void)
 
 static int __init calibrate_APIC_clock(void)
 {
-    unsigned long long t1, t2;
     long tt1, tt2;
     long result;
     int i;
@@ -1138,33 +1097,15 @@ static int __init calibrate_APIC_clock(void)
      */
     __setup_APIC_LVTT(1000000000);
 
-    if ( !xen_guest )
-        /*
-         * The timer chip counts down to zero. Let's wait
-         * for a wraparound to start exact measurement:
-         * (the current tick might have been already half done)
-         */
-        wait_8254_wraparound();
-    else
-        wait_tick_pvh();
-
-    /*
-     * We wrapped around just now. Let's start:
-     */
-    t1 = rdtsc_ordered();
     tt1 = apic_read(APIC_TMCCT);
 
     /*
      * Let's wait LOOPS ticks:
      */
     for (i = 0; i < LOOPS; i++)
-        if ( !xen_guest )
-            wait_8254_wraparound();
-        else
-            wait_tick_pvh();
+        wait_tick();
 
     tt2 = apic_read(APIC_TMCCT);
-    t2 = rdtsc_ordered();
 
     /*
      * The APIC bus clock counter is 32 bits only, it
@@ -1176,10 +1117,6 @@ static int __init calibrate_APIC_clock(void)
 
     result = (tt1-tt2)*APIC_DIVISOR/LOOPS;
 
-    apic_printk(APIC_VERBOSE, "..... CPU clock speed is %ld.%04ld MHz.\n",
-                ((long)(t2 - t1) / LOOPS) / (1000000 / HZ),
-                ((long)(t2 - t1) / LOOPS) % (1000000 / HZ));
-
     apic_printk(APIC_VERBOSE, "..... host bus clock speed is %ld.%04ld MHz.\n",
                 result / (1000000 / HZ), result % (1000000 / HZ));
 
Signature · 6 lines
2.22.0

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#36 lore raw
Quoted 21 lines · Roger Pau Monné
Quoted 15 lines · Wei Liu

Hi all

In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network.

This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon.

There are two major areas to be worked on:
* Make Dom0 able to use Hyper-V's synthetic devices.
* Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces.

Quoted 4 lines

They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready.

Questions and comments are welcome.

Quoted 7 lines

Thanks for doing this!

In the past I've played with trying to get Xen to boot as a guest on HyperV gen2 instances, and I did manage to get it booting, Linux however would get extremely confused because gen2 lacks emulated devices and instead the guest is forced to use the HyperV PV devices AFAIK.

How did you force Linux to use Hyper-V PV devices? AIUI Linux won't have access to VMBus (like XenBus but with some differences) and SynIC (Synthetic interrupt controller, think of it as an abstraction around event channel in Xen world) in the first place. But I haven't tried gen2 so I could be wrong.

Quoted 4 lines

Anyway, I had the following patch on my backlog which allowed me to boot on gen2 instances, posting it here in case it helps you if you plan to go that route.

The long term plan for the time source is to use the reference TSC page provided by Hyper-V, just like we did for Xen.

Your patch seems to be an useful thing to do on its own anyway.

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#37 lore raw
Quoted 30 lines · Wei Liu
Quoted 21 lines · Roger Pau Monné
Quoted 15 lines · Wei Liu

Hi all

In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network.

This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon.

There are two major areas to be worked on:
* Make Dom0 able to use Hyper-V's synthetic devices.
* Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces.

Quoted 4 lines

They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready.

Questions and comments are welcome.

Quoted 7 lines

Thanks for doing this!

In the past I've played with trying to get Xen to boot as a guest on HyperV gen2 instances, and I did manage to get it booting, Linux however would get extremely confused because gen2 lacks emulated devices and instead the guest is forced to use the HyperV PV devices AFAIK.

Quoted 6 lines

How did you force Linux to use Hyper-V PV devices? AIUI Linux won't have access to VMBus (like XenBus but with some differences) and SynIC (Synthetic interrupt controller, think of it as an abstraction around event channel in Xen world) in the first place. But I haven't tried gen2 so I could be wrong.

I try Gen2 this morning. I need to apply your patch and give Xen noapic option to boot. At the end I only get a blank screen so I don't think Linux has booted successfully.

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#38 lore raw
In reply to Wei Liu
Quoted 24 lines · Wei Liu
Quoted 23 lines · Roger Pau Monné
Quoted 15 lines · Wei Liu

Hi all

In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network.

This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon.

There are two major areas to be worked on:
* Make Dom0 able to use Hyper-V's synthetic devices.
* Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces.

Quoted 4 lines

They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready.

Questions and comments are welcome.

Thanks for doing this!

Quoted 7 lines
Quoted 5 lines

In the past I've played with trying to get Xen to boot as a guest on HyperV gen2 instances, and I did manage to get it booting, Linux however would get extremely confused because gen2 lacks emulated devices and instead the guest is forced to use the HyperV PV devices AFAIK.

How did you force Linux to use Hyper-V PV devices?

I didn't, that's where I stopped. I just managed to get Xen booting on gen2, but Linux dom0 won't boot due to what you pointed out below.

Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#39 lore raw
Quoted 31 lines · Roger Pau Monné
Quoted 30 lines · Wei Liu
Quoted 28 lines · Roger Pau Monné
Quoted 15 lines · Wei Liu

Hi all

In case you're wondering, I can already run a fully fledged Xen system on Hyper-V with emulated disk and network.

This is the very first stage for porting Xen to run on Hyper-V with all the goodies Hyper-V has to offer. With this series, Xen can successfully detect Hyper-V and prints out a message. I would like to first get the directory structure and kconfig options agreed upon.

There are two major areas to be worked on:
* Make Dom0 able to use Hyper-V's synthetic devices.
* Make Xen use of the synthetic timer, reference TSC and enlightenment VMCS and other interfaces.

Quoted 4 lines

They aren't trivial, and time can be scarce on my side, so I intend to post patches piece meal when they are ready.

Questions and comments are welcome.

Thanks for doing this!

In the past I've played with trying to get Xen to boot as a guest on HyperV gen2 instances, and I did manage to get it booting, Linux however would get extremely confused because gen2 lacks emulated devices and instead the guest is forced to use the HyperV PV devices AFAIK.

Quoted 5 lines
Quoted 1 line

How did you force Linux to use Hyper-V PV devices?

I didn't, that's where I stopped. I just managed to get Xen booting on gen2, but Linux dom0 won't boot due to what you pointed out below.

OK. Thanks for clarification.

I think when the port is complete a Xen system should be able to boot on both Gen1 and Gen2.

Wei.

Quoted 2 lines

Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel