Skip to contentSkip to conversation

xen/pvh: Dom0 support

PATCH Superseded Roger Pau Monne 2 participants 9 messages May 3, 2018 – May 4, 2018

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover xen/pvh: Dom0 support

Hello,

The following patches enable booting a PVH Dom0. So far I've only tested them in my test box, so I expect more issues will show up as this gets further testing. In any case, this is the bare minimum to get a PVH Dom0 working.

In order to test them Xen 4.11 RC or plain master/staging should be used. Compile a Linux kernel with this patches and PVH support and add dom0=pvh to the Xen command line. The following message will be displayed during boot by the Linux kernel if PVH mode is used:

"Booting paravirtualized kernel on Xen PVH"

Thanks, Roger.

Roger Pau Monne (3):
xen/pvh: enable and set default MTRR type xen/store: do not store local values in xen_start_info
xen: share start flags between PV and PVH

 arch/x86/xen/enlighten.c          | 7 +++++++
 arch/x86/xen/enlighten_pv.c       | 1 +
 arch/x86/xen/enlighten_pvh.c      | 4 ++++
 drivers/xen/xenbus/xenbus_probe.c | 5 ++---
 include/xen/xen.h                 | 4 +++-
 5 files changed, 17 insertions(+), 4 deletions(-)
Signature · 1 line
2.17.0
#1 lore raw

PATCH 1/3 xen/pvh: enable and set default MTRR type

On PVH MTRR is not initialized by the firmware (because there's no firmware), so the kernel is started with MTRR disabled which means all memory accesses are UC.

So far there have been no issues (ie: slowdowns) caused by this because PVH only supported DomU mode without passed-through devices, so Xen was using WB as the default memory type instead of UC.

Fix this by enabling MTRR and setting the default type to WB. Linux will use PAT to set the actual memory cache attributes.

---
Cc: Boris Ostrovsky <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: [email protected]

---
 arch/x86/xen/enlighten_pvh.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/xen/enlighten_pvh.c +3 -0
index aa1c6a6831a9..e039d1809809 100644
@@ -6,6 +6,7 @@
 #include <asm/io_apic.h>
 #include <asm/hypervisor.h>
 #include <asm/e820/api.h>
+#include <asm/mtrr.h>
 #include <asm/x86_init.h>
 
 #include <asm/xen/interface.h>
@@ -98,6 +99,8 @@ void __init xen_prepare_pvh(void)
 
 	xen_pvh = 1;
 
+	wrmsr_safe(MSR_MTRRdefType, 0x800 | MTRR_TYPE_WRBACK, 0);
+
 	msr = cpuid_ebx(xen_cpuid_base() + 2);
 	pfn = __pa(hypercall_page);
 	wrmsr_safe(msr, (u32)pfn, (u32)(pfn >> 32));
Signature · 7 lines
2.17.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

PATCH 1/3 xen/pvh: enable and set default MTRR type

In reply to Roger Pau Monne

On PVH MTRR is not initialized by the firmware (because there's no firmware), so the kernel is started with MTRR disabled which means all memory accesses are UC.

So far there have been no issues (ie: slowdowns) caused by this because PVH only supported DomU mode without passed-through devices, so Xen was using WB as the default memory type instead of UC.

Fix this by enabling MTRR and setting the default type to WB. Linux will use PAT to set the actual memory cache attributes.

---
Cc: Boris Ostrovsky <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: [email protected]

---
 arch/x86/xen/enlighten_pvh.c | 3 +++
 1 file changed, 3 insertions(+)
arch/x86/xen/enlighten_pvh.c +3 -0
index aa1c6a6831a9..e039d1809809 100644
@@ -6,6 +6,7 @@
 #include <asm/io_apic.h>
 #include <asm/hypervisor.h>
 #include <asm/e820/api.h>
+#include <asm/mtrr.h>
 #include <asm/x86_init.h>
 
 #include <asm/xen/interface.h>
@@ -98,6 +99,8 @@ void __init xen_prepare_pvh(void)
 
 	xen_pvh = 1;
 
+	wrmsr_safe(MSR_MTRRdefType, 0x800 | MTRR_TYPE_WRBACK, 0);
+
 	msr = cpuid_ebx(xen_cpuid_base() + 2);
 	pfn = __pa(hypercall_page);
 	wrmsr_safe(msr, (u32)pfn, (u32)(pfn >> 32));
Signature · 1 line
2.17.0
#3 lore raw

PATCH 2/3 xen/store: do not store local values in xen_start_info

In reply to Roger Pau Monne

There's no need to store the xenstore page or event channel in xen_start_info if they are locally initialized.

This also fixes PVH local xenstore initialization due to the lack of xen_start_info in that case.

---
Cc: Boris Ostrovsky <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: [email protected]

---
 drivers/xen/xenbus/xenbus_probe.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/xen/xenbus/xenbus_probe.c +2 -3
index ec9eb4fba59c..f2088838f690 100644
@@ -710,7 +710,7 @@ static int __init xenstored_local_init(void)
 	if (!page)
 		goto out_err;
 
-	xen_store_gfn = xen_start_info->store_mfn = virt_to_gfn((void *)page);
+	xen_store_gfn = virt_to_gfn((void *)page);
 
 	/* Next allocate a local port which xenstored can bind to */
 	alloc_unbound.dom        = DOMID_SELF;
@@ -722,8 +722,7 @@ static int __init xenstored_local_init(void)
 		goto out_err;
 
 	BUG_ON(err);
-	xen_store_evtchn = xen_start_info->store_evtchn =
-		alloc_unbound.port;
+	xen_store_evtchn = alloc_unbound.port;
 
 	return 0;
 
Signature · 7 lines
2.17.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#4 lore raw

PATCH 2/3 xen/store: do not store local values in xen_start_info

In reply to Roger Pau Monne

There's no need to store the xenstore page or event channel in xen_start_info if they are locally initialized.

This also fixes PVH local xenstore initialization due to the lack of xen_start_info in that case.

---
Cc: Boris Ostrovsky <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: [email protected]

---
 drivers/xen/xenbus/xenbus_probe.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
drivers/xen/xenbus/xenbus_probe.c +2 -3
index ec9eb4fba59c..f2088838f690 100644
@@ -710,7 +710,7 @@ static int __init xenstored_local_init(void)
 	if (!page)
 		goto out_err;
 
-	xen_store_gfn = xen_start_info->store_mfn = virt_to_gfn((void *)page);
+	xen_store_gfn = virt_to_gfn((void *)page);
 
 	/* Next allocate a local port which xenstored can bind to */
 	alloc_unbound.dom        = DOMID_SELF;
@@ -722,8 +722,7 @@ static int __init xenstored_local_init(void)
 		goto out_err;
 
 	BUG_ON(err);
-	xen_store_evtchn = xen_start_info->store_evtchn =
-		alloc_unbound.port;
+	xen_store_evtchn = alloc_unbound.port;
 
 	return 0;
 
Signature · 1 line
2.17.0
#5 lore raw

PATCH 3/3 xen: share start flags between PV and PVH

In reply to Roger Pau Monne

Use a global variable to store the start flags for both PV and PVH. This allows the xen_initial_domain macro to work properly on PVH.

---
Cc: Boris Ostrovsky <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: [email protected]

---
 arch/x86/xen/enlighten.c     | 7 +++++++
 arch/x86/xen/enlighten_pv.c  | 1 +
 arch/x86/xen/enlighten_pvh.c | 1 +
 include/xen/xen.h            | 4 +++-
 4 files changed, 12 insertions(+), 1 deletion(-)
arch/x86/xen/enlighten.c +7 -0
index c9081c6671f0..cdbd7b524133 100644
@@ -64,6 +64,13 @@ struct shared_info xen_dummy_shared_info;
 __read_mostly int xen_have_vector_callback;
 EXPORT_SYMBOL_GPL(xen_have_vector_callback);
 
+/*
+ * NB: needs to live in .data because it's used by xen_prepare_pvh which runs
+ * before clearing the bss.
+ */
+uint32_t xen_start_flags __attribute__((section(".data"))) = 0;
+EXPORT_SYMBOL_GPL(xen_start_flags);
+
 /*
  * Point at some empty memory to start with. We map the real shared_info
  * page as soon as fixmap is up and running.
arch/x86/xen/enlighten_pv.c +1 -0
index c36d23aa6c35..04a6914b8b85 100644
@@ -1227,6 +1227,7 @@ asmlinkage __visible void __init xen_start_kernel(void)
 		return;
 
 	xen_domain_type = XEN_PV_DOMAIN;
+	xen_start_flags = xen_start_info->flags;
 
 	xen_setup_features();
 
arch/x86/xen/enlighten_pvh.c +1 -0
index e039d1809809..2653eb9b5dd8 100644
@@ -98,6 +98,7 @@ void __init xen_prepare_pvh(void)
 	}
 
 	xen_pvh = 1;
+	xen_start_flags = pvh_start_info.flags;
 
 	wrmsr_safe(MSR_MTRRdefType, 0x800 | MTRR_TYPE_WRBACK, 0);
 
include/xen/xen.h +3 -1
index 9d4340c907d1..6b5bee7041db 100644
@@ -25,12 +25,14 @@ extern bool xen_pvh;
 #define xen_hvm_domain()	(xen_domain_type == XEN_HVM_DOMAIN)
 #define xen_pvh_domain()	(xen_pvh)
 
+extern uint32_t xen_start_flags;
+
 #ifdef CONFIG_XEN_DOM0
 #include <xen/interface/xen.h>
 #include <asm/xen/hypervisor.h>
 
 #define xen_initial_domain()	(xen_domain() && \
-				 xen_start_info && xen_start_info->flags & SIF_INITDOMAIN)
+				 (xen_start_flags & SIF_INITDOMAIN))
 #else  /* !CONFIG_XEN_DOM0 */
 #define xen_initial_domain()	(0)
 #endif	/* CONFIG_XEN_DOM0 */
Signature · 7 lines
2.17.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#6 lore raw

PATCH 3/3 xen: share start flags between PV and PVH

In reply to Roger Pau Monne

Use a global variable to store the start flags for both PV and PVH. This allows the xen_initial_domain macro to work properly on PVH.

---
Cc: Boris Ostrovsky <[email protected]>
Cc: Juergen Gross <[email protected]>
Cc: [email protected]

---
 arch/x86/xen/enlighten.c     | 7 +++++++
 arch/x86/xen/enlighten_pv.c  | 1 +
 arch/x86/xen/enlighten_pvh.c | 1 +
 include/xen/xen.h            | 4 +++-
 4 files changed, 12 insertions(+), 1 deletion(-)
arch/x86/xen/enlighten.c +7 -0
index c9081c6671f0..cdbd7b524133 100644
@@ -64,6 +64,13 @@ struct shared_info xen_dummy_shared_info;
 __read_mostly int xen_have_vector_callback;
 EXPORT_SYMBOL_GPL(xen_have_vector_callback);
 
+/*
+ * NB: needs to live in .data because it's used by xen_prepare_pvh which runs
+ * before clearing the bss.
+ */
+uint32_t xen_start_flags __attribute__((section(".data"))) = 0;
+EXPORT_SYMBOL_GPL(xen_start_flags);
+
 /*
  * Point at some empty memory to start with. We map the real shared_info
  * page as soon as fixmap is up and running.
arch/x86/xen/enlighten_pv.c +1 -0
index c36d23aa6c35..04a6914b8b85 100644
@@ -1227,6 +1227,7 @@ asmlinkage __visible void __init xen_start_kernel(void)
 		return;
 
 	xen_domain_type = XEN_PV_DOMAIN;
+	xen_start_flags = xen_start_info->flags;
 
 	xen_setup_features();
 
arch/x86/xen/enlighten_pvh.c +1 -0
index e039d1809809..2653eb9b5dd8 100644
@@ -98,6 +98,7 @@ void __init xen_prepare_pvh(void)
 	}
 
 	xen_pvh = 1;
+	xen_start_flags = pvh_start_info.flags;
 
 	wrmsr_safe(MSR_MTRRdefType, 0x800 | MTRR_TYPE_WRBACK, 0);
 
include/xen/xen.h +3 -1
index 9d4340c907d1..6b5bee7041db 100644
@@ -25,12 +25,14 @@ extern bool xen_pvh;
 #define xen_hvm_domain()	(xen_domain_type == XEN_HVM_DOMAIN)
 #define xen_pvh_domain()	(xen_pvh)
 
+extern uint32_t xen_start_flags;
+
 #ifdef CONFIG_XEN_DOM0
 #include <xen/interface/xen.h>
 #include <asm/xen/hypervisor.h>
 
 #define xen_initial_domain()	(xen_domain() && \
-				 xen_start_info && xen_start_info->flags & SIF_INITDOMAIN)
+				 (xen_start_flags & SIF_INITDOMAIN))
 #else  /* !CONFIG_XEN_DOM0 */
 #define xen_initial_domain()	(0)
 #endif	/* CONFIG_XEN_DOM0 */
Signature · 1 line
2.17.0
#7 lore raw

Re: [PATCH 3/3] xen: share start flags between PV and PVH

Thank you for the patch! Yet something to improve:

Hi Roger,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linus/master]
[also build test ERROR on v4.17-rc3 next-20180503]
[cannot apply to xen-tip/linux-next]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Roger-Pau-Monne/xen-pvh-Dom0-support/20180504-102629
config: arm64-defconfig (attached as .config)
compiler: aarch64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=arm64

All errors (new ones prefixed by >>):

In file included from arch/arm64/xen/../../arm/xen/enlighten.c:1:0:

>> include/xen/xen.h:28:8: error: unknown type name 'uint32_t'
    extern uint32_t xen_start_flags;
           ^~~~~~~~

vim +/uint32_t +28 include/xen/xen.h

    27	
  > 28	extern uint32_t xen_start_flags;
    29	

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#8 lore raw

Re: [PATCH 3/3] xen: share start flags between PV and PVH

In reply to Roger Pau Monne
Thank you for the patch! Yet something to improve:

Hi Roger,

Thank you for the patch! Yet something to improve:

[auto build test ERROR on linus/master]
[also build test ERROR on v4.17-rc3 next-20180503]
[cannot apply to xen-tip/linux-next]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Roger-Pau-Monne/xen-pvh-Dom0-support/20180504-102629
config: arm64-defconfig (attached as .config)
compiler: aarch64-linux-gnu-gcc (Debian 7.2.0-11) 7.2.0
reproduce:
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=arm64

All errors (new ones prefixed by >>):

In file included from arch/arm64/xen/../../arm/xen/enlighten.c:1:0:

>> include/xen/xen.h:28:8: error: unknown type name 'uint32_t'
    extern uint32_t xen_start_flags;
           ^~~~~~~~

vim +/uint32_t +28 include/xen/xen.h

    27	
  > 28	extern uint32_t xen_start_flags;
    29	

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel