Skip to contentSkip to conversation

xen/pvhvm: fix shared_info and pirq issues with kexec

RFC PATCH In review Vitaly Kuznetsov 5 participants 76 messages Jul 15, 2014 – Aug 4, 2014

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover xen/pvhvm: fix shared_info and pirq issues with kexec

With this patch series I'm trying to address several issues with kexec on pvhvm:
- shared_info issue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)

This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.

Olaf Hering (1):
xen PVonHVM: use E820_Reserved area for shared_info

Vitaly Kuznetsov (3):
xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown() xen/pvhvm: Unmap all PIRQs on startup and shutdown xen/pvhvm: Make MSI IRQs work after kexec

 arch/x86/pci/xen.c               |  3 +-
 arch/x86/xen/enlighten.c         | 83 +++++++++++++++++++++++++++++++---------
 arch/x86/xen/smp.c               | 10 +++++
 arch/x86/xen/smp.h               |  1 +
 arch/x86/xen/suspend.c           |  2 +-
 arch/x86/xen/xen-ops.h           |  2 +-
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 8 files changed, 158 insertions(+), 22 deletions(-)
Signature · 1 line
1.9.3
#1 lore raw

RFC PATCH 1/4 xen PVonHVM: use E820_Reserved area for shared_info

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

[On resume we need to reset the xen_vcpu_info, which the original patch did not do]

---
 arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------
 arch/x86/xen/suspend.c   |  2 +-
 arch/x86/xen/xen-ops.h   |  2 +-
 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
index ffb101e..a11af62 100644
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL
+static struct shared_info *xen_hvm_shared_info;
+static unsigned long xen_hvm_sip_phys;
+static int xen_major, xen_minor;
+
+static void xen_hvm_connect_shared_info(unsigned long pfn)
 {
-	int cpu;
 	struct xen_add_to_physmap xatp;
-	static struct shared_info *shared_info_page = 0;
 
-	if (!shared_info_page)
-		shared_info_page = (struct shared_info *)
-			extend_brk(PAGE_SIZE, PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
-	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	xatp.gpfn = pfn;
 	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
 		BUG();
 
-	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+}
+static void __init xen_hvm_set_shared_info(struct shared_info *sip)
+{
+	int cpu;
+
+	HYPERVISOR_shared_info = sip;
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void)
 	}
 }
 
-#ifdef CONFIG_XEN_PVHVM
+/* Reconnect the shared_info pfn to a (new) mfn */
+void xen_hvm_resume_shared_info(void)
+{
+	xen_hvm_connect_shared_info(xen_hvm_sip_phys >> PAGE_SHIFT);
+	xen_hvm_set_shared_info(xen_hvm_shared_info);
+}
+
+/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage.
+ * On these old tools the shared info page will be placed in E820_Ram.
+ * Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects
+ * that nothing is mapped up to HVM_SHARED_INFO_ADDR.
+ * Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used
+ * here for the shared info page. */
+static void __init xen_hvm_init_shared_info(void)
+{
+	if (xen_major < 4) {
+		xen_hvm_shared_info = extend_brk(PAGE_SIZE, PAGE_SIZE);
+		xen_hvm_sip_phys = __pa(xen_hvm_shared_info);
+	} else {
+		xen_hvm_sip_phys = HVM_SHARED_INFO_ADDR;
+		set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_hvm_sip_phys);
+		xen_hvm_shared_info =
+		(struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
+	}
+	xen_hvm_resume_shared_info();
+}
+
 static void __init init_hvm_pv_info(void)
 {
-	int major, minor;
-	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	uint32_t  ecx, edx, pages, msr, base;
 	u64 pfn;
 
 	base = xen_cpuid_base();
-	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
-
-	major = eax >> 16;
-	minor = eax & 0xffff;
-	printk(KERN_INFO "Xen version %d.%d.\n", major, minor);
-
 	cpuid(base + 2, &pages, &msr, &ecx, &edx);
 
 	pfn = __pa(hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;
 }
 
 bool xen_hvm_need_lapic(void)
arch/x86/xen/suspend.c +1 -1
index c4df9db..8d6793e 100644
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled)
 {
 #ifdef CONFIG_XEN_PVHVM
 	int cpu;
-	xen_hvm_init_shared_info();
+	xen_hvm_resume_shared_info();
 	xen_callback_vector();
 	xen_unplug_emulated_devices();
 	if (xen_feature(XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
index 97d8765..d083e82 100644
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
-void xen_hvm_init_shared_info(void);
+void xen_hvm_resume_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
Signature · 1 line
1.9.3
#2 lore raw

RFC PATCH 1/4 xen PVonHVM: use E820_Reserved area for shared_info

In reply to Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

[On resume we need to reset the xen_vcpu_info, which the original patch did not do]

---
 arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------
 arch/x86/xen/suspend.c   |  2 +-
 arch/x86/xen/xen-ops.h   |  2 +-
 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
index ffb101e..a11af62 100644
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL
+static struct shared_info *xen_hvm_shared_info;
+static unsigned long xen_hvm_sip_phys;
+static int xen_major, xen_minor;
+
+static void xen_hvm_connect_shared_info(unsigned long pfn)
 {
-	int cpu;
 	struct xen_add_to_physmap xatp;
-	static struct shared_info *shared_info_page = 0;
 
-	if (!shared_info_page)
-		shared_info_page = (struct shared_info *)
-			extend_brk(PAGE_SIZE, PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
-	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	xatp.gpfn = pfn;
 	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
 		BUG();
 
-	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+}
+static void __init xen_hvm_set_shared_info(struct shared_info *sip)
+{
+	int cpu;
+
+	HYPERVISOR_shared_info = sip;
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void)
 	}
 }
 
-#ifdef CONFIG_XEN_PVHVM
+/* Reconnect the shared_info pfn to a (new) mfn */
+void xen_hvm_resume_shared_info(void)
+{
+	xen_hvm_connect_shared_info(xen_hvm_sip_phys >> PAGE_SHIFT);
+	xen_hvm_set_shared_info(xen_hvm_shared_info);
+}
+
+/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage.
+ * On these old tools the shared info page will be placed in E820_Ram.
+ * Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects
+ * that nothing is mapped up to HVM_SHARED_INFO_ADDR.
+ * Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used
+ * here for the shared info page. */
+static void __init xen_hvm_init_shared_info(void)
+{
+	if (xen_major < 4) {
+		xen_hvm_shared_info = extend_brk(PAGE_SIZE, PAGE_SIZE);
+		xen_hvm_sip_phys = __pa(xen_hvm_shared_info);
+	} else {
+		xen_hvm_sip_phys = HVM_SHARED_INFO_ADDR;
+		set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_hvm_sip_phys);
+		xen_hvm_shared_info =
+		(struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
+	}
+	xen_hvm_resume_shared_info();
+}
+
 static void __init init_hvm_pv_info(void)
 {
-	int major, minor;
-	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	uint32_t  ecx, edx, pages, msr, base;
 	u64 pfn;
 
 	base = xen_cpuid_base();
-	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
-
-	major = eax >> 16;
-	minor = eax & 0xffff;
-	printk(KERN_INFO "Xen version %d.%d.\n", major, minor);
-
 	cpuid(base + 2, &pages, &msr, &ecx, &edx);
 
 	pfn = __pa(hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;
 }
 
 bool xen_hvm_need_lapic(void)
arch/x86/xen/suspend.c +1 -1
index c4df9db..8d6793e 100644
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled)
 {
 #ifdef CONFIG_XEN_PVHVM
 	int cpu;
-	xen_hvm_init_shared_info();
+	xen_hvm_resume_shared_info();
 	xen_callback_vector();
 	xen_unplug_emulated_devices();
 	if (xen_feature(XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
index 97d8765..d083e82 100644
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
-void xen_hvm_init_shared_info(void);
+void xen_hvm_resume_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
Signature · 1 line
1.9.3
#3 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Quoted 160 lines
(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

[On resume we need to reset the xen_vcpu_info, which the original patch did not do]

---
 arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------
 arch/x86/xen/suspend.c   |  2 +-
 arch/x86/xen/xen-ops.h   |  2 +-
 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
index ffb101e..a11af62 100644
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL
+static struct shared_info *xen_hvm_shared_info;
+static unsigned long xen_hvm_sip_phys;
+static int xen_major, xen_minor;
+
+static void xen_hvm_connect_shared_info(unsigned long pfn)
 {
-	int cpu;
 	struct xen_add_to_physmap xatp;
-	static struct shared_info *shared_info_page = 0;
 
-	if (!shared_info_page)
-		shared_info_page = (struct shared_info *)
-			extend_brk(PAGE_SIZE, PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
-	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	xatp.gpfn = pfn;
 	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
 		BUG();
 
-	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+}
+static void __init xen_hvm_set_shared_info(struct shared_info *sip)
+{
+	int cpu;
+
+	HYPERVISOR_shared_info = sip;
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void)
 	}
 }
 
-#ifdef CONFIG_XEN_PVHVM
+/* Reconnect the shared_info pfn to a (new) mfn */
+void xen_hvm_resume_shared_info(void)
+{
+	xen_hvm_connect_shared_info(xen_hvm_sip_phys >> PAGE_SHIFT);
+	xen_hvm_set_shared_info(xen_hvm_shared_info);
+}
+
+/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage.
+ * On these old tools the shared info page will be placed in E820_Ram.
+ * Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects
+ * that nothing is mapped up to HVM_SHARED_INFO_ADDR.
+ * Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used
+ * here for the shared info page. */
+static void __init xen_hvm_init_shared_info(void)
+{
+	if (xen_major < 4) {
+		xen_hvm_shared_info = extend_brk(PAGE_SIZE, PAGE_SIZE);
+		xen_hvm_sip_phys = __pa(xen_hvm_shared_info);
+	} else {
+		xen_hvm_sip_phys = HVM_SHARED_INFO_ADDR;
+		set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_hvm_sip_phys);
+		xen_hvm_shared_info =
+		(struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
+	}
+	xen_hvm_resume_shared_info();
+}
+
 static void __init init_hvm_pv_info(void)
 {
-	int major, minor;
-	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	uint32_t  ecx, edx, pages, msr, base;
 	u64 pfn;
 
 	base = xen_cpuid_base();
-	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
-
-	major = eax >> 16;
-	minor = eax & 0xffff;
-	printk(KERN_INFO "Xen version %d.%d.\n", major, minor);
-
 	cpuid(base + 2, &pages, &msr, &ecx, &edx);
 
 	pfn = __pa(hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;
 }
 
 bool xen_hvm_need_lapic(void)
arch/x86/xen/suspend.c +1 -1
index c4df9db..8d6793e 100644
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled)
 {
 #ifdef CONFIG_XEN_PVHVM
 	int cpu;
-	xen_hvm_init_shared_info();
+	xen_hvm_resume_shared_info();
 	xen_callback_vector();
 	xen_unplug_emulated_devices();
 	if (xen_feature(XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
index 97d8765..d083e82 100644
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
-void xen_hvm_init_shared_info(void);
+void xen_hvm_resume_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
Signature · 1 line
1.9.3
#4 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

In reply to Vitaly Kuznetsov
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Quoted 160 lines
(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

[On resume we need to reset the xen_vcpu_info, which the original patch did not do]

---
 arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------
 arch/x86/xen/suspend.c   |  2 +-
 arch/x86/xen/xen-ops.h   |  2 +-
 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
index ffb101e..a11af62 100644
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL
+static struct shared_info *xen_hvm_shared_info;
+static unsigned long xen_hvm_sip_phys;
+static int xen_major, xen_minor;
+
+static void xen_hvm_connect_shared_info(unsigned long pfn)
 {
-	int cpu;
 	struct xen_add_to_physmap xatp;
-	static struct shared_info *shared_info_page = 0;
 
-	if (!shared_info_page)
-		shared_info_page = (struct shared_info *)
-			extend_brk(PAGE_SIZE, PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
-	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	xatp.gpfn = pfn;
 	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
 		BUG();
 
-	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+}
+static void __init xen_hvm_set_shared_info(struct shared_info *sip)
+{
+	int cpu;
+
+	HYPERVISOR_shared_info = sip;
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void)
 	}
 }
 
-#ifdef CONFIG_XEN_PVHVM
+/* Reconnect the shared_info pfn to a (new) mfn */
+void xen_hvm_resume_shared_info(void)
+{
+	xen_hvm_connect_shared_info(xen_hvm_sip_phys >> PAGE_SHIFT);
+	xen_hvm_set_shared_info(xen_hvm_shared_info);
+}
+
+/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage.
+ * On these old tools the shared info page will be placed in E820_Ram.
+ * Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects
+ * that nothing is mapped up to HVM_SHARED_INFO_ADDR.
+ * Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used
+ * here for the shared info page. */
+static void __init xen_hvm_init_shared_info(void)
+{
+	if (xen_major < 4) {
+		xen_hvm_shared_info = extend_brk(PAGE_SIZE, PAGE_SIZE);
+		xen_hvm_sip_phys = __pa(xen_hvm_shared_info);
+	} else {
+		xen_hvm_sip_phys = HVM_SHARED_INFO_ADDR;
+		set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_hvm_sip_phys);
+		xen_hvm_shared_info =
+		(struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
+	}
+	xen_hvm_resume_shared_info();
+}
+
 static void __init init_hvm_pv_info(void)
 {
-	int major, minor;
-	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	uint32_t  ecx, edx, pages, msr, base;
 	u64 pfn;
 
 	base = xen_cpuid_base();
-	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
-
-	major = eax >> 16;
-	minor = eax & 0xffff;
-	printk(KERN_INFO "Xen version %d.%d.\n", major, minor);
-
 	cpuid(base + 2, &pages, &msr, &ecx, &edx);
 
 	pfn = __pa(hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;
 }
 
 bool xen_hvm_need_lapic(void)
arch/x86/xen/suspend.c +1 -1
index c4df9db..8d6793e 100644
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled)
 {
 #ifdef CONFIG_XEN_PVHVM
 	int cpu;
-	xen_hvm_init_shared_info();
+	xen_hvm_resume_shared_info();
 	xen_callback_vector();
 	xen_unplug_emulated_devices();
 	if (xen_feature(XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
index 97d8765..d083e82 100644
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
-void xen_hvm_init_shared_info(void);
+void xen_hvm_resume_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
Signature · 1 line
1.9.3
#5 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

Quoted 9 lines

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.

Quoted 160 lines
Quoted 160 lines
(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

[On resume we need to reset the xen_vcpu_info, which the original patch did not do]

---
 arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------
 arch/x86/xen/suspend.c   |  2 +-
 arch/x86/xen/xen-ops.h   |  2 +-
 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
index ffb101e..a11af62 100644
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL
+static struct shared_info *xen_hvm_shared_info;
+static unsigned long xen_hvm_sip_phys;
+static int xen_major, xen_minor;
+
+static void xen_hvm_connect_shared_info(unsigned long pfn)
 {
-	int cpu;
 	struct xen_add_to_physmap xatp;
-	static struct shared_info *shared_info_page = 0;
 
-	if (!shared_info_page)
-		shared_info_page = (struct shared_info *)
-			extend_brk(PAGE_SIZE, PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
-	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	xatp.gpfn = pfn;
 	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
 		BUG();
 
-	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+}
+static void __init xen_hvm_set_shared_info(struct shared_info *sip)
+{
+	int cpu;
+
+	HYPERVISOR_shared_info = sip;
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void)
 	}
 }
 
-#ifdef CONFIG_XEN_PVHVM
+/* Reconnect the shared_info pfn to a (new) mfn */
+void xen_hvm_resume_shared_info(void)
+{
+	xen_hvm_connect_shared_info(xen_hvm_sip_phys >> PAGE_SHIFT);
+	xen_hvm_set_shared_info(xen_hvm_shared_info);
+}
+
+/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage.
+ * On these old tools the shared info page will be placed in E820_Ram.
+ * Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects
+ * that nothing is mapped up to HVM_SHARED_INFO_ADDR.
+ * Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used
+ * here for the shared info page. */
+static void __init xen_hvm_init_shared_info(void)
+{
+	if (xen_major < 4) {
+		xen_hvm_shared_info = extend_brk(PAGE_SIZE, PAGE_SIZE);
+		xen_hvm_sip_phys = __pa(xen_hvm_shared_info);
+	} else {
+		xen_hvm_sip_phys = HVM_SHARED_INFO_ADDR;
+		set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_hvm_sip_phys);
+		xen_hvm_shared_info =
+		(struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
+	}
+	xen_hvm_resume_shared_info();
+}
+
 static void __init init_hvm_pv_info(void)
 {
-	int major, minor;
-	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	uint32_t  ecx, edx, pages, msr, base;
 	u64 pfn;
 
 	base = xen_cpuid_base();
-	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
-
-	major = eax >> 16;
-	minor = eax & 0xffff;
-	printk(KERN_INFO "Xen version %d.%d.\n", major, minor);
-
 	cpuid(base + 2, &pages, &msr, &ecx, &edx);
 
 	pfn = __pa(hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;
 }
 
 bool xen_hvm_need_lapic(void)
arch/x86/xen/suspend.c +1 -1
index c4df9db..8d6793e 100644
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled)
 {
 #ifdef CONFIG_XEN_PVHVM
 	int cpu;
-	xen_hvm_init_shared_info();
+	xen_hvm_resume_shared_info();
 	xen_callback_vector();
 	xen_unplug_emulated_devices();
 	if (xen_feature(XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
index 97d8765..d083e82 100644
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
-void xen_hvm_init_shared_info(void);
+void xen_hvm_resume_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#6 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

In reply to Konrad Rzeszutek Wilk
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

Quoted 9 lines

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.

Quoted 160 lines
Quoted 160 lines
(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

[On resume we need to reset the xen_vcpu_info, which the original patch did not do]

---
 arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------
 arch/x86/xen/suspend.c   |  2 +-
 arch/x86/xen/xen-ops.h   |  2 +-
 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
index ffb101e..a11af62 100644
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL
+static struct shared_info *xen_hvm_shared_info;
+static unsigned long xen_hvm_sip_phys;
+static int xen_major, xen_minor;
+
+static void xen_hvm_connect_shared_info(unsigned long pfn)
 {
-	int cpu;
 	struct xen_add_to_physmap xatp;
-	static struct shared_info *shared_info_page = 0;
 
-	if (!shared_info_page)
-		shared_info_page = (struct shared_info *)
-			extend_brk(PAGE_SIZE, PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
-	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	xatp.gpfn = pfn;
 	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
 		BUG();
 
-	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+}
+static void __init xen_hvm_set_shared_info(struct shared_info *sip)
+{
+	int cpu;
+
+	HYPERVISOR_shared_info = sip;
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void)
 	}
 }
 
-#ifdef CONFIG_XEN_PVHVM
+/* Reconnect the shared_info pfn to a (new) mfn */
+void xen_hvm_resume_shared_info(void)
+{
+	xen_hvm_connect_shared_info(xen_hvm_sip_phys >> PAGE_SHIFT);
+	xen_hvm_set_shared_info(xen_hvm_shared_info);
+}
+
+/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage.
+ * On these old tools the shared info page will be placed in E820_Ram.
+ * Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects
+ * that nothing is mapped up to HVM_SHARED_INFO_ADDR.
+ * Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used
+ * here for the shared info page. */
+static void __init xen_hvm_init_shared_info(void)
+{
+	if (xen_major < 4) {
+		xen_hvm_shared_info = extend_brk(PAGE_SIZE, PAGE_SIZE);
+		xen_hvm_sip_phys = __pa(xen_hvm_shared_info);
+	} else {
+		xen_hvm_sip_phys = HVM_SHARED_INFO_ADDR;
+		set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_hvm_sip_phys);
+		xen_hvm_shared_info =
+		(struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
+	}
+	xen_hvm_resume_shared_info();
+}
+
 static void __init init_hvm_pv_info(void)
 {
-	int major, minor;
-	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	uint32_t  ecx, edx, pages, msr, base;
 	u64 pfn;
 
 	base = xen_cpuid_base();
-	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
-
-	major = eax >> 16;
-	minor = eax & 0xffff;
-	printk(KERN_INFO "Xen version %d.%d.\n", major, minor);
-
 	cpuid(base + 2, &pages, &msr, &ecx, &edx);
 
 	pfn = __pa(hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;
 }
 
 bool xen_hvm_need_lapic(void)
arch/x86/xen/suspend.c +1 -1
index c4df9db..8d6793e 100644
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled)
 {
 #ifdef CONFIG_XEN_PVHVM
 	int cpu;
-	xen_hvm_init_shared_info();
+	xen_hvm_resume_shared_info();
 	xen_callback_vector();
 	xen_unplug_emulated_devices();
 	if (xen_feature(XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
index 97d8765..d083e82 100644
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
-void xen_hvm_init_shared_info(void);
+void xen_hvm_resume_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#7 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

Quoted 40 lines · Vitaly Kuznetsov
Quoted 38 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Quoted 8 lines
Quoted 3 lines

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.

Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.

Thank you!

Quoted 164 lines
Quoted 161 lines
Quoted 161 lines
(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

[On resume we need to reset the xen_vcpu_info, which the original patch did not do]

---
 arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------
 arch/x86/xen/suspend.c   |  2 +-
 arch/x86/xen/xen-ops.h   |  2 +-
 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
index ffb101e..a11af62 100644
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL
+static struct shared_info *xen_hvm_shared_info;
+static unsigned long xen_hvm_sip_phys;
+static int xen_major, xen_minor;
+
+static void xen_hvm_connect_shared_info(unsigned long pfn)
 {
-	int cpu;
 	struct xen_add_to_physmap xatp;
-	static struct shared_info *shared_info_page = 0;
 
-	if (!shared_info_page)
-		shared_info_page = (struct shared_info *)
-			extend_brk(PAGE_SIZE, PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
-	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	xatp.gpfn = pfn;
 	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
 		BUG();
 
-	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+}
+static void __init xen_hvm_set_shared_info(struct shared_info *sip)
+{
+	int cpu;
+
+	HYPERVISOR_shared_info = sip;
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void)
 	}
 }
 
-#ifdef CONFIG_XEN_PVHVM
+/* Reconnect the shared_info pfn to a (new) mfn */
+void xen_hvm_resume_shared_info(void)
+{
+	xen_hvm_connect_shared_info(xen_hvm_sip_phys >> PAGE_SHIFT);
+	xen_hvm_set_shared_info(xen_hvm_shared_info);
+}
+
+/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage.
+ * On these old tools the shared info page will be placed in E820_Ram.
+ * Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects
+ * that nothing is mapped up to HVM_SHARED_INFO_ADDR.
+ * Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used
+ * here for the shared info page. */
+static void __init xen_hvm_init_shared_info(void)
+{
+	if (xen_major < 4) {
+		xen_hvm_shared_info = extend_brk(PAGE_SIZE, PAGE_SIZE);
+		xen_hvm_sip_phys = __pa(xen_hvm_shared_info);
+	} else {
+		xen_hvm_sip_phys = HVM_SHARED_INFO_ADDR;
+		set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_hvm_sip_phys);
+		xen_hvm_shared_info =
+		(struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
+	}
+	xen_hvm_resume_shared_info();
+}
+
 static void __init init_hvm_pv_info(void)
 {
-	int major, minor;
-	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	uint32_t  ecx, edx, pages, msr, base;
 	u64 pfn;
 
 	base = xen_cpuid_base();
-	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
-
-	major = eax >> 16;
-	minor = eax & 0xffff;
-	printk(KERN_INFO "Xen version %d.%d.\n", major, minor);
-
 	cpuid(base + 2, &pages, &msr, &ecx, &edx);
 
 	pfn = __pa(hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;
 }
 
 bool xen_hvm_need_lapic(void)
arch/x86/xen/suspend.c +1 -1
index c4df9db..8d6793e 100644
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled)
 {
 #ifdef CONFIG_XEN_PVHVM
 	int cpu;
-	xen_hvm_init_shared_info();
+	xen_hvm_resume_shared_info();
 	xen_callback_vector();
 	xen_unplug_emulated_devices();
 	if (xen_feature(XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
index 97d8765..d083e82 100644
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
-void xen_hvm_init_shared_info(void);
+void xen_hvm_resume_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#8 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

In reply to Vitaly Kuznetsov
Quoted 40 lines · Vitaly Kuznetsov
Quoted 38 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Quoted 8 lines
Quoted 3 lines

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.

Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.

Thank you!

Quoted 164 lines
Quoted 161 lines
Quoted 161 lines
(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

[On resume we need to reset the xen_vcpu_info, which the original patch did not do]

---
 arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------
 arch/x86/xen/suspend.c   |  2 +-
 arch/x86/xen/xen-ops.h   |  2 +-
 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
index ffb101e..a11af62 100644
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL
+static struct shared_info *xen_hvm_shared_info;
+static unsigned long xen_hvm_sip_phys;
+static int xen_major, xen_minor;
+
+static void xen_hvm_connect_shared_info(unsigned long pfn)
 {
-	int cpu;
 	struct xen_add_to_physmap xatp;
-	static struct shared_info *shared_info_page = 0;
 
-	if (!shared_info_page)
-		shared_info_page = (struct shared_info *)
-			extend_brk(PAGE_SIZE, PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
-	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	xatp.gpfn = pfn;
 	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
 		BUG();
 
-	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+}
+static void __init xen_hvm_set_shared_info(struct shared_info *sip)
+{
+	int cpu;
+
+	HYPERVISOR_shared_info = sip;
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void)
 	}
 }
 
-#ifdef CONFIG_XEN_PVHVM
+/* Reconnect the shared_info pfn to a (new) mfn */
+void xen_hvm_resume_shared_info(void)
+{
+	xen_hvm_connect_shared_info(xen_hvm_sip_phys >> PAGE_SHIFT);
+	xen_hvm_set_shared_info(xen_hvm_shared_info);
+}
+
+/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage.
+ * On these old tools the shared info page will be placed in E820_Ram.
+ * Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects
+ * that nothing is mapped up to HVM_SHARED_INFO_ADDR.
+ * Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used
+ * here for the shared info page. */
+static void __init xen_hvm_init_shared_info(void)
+{
+	if (xen_major < 4) {
+		xen_hvm_shared_info = extend_brk(PAGE_SIZE, PAGE_SIZE);
+		xen_hvm_sip_phys = __pa(xen_hvm_shared_info);
+	} else {
+		xen_hvm_sip_phys = HVM_SHARED_INFO_ADDR;
+		set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_hvm_sip_phys);
+		xen_hvm_shared_info =
+		(struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
+	}
+	xen_hvm_resume_shared_info();
+}
+
 static void __init init_hvm_pv_info(void)
 {
-	int major, minor;
-	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	uint32_t  ecx, edx, pages, msr, base;
 	u64 pfn;
 
 	base = xen_cpuid_base();
-	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
-
-	major = eax >> 16;
-	minor = eax & 0xffff;
-	printk(KERN_INFO "Xen version %d.%d.\n", major, minor);
-
 	cpuid(base + 2, &pages, &msr, &ecx, &edx);
 
 	pfn = __pa(hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;
 }
 
 bool xen_hvm_need_lapic(void)
arch/x86/xen/suspend.c +1 -1
index c4df9db..8d6793e 100644
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled)
 {
 #ifdef CONFIG_XEN_PVHVM
 	int cpu;
-	xen_hvm_init_shared_info();
+	xen_hvm_resume_shared_info();
 	xen_callback_vector();
 	xen_unplug_emulated_devices();
 	if (xen_feature(XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
index 97d8765..d083e82 100644
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
-void xen_hvm_init_shared_info(void);
+void xen_hvm_resume_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#9 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

Quoted 45 lines · Konrad Rzeszutek Wilk
Quoted 44 lines · Vitaly Kuznetsov
Quoted 41 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Quoted 8 lines
Quoted 4 lines

Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.

Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.

The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce that on old patch version and everything works fine with your fix (calling xen_hvm_set_shared_info() in addition to xen_hvm_connect_shared_info() on resume). We're probably safe to apply it now, thanks!

However I'd like to suggest we remove '__init' from xen_hvm_set_shared_info() as now we call it on resume.

Quoted 165 lines

Thank you!

Quoted 164 lines
Quoted 161 lines
Quoted 161 lines
(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

[On resume we need to reset the xen_vcpu_info, which the original patch did not do]

---
 arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------
 arch/x86/xen/suspend.c   |  2 +-
 arch/x86/xen/xen-ops.h   |  2 +-
 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
index ffb101e..a11af62 100644
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL
+static struct shared_info *xen_hvm_shared_info;
+static unsigned long xen_hvm_sip_phys;
+static int xen_major, xen_minor;
+
+static void xen_hvm_connect_shared_info(unsigned long pfn)
 {
-	int cpu;
 	struct xen_add_to_physmap xatp;
-	static struct shared_info *shared_info_page = 0;
 
-	if (!shared_info_page)
-		shared_info_page = (struct shared_info *)
-			extend_brk(PAGE_SIZE, PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
-	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	xatp.gpfn = pfn;
 	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
 		BUG();
 
-	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+}
+static void __init xen_hvm_set_shared_info(struct shared_info *sip)
+{
+	int cpu;
+
+	HYPERVISOR_shared_info = sip;
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void)
 	}
 }
 
-#ifdef CONFIG_XEN_PVHVM
+/* Reconnect the shared_info pfn to a (new) mfn */
+void xen_hvm_resume_shared_info(void)
+{
+	xen_hvm_connect_shared_info(xen_hvm_sip_phys >> PAGE_SHIFT);
+	xen_hvm_set_shared_info(xen_hvm_shared_info);
+}
+
+/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage.
+ * On these old tools the shared info page will be placed in E820_Ram.
+ * Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects
+ * that nothing is mapped up to HVM_SHARED_INFO_ADDR.
+ * Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used
+ * here for the shared info page. */
+static void __init xen_hvm_init_shared_info(void)
+{
+	if (xen_major < 4) {
+		xen_hvm_shared_info = extend_brk(PAGE_SIZE, PAGE_SIZE);
+		xen_hvm_sip_phys = __pa(xen_hvm_shared_info);
+	} else {
+		xen_hvm_sip_phys = HVM_SHARED_INFO_ADDR;
+		set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_hvm_sip_phys);
+		xen_hvm_shared_info =
+		(struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
+	}
+	xen_hvm_resume_shared_info();
+}
+
 static void __init init_hvm_pv_info(void)
 {
-	int major, minor;
-	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	uint32_t  ecx, edx, pages, msr, base;
 	u64 pfn;
 
 	base = xen_cpuid_base();
-	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
-
-	major = eax >> 16;
-	minor = eax & 0xffff;
-	printk(KERN_INFO "Xen version %d.%d.\n", major, minor);
-
 	cpuid(base + 2, &pages, &msr, &ecx, &edx);
 
 	pfn = __pa(hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;
 }
 
 bool xen_hvm_need_lapic(void)
arch/x86/xen/suspend.c +1 -1
index c4df9db..8d6793e 100644
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled)
 {
 #ifdef CONFIG_XEN_PVHVM
 	int cpu;
-	xen_hvm_init_shared_info();
+	xen_hvm_resume_shared_info();
 	xen_callback_vector();
 	xen_unplug_emulated_devices();
 	if (xen_feature(XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
index 97d8765..d083e82 100644
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
-void xen_hvm_init_shared_info(void);
+void xen_hvm_resume_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
#10 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

In reply to Konrad Rzeszutek Wilk
Quoted 45 lines · Konrad Rzeszutek Wilk
Quoted 44 lines · Vitaly Kuznetsov
Quoted 41 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Quoted 8 lines
Quoted 4 lines

Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.

Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.

The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce that on old patch version and everything works fine with your fix (calling xen_hvm_set_shared_info() in addition to xen_hvm_connect_shared_info() on resume). We're probably safe to apply it now, thanks!

However I'd like to suggest we remove '__init' from xen_hvm_set_shared_info() as now we call it on resume.

Quoted 165 lines

Thank you!

Quoted 164 lines
Quoted 161 lines
Quoted 161 lines
(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

[On resume we need to reset the xen_vcpu_info, which the original patch did not do]

---
 arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------
 arch/x86/xen/suspend.c   |  2 +-
 arch/x86/xen/xen-ops.h   |  2 +-
 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
index ffb101e..a11af62 100644
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL
+static struct shared_info *xen_hvm_shared_info;
+static unsigned long xen_hvm_sip_phys;
+static int xen_major, xen_minor;
+
+static void xen_hvm_connect_shared_info(unsigned long pfn)
 {
-	int cpu;
 	struct xen_add_to_physmap xatp;
-	static struct shared_info *shared_info_page = 0;
 
-	if (!shared_info_page)
-		shared_info_page = (struct shared_info *)
-			extend_brk(PAGE_SIZE, PAGE_SIZE);
 	xatp.domid = DOMID_SELF;
 	xatp.idx = 0;
 	xatp.space = XENMAPSPACE_shared_info;
-	xatp.gpfn = __pa(shared_info_page) >> PAGE_SHIFT;
+	xatp.gpfn = pfn;
 	if (HYPERVISOR_memory_op(XENMEM_add_to_physmap, &xatp))
 		BUG();
 
-	HYPERVISOR_shared_info = (struct shared_info *)shared_info_page;
+}
+static void __init xen_hvm_set_shared_info(struct shared_info *sip)
+{
+	int cpu;
+
+	HYPERVISOR_shared_info = sip;
 
 	/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info
 	 * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void)
 	}
 }
 
-#ifdef CONFIG_XEN_PVHVM
+/* Reconnect the shared_info pfn to a (new) mfn */
+void xen_hvm_resume_shared_info(void)
+{
+	xen_hvm_connect_shared_info(xen_hvm_sip_phys >> PAGE_SHIFT);
+	xen_hvm_set_shared_info(xen_hvm_shared_info);
+}
+
+/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage.
+ * On these old tools the shared info page will be placed in E820_Ram.
+ * Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects
+ * that nothing is mapped up to HVM_SHARED_INFO_ADDR.
+ * Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used
+ * here for the shared info page. */
+static void __init xen_hvm_init_shared_info(void)
+{
+	if (xen_major < 4) {
+		xen_hvm_shared_info = extend_brk(PAGE_SIZE, PAGE_SIZE);
+		xen_hvm_sip_phys = __pa(xen_hvm_shared_info);
+	} else {
+		xen_hvm_sip_phys = HVM_SHARED_INFO_ADDR;
+		set_fixmap(FIX_PARAVIRT_BOOTMAP, xen_hvm_sip_phys);
+		xen_hvm_shared_info =
+		(struct shared_info *)fix_to_virt(FIX_PARAVIRT_BOOTMAP);
+	}
+	xen_hvm_resume_shared_info();
+}
+
 static void __init init_hvm_pv_info(void)
 {
-	int major, minor;
-	uint32_t eax, ebx, ecx, edx, pages, msr, base;
+	uint32_t  ecx, edx, pages, msr, base;
 	u64 pfn;
 
 	base = xen_cpuid_base();
-	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
-
-	major = eax >> 16;
-	minor = eax & 0xffff;
-	printk(KERN_INFO "Xen version %d.%d.\n", major, minor);
-
 	cpuid(base + 2, &pages, &msr, &ecx, &edx);
 
 	pfn = __pa(hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;
 }
 
 bool xen_hvm_need_lapic(void)
arch/x86/xen/suspend.c +1 -1
index c4df9db..8d6793e 100644
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled)
 {
 #ifdef CONFIG_XEN_PVHVM
 	int cpu;
-	xen_hvm_init_shared_info();
+	xen_hvm_resume_shared_info();
 	xen_callback_vector();
 	xen_unplug_emulated_devices();
 	if (xen_feature(XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
index 97d8765..d083e82 100644
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);
 void xen_vcpu_restore(void);
 
 void xen_callback_vector(void);
-void xen_hvm_init_shared_info(void);
+void xen_hvm_resume_shared_info(void);
 void xen_unplug_emulated_devices(void);
 
 void __init xen_build_dynamic_phys_to_machine(void);
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
#11 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

Quoted 52 lines · Vitaly Kuznetsov
Quoted 50 lines · Konrad Rzeszutek Wilk
Quoted 48 lines · Vitaly Kuznetsov
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

Quoted 9 lines

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.

Quoted 6 lines
Quoted 3 lines

Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.

The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce

Yes, that is it.

Quoted 4 lines

that on old patch version and everything works fine with your fix (calling xen_hvm_set_shared_info() in addition to xen_hvm_connect_shared_info() on resume). We're probably safe to apply it now, thanks!

Woot! Could you include that tidbit of information in the commit please?

Quoted 3 lines

However I'd like to suggest we remove '__init' from xen_hvm_set_shared_info() as now we call it on resume.

Good idea.

Lets wait until Stefano responds (for the MSI PIRQ one), and if he does not have anything special to say, then repost the whole patchset including this tiny __init fix and the updated comment?

#12 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

In reply to Vitaly Kuznetsov
Quoted 52 lines · Vitaly Kuznetsov
Quoted 50 lines · Konrad Rzeszutek Wilk
Quoted 48 lines · Vitaly Kuznetsov
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

Quoted 9 lines

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.

Quoted 6 lines
Quoted 3 lines

Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.

The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce

Yes, that is it.

Quoted 4 lines

that on old patch version and everything works fine with your fix (calling xen_hvm_set_shared_info() in addition to xen_hvm_connect_shared_info() on resume). We're probably safe to apply it now, thanks!

Woot! Could you include that tidbit of information in the commit please?

Quoted 3 lines

However I'd like to suggest we remove '__init' from xen_hvm_set_shared_info() as now we call it on resume.

Good idea.

Lets wait until Stefano responds (for the MSI PIRQ one), and if he does not have anything special to say, then repost the whole patchset including this tiny __init fix and the updated comment?

#13 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

Quoted 60 lines · Konrad Rzeszutek Wilk
Quoted 52 lines · Vitaly Kuznetsov
Quoted 50 lines · Konrad Rzeszutek Wilk
Quoted 48 lines · Vitaly Kuznetsov
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

Quoted 9 lines

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.

Quoted 6 lines
Quoted 3 lines

Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.

The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce

Quoted 9 lines

Yes, that is it.

Quoted 4 lines

that on old patch version and everything works fine with your fix (calling xen_hvm_set_shared_info() in addition to xen_hvm_connect_shared_info() on resume). We're probably safe to apply it now, thanks!

Woot! Could you include that tidbit of information in the commit please?

Sure,

Quoted 10 lines
Quoted 3 lines

However I'd like to suggest we remove '__init' from xen_hvm_set_shared_info() as now we call it on resume.

Good idea.

Lets wait until Stefano responds (for the MSI PIRQ one), and if he does not have anything special to say, then repost the whole patchset including this tiny __init fix and the updated comment?

Deal :-) Please take a look at my '[PATCH RFC] evtchn: introduce EVTCHNOP_fifo_destroy hypercall'. In case that works we can fix FIFO case at the same time, no TODO required.

I'll be able to return to this work at the end of next week.

Signature · 1 line
  Vitaly
#14 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

In reply to Konrad Rzeszutek Wilk
Quoted 60 lines · Konrad Rzeszutek Wilk
Quoted 52 lines · Vitaly Kuznetsov
Quoted 50 lines · Konrad Rzeszutek Wilk
Quoted 48 lines · Vitaly Kuznetsov
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 3 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9

This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.

Quoted 9 lines

We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.

Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?

Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.

Quoted 6 lines
Quoted 3 lines

Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.

The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce

Quoted 9 lines

Yes, that is it.

Quoted 4 lines

that on old patch version and everything works fine with your fix (calling xen_hvm_set_shared_info() in addition to xen_hvm_connect_shared_info() on resume). We're probably safe to apply it now, thanks!

Woot! Could you include that tidbit of information in the commit please?

Sure,

Quoted 10 lines
Quoted 3 lines

However I'd like to suggest we remove '__init' from xen_hvm_set_shared_info() as now we call it on resume.

Good idea.

Lets wait until Stefano responds (for the MSI PIRQ one), and if he does not have anything special to say, then repost the whole patchset including this tiny __init fix and the updated comment?

Deal :-) Please take a look at my '[PATCH RFC] evtchn: introduce EVTCHNOP_fifo_destroy hypercall'. In case that works we can fix FIFO case at the same time, no TODO required.

I'll be able to return to this work at the end of next week.

Signature · 1 line
  Vitaly
#15 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

In reply to Vitaly Kuznetsov
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 7 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

Is this a useful commit to give? Isn't it one of Konrad's random trees?

arch/x86/xen/enlighten.c asmlinkage __visible void __init xen_start_kernel(void) · line 1730

Quoted 9 lines
arch/x86/xen/enlighten.c +2 -1
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL

I would like clearer documentation for this magic address. Perhaps referring to some toolstack documentation?

Quoted 12 lines
arch/x86/xen/enlighten.c +4 -1
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)

arch/x86/xen/enlighten.c static void __init xen_hvm_guest_init(void) · line 1872

Quoted 10 lines
arch/x86/xen/enlighten.c +10 -0
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;

You appear to have changed to return value here? Did you mean to return base here?

David

#16 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

In reply to Vitaly Kuznetsov
Quoted 22 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

Quoted 7 lines

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

Is this a useful commit to give? Isn't it one of Konrad's random trees?

arch/x86/xen/enlighten.c asmlinkage __visible void __init xen_start_kernel(void) · line 1730

Quoted 9 lines
arch/x86/xen/enlighten.c +2 -1
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)
 #endif
 }
 
-void __ref xen_hvm_init_shared_info(void)
+#ifdef CONFIG_XEN_PVHVM
+#define HVM_SHARED_INFO_ADDR 0xFE700000UL

I would like clearer documentation for this magic address. Perhaps referring to some toolstack documentation?

Quoted 12 lines
arch/x86/xen/enlighten.c +4 -1
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)
 
 static uint32_t __init xen_hvm_platform(void)
 {
+	uint32_t eax, ebx, ecx, edx, base;
+
 	if (xen_pv_domain())
 		return 0;
 
-	return xen_cpuid_base();
+	base = xen_cpuid_base();
+	if (!base)

arch/x86/xen/enlighten.c static void __init xen_hvm_guest_init(void) · line 1872

Quoted 10 lines
arch/x86/xen/enlighten.c +10 -0
+		return 0;
+
+	cpuid(base + 1, &eax, &ebx, &ecx, &edx);
+
+	xen_major = eax >> 16;
+	xen_minor = eax & 0xffff;
+
+	printk(KERN_INFO "Xen version %d.%d.\n", xen_major, xen_minor);
+
+	return 1;

You appear to have changed to return value here? Did you mean to return base here?

David

#17 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

Quoted 27 lines · David Vrabel
Quoted 26 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

Quoted 5 lines
Quoted 3 lines
(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

Is this a useful commit to give? Isn't it one of Konrad's random trees?

Actually it is from upstream. That patch was reverted at certain point.

Perhaps just mention that it was added in, then reverted, and now after a year or so, added back in with the fix.

#18 lore raw

Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info

In reply to David Vrabel
Quoted 27 lines · David Vrabel
Quoted 26 lines · Vitaly Kuznetsov

From: Olaf Hering <[email protected]>

This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").

Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.

The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.

The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.

In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.

Quoted 5 lines
Quoted 3 lines
(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)

Is this a useful commit to give? Isn't it one of Konrad's random trees?

Actually it is from upstream. That patch was reverted at certain point.

Perhaps just mention that it was added in, then reverted, and now after a year or so, added back in with the fix.

#19 lore raw

RFC PATCH 2/4 xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

In reply to Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

---
 arch/x86/xen/enlighten.c | 9 +++++++++
 arch/x86/xen/smp.c       | 9 +++++++++
 arch/x86/xen/smp.h       | 1 +
 3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +9 -0
index a11af62..8074e4a 100644
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();
+	native_machine_shutdown();
+}
+
 static void __init xen_hvm_guest_init(void)
 {
 	init_hvm_pv_info();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void)
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	xen_hvm_init_time_ops();
 	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+#endif
 }
 
 static uint32_t __init xen_hvm_platform(void)
arch/x86/xen/smp.c +9 -0
index 7005974..35dcf39 100644
@@ -18,6 +18,7 @@
 #include <linux/smp.h>
 #include <linux/irq_work.h>
 #include <linux/tick.h>
+#include <linux/kexec.h>
 
 #include <asm/paravirt.h>
 #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)
 	native_cpu_die(cpu);
 }
 
+void xen_kexec_shutdown(void)
+{
+#ifdef CONFIG_KEXEC
+	if (!kexec_in_progress)
+		return;
+#endif
+}
+
 void __init xen_hvm_smp_init(void)
 {
 	if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
index c7c2d89..1af0493 100644
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);
 extern void xen_send_IPI_all(int vector);
 extern void xen_send_IPI_self(int vector);
 
+extern void xen_kexec_shutdown(void);
 #endif
Signature · 1 line
1.9.3
#20 lore raw

RFC PATCH 2/4 xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

In reply to Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

---
 arch/x86/xen/enlighten.c | 9 +++++++++
 arch/x86/xen/smp.c       | 9 +++++++++
 arch/x86/xen/smp.h       | 1 +
 3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +9 -0
index a11af62..8074e4a 100644
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();
+	native_machine_shutdown();
+}
+
 static void __init xen_hvm_guest_init(void)
 {
 	init_hvm_pv_info();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void)
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	xen_hvm_init_time_ops();
 	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+#endif
 }
 
 static uint32_t __init xen_hvm_platform(void)
arch/x86/xen/smp.c +9 -0
index 7005974..35dcf39 100644
@@ -18,6 +18,7 @@
 #include <linux/smp.h>
 #include <linux/irq_work.h>
 #include <linux/tick.h>
+#include <linux/kexec.h>
 
 #include <asm/paravirt.h>
 #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)
 	native_cpu_die(cpu);
 }
 
+void xen_kexec_shutdown(void)
+{
+#ifdef CONFIG_KEXEC
+	if (!kexec_in_progress)
+		return;
+#endif
+}
+
 void __init xen_hvm_smp_init(void)
 {
 	if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
index c7c2d89..1af0493 100644
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);
 extern void xen_send_IPI_all(int vector);
 extern void xen_send_IPI_self(int vector);
 
+extern void xen_kexec_shutdown(void);
 #endif
Signature · 1 line
1.9.3
#21 lore raw

Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

Quoted 3 lines · Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

This looks close to what I had sent as an RFC to you?

Quoted 74 lines
---
 arch/x86/xen/enlighten.c | 9 +++++++++
 arch/x86/xen/smp.c       | 9 +++++++++
 arch/x86/xen/smp.h       | 1 +
 3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +9 -0
index a11af62..8074e4a 100644
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();
+	native_machine_shutdown();
+}
+
 static void __init xen_hvm_guest_init(void)
 {
 	init_hvm_pv_info();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void)
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	xen_hvm_init_time_ops();
 	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+#endif
 }
 
 static uint32_t __init xen_hvm_platform(void)
arch/x86/xen/smp.c +9 -0
index 7005974..35dcf39 100644
@@ -18,6 +18,7 @@
 #include <linux/smp.h>
 #include <linux/irq_work.h>
 #include <linux/tick.h>
+#include <linux/kexec.h>
 
 #include <asm/paravirt.h>
 #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)
 	native_cpu_die(cpu);
 }
 
+void xen_kexec_shutdown(void)
+{
+#ifdef CONFIG_KEXEC
+	if (!kexec_in_progress)
+		return;
+#endif
+}
+
 void __init xen_hvm_smp_init(void)
 {
 	if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
index c7c2d89..1af0493 100644
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);
 extern void xen_send_IPI_all(int vector);
 extern void xen_send_IPI_self(int vector);
 
+extern void xen_kexec_shutdown(void);
 #endif
Signature · 1 line
1.9.3
#22 lore raw

Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

In reply to Vitaly Kuznetsov
Quoted 3 lines · Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

This looks close to what I had sent as an RFC to you?

Quoted 74 lines
---
 arch/x86/xen/enlighten.c | 9 +++++++++
 arch/x86/xen/smp.c       | 9 +++++++++
 arch/x86/xen/smp.h       | 1 +
 3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +9 -0
index a11af62..8074e4a 100644
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();
+	native_machine_shutdown();
+}
+
 static void __init xen_hvm_guest_init(void)
 {
 	init_hvm_pv_info();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void)
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	xen_hvm_init_time_ops();
 	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+#endif
 }
 
 static uint32_t __init xen_hvm_platform(void)
arch/x86/xen/smp.c +9 -0
index 7005974..35dcf39 100644
@@ -18,6 +18,7 @@
 #include <linux/smp.h>
 #include <linux/irq_work.h>
 #include <linux/tick.h>
+#include <linux/kexec.h>
 
 #include <asm/paravirt.h>
 #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)
 	native_cpu_die(cpu);
 }
 
+void xen_kexec_shutdown(void)
+{
+#ifdef CONFIG_KEXEC
+	if (!kexec_in_progress)
+		return;
+#endif
+}
+
 void __init xen_hvm_smp_init(void)
 {
 	if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
index c7c2d89..1af0493 100644
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);
 extern void xen_send_IPI_all(int vector);
 extern void xen_send_IPI_self(int vector);
 
+extern void xen_kexec_shutdown(void);
 #endif
Signature · 1 line
1.9.3
#23 lore raw

Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

This looks close to what I had sent as an RFC to you?

Yes, I stole that part from your "RFC: VCPU_reset_cpu_info" patch to call xen_unmap_all_pirqs() on shutdown. I'm looking at SMP issues your patches address as well.

Quoted 74 lines
Quoted 74 lines
---
 arch/x86/xen/enlighten.c | 9 +++++++++
 arch/x86/xen/smp.c       | 9 +++++++++
 arch/x86/xen/smp.h       | 1 +
 3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +9 -0
index a11af62..8074e4a 100644
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();
+	native_machine_shutdown();
+}
+
 static void __init xen_hvm_guest_init(void)
 {
 	init_hvm_pv_info();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void)
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	xen_hvm_init_time_ops();
 	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+#endif
 }
 
 static uint32_t __init xen_hvm_platform(void)
arch/x86/xen/smp.c +9 -0
index 7005974..35dcf39 100644
@@ -18,6 +18,7 @@
 #include <linux/smp.h>
 #include <linux/irq_work.h>
 #include <linux/tick.h>
+#include <linux/kexec.h>
 
 #include <asm/paravirt.h>
 #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)
 	native_cpu_die(cpu);
 }
 
+void xen_kexec_shutdown(void)
+{
+#ifdef CONFIG_KEXEC
+	if (!kexec_in_progress)
+		return;
+#endif
+}
+
 void __init xen_hvm_smp_init(void)
 {
 	if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
index c7c2d89..1af0493 100644
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);
 extern void xen_send_IPI_all(int vector);
 extern void xen_send_IPI_self(int vector);
 
+extern void xen_kexec_shutdown(void);
 #endif
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#24 lore raw

Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

In reply to Konrad Rzeszutek Wilk
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

This looks close to what I had sent as an RFC to you?

Yes, I stole that part from your "RFC: VCPU_reset_cpu_info" patch to call xen_unmap_all_pirqs() on shutdown. I'm looking at SMP issues your patches address as well.

Quoted 74 lines
Quoted 74 lines
---
 arch/x86/xen/enlighten.c | 9 +++++++++
 arch/x86/xen/smp.c       | 9 +++++++++
 arch/x86/xen/smp.h       | 1 +
 3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +9 -0
index a11af62..8074e4a 100644
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();
+	native_machine_shutdown();
+}
+
 static void __init xen_hvm_guest_init(void)
 {
 	init_hvm_pv_info();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void)
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	xen_hvm_init_time_ops();
 	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+#endif
 }
 
 static uint32_t __init xen_hvm_platform(void)
arch/x86/xen/smp.c +9 -0
index 7005974..35dcf39 100644
@@ -18,6 +18,7 @@
 #include <linux/smp.h>
 #include <linux/irq_work.h>
 #include <linux/tick.h>
+#include <linux/kexec.h>
 
 #include <asm/paravirt.h>
 #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)
 	native_cpu_die(cpu);
 }
 
+void xen_kexec_shutdown(void)
+{
+#ifdef CONFIG_KEXEC
+	if (!kexec_in_progress)
+		return;
+#endif
+}
+
 void __init xen_hvm_smp_init(void)
 {
 	if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
index c7c2d89..1af0493 100644
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);
 extern void xen_send_IPI_all(int vector);
 extern void xen_send_IPI_self(int vector);
 
+extern void xen_kexec_shutdown(void);
 #endif
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#25 lore raw

Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

Quoted 7 lines · Vitaly Kuznetsov
Quoted 5 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

Quoted 6 lines
Quoted 2 lines

This looks close to what I had sent as an RFC to you?

Yes, I stole that part from your "RFC: VCPU_reset_cpu_info" patch to call xen_unmap_all_pirqs() on shutdown. I'm looking at SMP issues your patches address as well.

It is customary to attribute that in the patch. Such as:

"Original idea/patch from Konrad Rzeszutek Wilk" is suffice.

Thanks!

Quoted 78 lines
Quoted 75 lines
Quoted 75 lines
---
 arch/x86/xen/enlighten.c | 9 +++++++++
 arch/x86/xen/smp.c       | 9 +++++++++
 arch/x86/xen/smp.h       | 1 +
 3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +9 -0
index a11af62..8074e4a 100644
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();
+	native_machine_shutdown();
+}
+
 static void __init xen_hvm_guest_init(void)
 {
 	init_hvm_pv_info();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void)
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	xen_hvm_init_time_ops();
 	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+#endif
 }
 
 static uint32_t __init xen_hvm_platform(void)
arch/x86/xen/smp.c +9 -0
index 7005974..35dcf39 100644
@@ -18,6 +18,7 @@
 #include <linux/smp.h>
 #include <linux/irq_work.h>
 #include <linux/tick.h>
+#include <linux/kexec.h>
 
 #include <asm/paravirt.h>
 #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)
 	native_cpu_die(cpu);
 }
 
+void xen_kexec_shutdown(void)
+{
+#ifdef CONFIG_KEXEC
+	if (!kexec_in_progress)
+		return;
+#endif
+}
+
 void __init xen_hvm_smp_init(void)
 {
 	if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
index c7c2d89..1af0493 100644
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);
 extern void xen_send_IPI_all(int vector);
 extern void xen_send_IPI_self(int vector);
 
+extern void xen_kexec_shutdown(void);
 #endif
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#26 lore raw

Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

In reply to Vitaly Kuznetsov
Quoted 7 lines · Vitaly Kuznetsov
Quoted 5 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

Quoted 6 lines
Quoted 2 lines

This looks close to what I had sent as an RFC to you?

Yes, I stole that part from your "RFC: VCPU_reset_cpu_info" patch to call xen_unmap_all_pirqs() on shutdown. I'm looking at SMP issues your patches address as well.

It is customary to attribute that in the patch. Such as:

"Original idea/patch from Konrad Rzeszutek Wilk" is suffice.

Thanks!

Quoted 78 lines
Quoted 75 lines
Quoted 75 lines
---
 arch/x86/xen/enlighten.c | 9 +++++++++
 arch/x86/xen/smp.c       | 9 +++++++++
 arch/x86/xen/smp.h       | 1 +
 3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +9 -0
index a11af62..8074e4a 100644
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();
+	native_machine_shutdown();
+}
+
 static void __init xen_hvm_guest_init(void)
 {
 	init_hvm_pv_info();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void)
 	x86_init.irqs.intr_init = xen_init_IRQ;
 	xen_hvm_init_time_ops();
 	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+#endif
 }
 
 static uint32_t __init xen_hvm_platform(void)
arch/x86/xen/smp.c +9 -0
index 7005974..35dcf39 100644
@@ -18,6 +18,7 @@
 #include <linux/smp.h>
 #include <linux/irq_work.h>
 #include <linux/tick.h>
+#include <linux/kexec.h>
 
 #include <asm/paravirt.h>
 #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)
 	native_cpu_die(cpu);
 }
 
+void xen_kexec_shutdown(void)
+{
+#ifdef CONFIG_KEXEC
+	if (!kexec_in_progress)
+		return;
+#endif
+}
+
 void __init xen_hvm_smp_init(void)
 {
 	if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
index c7c2d89..1af0493 100644
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);
 extern void xen_send_IPI_all(int vector);
 extern void xen_send_IPI_self(int vector);
 
+extern void xen_kexec_shutdown(void);
 #endif
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#27 lore raw

Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

In reply to Vitaly Kuznetsov
Quoted 15 lines · Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

---
  arch/x86/xen/enlighten.c | 9 +++++++++
  arch/x86/xen/smp.c       | 9 +++++++++
  arch/x86/xen/smp.h       | 1 +
  3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +0 -0
index a11af62..8074e4a 100644

arch/x86/xen/enlighten.c static struct notifier_block xen_hvm_cpu_notifier = { · line 1841

Quoted 9 lines
arch/x86/xen/enlighten.c +6 -0
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
  	.notifier_call	= xen_hvm_cpu_notify,
  };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();
+	native_machine_shutdown();
+}
+

This should probably be under ifdef CONFIG_KEXEC ?

Quoted 27 lines
arch/x86/xen/enlighten.c +3 -0
  static void __init xen_hvm_guest_init(void)
  {
  	init_hvm_pv_info();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void)
  	x86_init.irqs.intr_init = xen_init_IRQ;
  	xen_hvm_init_time_ops();
  	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+#endif
  }
 
  static uint32_t __init xen_hvm_platform(void)
arch/x86/xen/smp.c +1 -0
index 7005974..35dcf39 100644
@@ -18,6 +18,7 @@
  #include <linux/smp.h>
  #include <linux/irq_work.h>
  #include <linux/tick.h>
+#include <linux/kexec.h>
 
  #include <asm/paravirt.h>
  #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)
  	native_cpu_die(cpu);

arch/x86/xen/smp.c static void xen_hvm_cpu_die(unsigned int cpu) · line 773

Quoted 10 lines
arch/x86/xen/smp.c +8 -0
  }
 
+void xen_kexec_shutdown(void)
+{
+#ifdef CONFIG_KEXEC
+	if (!kexec_in_progress)
+		return;
+#endif
+}
+

and this one too (whole routine)? And "#include" above as well.

Quoted 13 lines
arch/x86/xen/smp.c +0 -0
  void __init xen_hvm_smp_init(void)
  {
  	if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
index c7c2d89..1af0493 100644
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);
  extern void xen_send_IPI_all(int vector);
  extern void xen_send_IPI_self(int vector);
 
+extern void xen_kexec_shutdown(void);
  #endif
#28 lore raw

Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

In reply to Vitaly Kuznetsov
Quoted 15 lines · Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

---
  arch/x86/xen/enlighten.c | 9 +++++++++
  arch/x86/xen/smp.c       | 9 +++++++++
  arch/x86/xen/smp.h       | 1 +
  3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +0 -0
index a11af62..8074e4a 100644

arch/x86/xen/enlighten.c static struct notifier_block xen_hvm_cpu_notifier = { · line 1841

Quoted 9 lines
arch/x86/xen/enlighten.c +6 -0
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
  	.notifier_call	= xen_hvm_cpu_notify,
  };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();
+	native_machine_shutdown();
+}
+

This should probably be under ifdef CONFIG_KEXEC ?

Quoted 27 lines
arch/x86/xen/enlighten.c +3 -0
  static void __init xen_hvm_guest_init(void)
  {
  	init_hvm_pv_info();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void)
  	x86_init.irqs.intr_init = xen_init_IRQ;
  	xen_hvm_init_time_ops();
  	xen_hvm_init_mmu_ops();
+#ifdef CONFIG_KEXEC
+	machine_ops.shutdown = xen_pvhvm_kexec_shutdown;
+#endif
  }
 
  static uint32_t __init xen_hvm_platform(void)
arch/x86/xen/smp.c +1 -0
index 7005974..35dcf39 100644
@@ -18,6 +18,7 @@
  #include <linux/smp.h>
  #include <linux/irq_work.h>
  #include <linux/tick.h>
+#include <linux/kexec.h>
 
  #include <asm/paravirt.h>
  #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)
  	native_cpu_die(cpu);

arch/x86/xen/smp.c static void xen_hvm_cpu_die(unsigned int cpu) · line 773

Quoted 10 lines
arch/x86/xen/smp.c +8 -0
  }
 
+void xen_kexec_shutdown(void)
+{
+#ifdef CONFIG_KEXEC
+	if (!kexec_in_progress)
+		return;
+#endif
+}
+

and this one too (whole routine)? And "#include" above as well.

Quoted 13 lines
arch/x86/xen/smp.c +0 -0
  void __init xen_hvm_smp_init(void)
  {
  	if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
index c7c2d89..1af0493 100644
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);
  extern void xen_send_IPI_all(int vector);
  extern void xen_send_IPI_self(int vector);
 
+extern void xen_kexec_shutdown(void);
  #endif
#29 lore raw

Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

In reply to Vitaly Kuznetsov
Quoted 2 lines · Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

[...]

arch/x86/xen/enlighten.c static struct notifier_block xen_hvm_cpu_notifier = { · line 1838

Quoted 9 lines
arch/x86/xen/enlighten.c +3 -0
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();

I think it would be preferrable to have

    if (kexec_in_progress)
        xen_kexec_shutdown();

David

#30 lore raw

Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()

In reply to Vitaly Kuznetsov
Quoted 2 lines · Vitaly Kuznetsov

PVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().

[...]

arch/x86/xen/enlighten.c static struct notifier_block xen_hvm_cpu_notifier = { · line 1838

Quoted 9 lines
arch/x86/xen/enlighten.c +3 -0
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = {
 	.notifier_call	= xen_hvm_cpu_notify,
 };
 
+static void xen_pvhvm_kexec_shutdown(void)
+{
+	xen_kexec_shutdown();

I think it would be preferrable to have

    if (kexec_in_progress)
        xen_kexec_shutdown();

David

#31 lore raw

RFC PATCH 3/4 xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

---
 arch/x86/xen/smp.c               |  1 +
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
index 35dcf39..e2b4deb 100644
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +76 -0

88 lines not on this page · raw message

include/xen/events.h +3 -0
index 8bee7a7..3f9f428 100644
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* Unmap all PIRQs and close event channels */
+void xen_unmap_all_pirqs(void);
+
 /* initialize Xen IRQ subsystem */
 void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
#32 lore raw

RFC PATCH 3/4 xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

---
 arch/x86/xen/smp.c               |  1 +
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
index 35dcf39..e2b4deb 100644
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +76 -0

88 lines not on this page · raw message

include/xen/events.h +3 -0
index 8bee7a7..3f9f428 100644
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* Unmap all PIRQs and close event channels */
+void xen_unmap_all_pirqs(void);
+
 /* initialize Xen IRQ subsystem */
 void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
#33 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?

Thanks.

Quoted 58 lines
---
 arch/x86/xen/smp.c               |  1 +
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
index 35dcf39..e2b4deb 100644
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +30 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;
+
+		pirq = info->u.pirq.pirq;
+		gsi = info->u.pirq.gsi;
+		evtchn = info->evtchn;
+		irq = info->irq;
+
+		pr_debug("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n",
+			gsi, pirq, irq, evtchn);
+
+		if (evtchn > 0) {
+			close.port = evtchn;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("close evtchn %d failed\n", evtchn);
+		}
+
+		unmap_irq.pirq = pirq;
+		unmap_irq.domid = DOMID_SELF;

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
+		if (rc)
+			pr_warn("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n",
+				gsi, pirq, irq, rc);
+	}
+
+	mutex_unlock(&irq_mapping_update_lock);
+}
+EXPORT_SYMBOL_GPL(xen_unmap_all_pirqs);

Why the EXPORT? Is this used by modules?

Quoted 63 lines
drivers/xen/events/events_base.c +36 -0
+
+static void xen_startup_unmap_pirqs(void)
+{
+	struct evtchn_status status;
+	int port, rc = -ENOENT;
+	struct physdev_unmap_pirq unmap_irq;
+	struct evtchn_close close;
+
+	memset(&status, 0, sizeof(status));
+	for (port = 0; port < xen_evtchn_max_channels(); port++) {
+		status.dom = DOMID_SELF;
+		status.port = port;
+		rc = HYPERVISOR_event_channel_op(EVTCHNOP_status, &status);
+		if (rc < 0)
+			continue;
+		if (status.status == EVTCHNSTAT_pirq) {
+			close.port = port;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("xen: failed to close evtchn %d\n",
+					port);
+			unmap_irq.pirq = status.u.pirq;
+			unmap_irq.domid = DOMID_SELF;
+			pr_warn("xen: unmapping previously mapped pirq %d\n",
+				unmap_irq.pirq);
+			if (HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq,
+						  &unmap_irq) != 0)
+				pr_warn("xen: failed to unmap pirq %d\n",
+					unmap_irq.pirq);
+		}
+	}
+}
+
+
 void __init xen_init_IRQ(void)
 {
 	int ret = -EINVAL;
@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)
 		xen_callback_vector();
 
 	if (xen_hvm_domain()) {
+		xen_startup_unmap_pirqs();
+
 		native_init_IRQ();
 		/* pci_xen_hvm_init must be called after native_init_IRQ so that
 		 * __acpi_register_gsi can point at the right function */
include/xen/events.h +3 -0
index 8bee7a7..3f9f428 100644
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* Unmap all PIRQs and close event channels */
+void xen_unmap_all_pirqs(void);
+
 /* initialize Xen IRQ subsystem */
 void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
#34 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to Vitaly Kuznetsov
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?

Thanks.

Quoted 58 lines
---
 arch/x86/xen/smp.c               |  1 +
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
index 35dcf39..e2b4deb 100644
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +30 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;
+
+		pirq = info->u.pirq.pirq;
+		gsi = info->u.pirq.gsi;
+		evtchn = info->evtchn;
+		irq = info->irq;
+
+		pr_debug("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n",
+			gsi, pirq, irq, evtchn);
+
+		if (evtchn > 0) {
+			close.port = evtchn;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("close evtchn %d failed\n", evtchn);
+		}
+
+		unmap_irq.pirq = pirq;
+		unmap_irq.domid = DOMID_SELF;

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
+		if (rc)
+			pr_warn("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n",
+				gsi, pirq, irq, rc);
+	}
+
+	mutex_unlock(&irq_mapping_update_lock);
+}
+EXPORT_SYMBOL_GPL(xen_unmap_all_pirqs);

Why the EXPORT? Is this used by modules?

Quoted 63 lines
drivers/xen/events/events_base.c +36 -0
+
+static void xen_startup_unmap_pirqs(void)
+{
+	struct evtchn_status status;
+	int port, rc = -ENOENT;
+	struct physdev_unmap_pirq unmap_irq;
+	struct evtchn_close close;
+
+	memset(&status, 0, sizeof(status));
+	for (port = 0; port < xen_evtchn_max_channels(); port++) {
+		status.dom = DOMID_SELF;
+		status.port = port;
+		rc = HYPERVISOR_event_channel_op(EVTCHNOP_status, &status);
+		if (rc < 0)
+			continue;
+		if (status.status == EVTCHNSTAT_pirq) {
+			close.port = port;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("xen: failed to close evtchn %d\n",
+					port);
+			unmap_irq.pirq = status.u.pirq;
+			unmap_irq.domid = DOMID_SELF;
+			pr_warn("xen: unmapping previously mapped pirq %d\n",
+				unmap_irq.pirq);
+			if (HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq,
+						  &unmap_irq) != 0)
+				pr_warn("xen: failed to unmap pirq %d\n",
+					unmap_irq.pirq);
+		}
+	}
+}
+
+
 void __init xen_init_IRQ(void)
 {
 	int ret = -EINVAL;
@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)
 		xen_callback_vector();
 
 	if (xen_hvm_domain()) {
+		xen_startup_unmap_pirqs();
+
 		native_init_IRQ();
 		/* pci_xen_hvm_init must be called after native_init_IRQ so that
 		 * __acpi_register_gsi can point at the right function */
include/xen/events.h +3 -0
index 8bee7a7..3f9f428 100644
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* Unmap all PIRQs and close event channels */
+void xen_unmap_all_pirqs(void);
+
 /* initialize Xen IRQ subsystem */
 void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
#35 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

Quoted 24 lines · Konrad Rzeszutek Wilk
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 2 lines

How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?

Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...

Quoted 135 lines

Thanks.

Quoted 58 lines
---
 arch/x86/xen/smp.c               |  1 +
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
index 35dcf39..e2b4deb 100644
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +30 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;
+
+		pirq = info->u.pirq.pirq;
+		gsi = info->u.pirq.gsi;
+		evtchn = info->evtchn;
+		irq = info->irq;
+
+		pr_debug("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n",
+			gsi, pirq, irq, evtchn);
+
+		if (evtchn > 0) {
+			close.port = evtchn;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("close evtchn %d failed\n", evtchn);
+		}
+
+		unmap_irq.pirq = pirq;
+		unmap_irq.domid = DOMID_SELF;

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
+		if (rc)
+			pr_warn("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n",
+				gsi, pirq, irq, rc);
+	}
+
+	mutex_unlock(&irq_mapping_update_lock);
+}
+EXPORT_SYMBOL_GPL(xen_unmap_all_pirqs);

Why the EXPORT? Is this used by modules?

Quoted 63 lines
drivers/xen/events/events_base.c +36 -0
+
+static void xen_startup_unmap_pirqs(void)
+{
+	struct evtchn_status status;
+	int port, rc = -ENOENT;
+	struct physdev_unmap_pirq unmap_irq;
+	struct evtchn_close close;
+
+	memset(&status, 0, sizeof(status));
+	for (port = 0; port < xen_evtchn_max_channels(); port++) {
+		status.dom = DOMID_SELF;
+		status.port = port;
+		rc = HYPERVISOR_event_channel_op(EVTCHNOP_status, &status);
+		if (rc < 0)
+			continue;
+		if (status.status == EVTCHNSTAT_pirq) {
+			close.port = port;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("xen: failed to close evtchn %d\n",
+					port);
+			unmap_irq.pirq = status.u.pirq;
+			unmap_irq.domid = DOMID_SELF;
+			pr_warn("xen: unmapping previously mapped pirq %d\n",
+				unmap_irq.pirq);
+			if (HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq,
+						  &unmap_irq) != 0)
+				pr_warn("xen: failed to unmap pirq %d\n",
+					unmap_irq.pirq);
+		}
+	}
+}
+
+
 void __init xen_init_IRQ(void)
 {
 	int ret = -EINVAL;
@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)
 		xen_callback_vector();
 
 	if (xen_hvm_domain()) {
+		xen_startup_unmap_pirqs();
+
 		native_init_IRQ();
 		/* pci_xen_hvm_init must be called after native_init_IRQ so that
 		 * __acpi_register_gsi can point at the right function */
include/xen/events.h +3 -0
index 8bee7a7..3f9f428 100644
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* Unmap all PIRQs and close event channels */
+void xen_unmap_all_pirqs(void);
+
 /* initialize Xen IRQ subsystem */
 void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#36 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to Konrad Rzeszutek Wilk
Quoted 24 lines · Konrad Rzeszutek Wilk
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 2 lines

How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?

Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...

Quoted 135 lines

Thanks.

Quoted 58 lines
---
 arch/x86/xen/smp.c               |  1 +
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
index 35dcf39..e2b4deb 100644
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +30 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;
+
+		pirq = info->u.pirq.pirq;
+		gsi = info->u.pirq.gsi;
+		evtchn = info->evtchn;
+		irq = info->irq;
+
+		pr_debug("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n",
+			gsi, pirq, irq, evtchn);
+
+		if (evtchn > 0) {
+			close.port = evtchn;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("close evtchn %d failed\n", evtchn);
+		}
+
+		unmap_irq.pirq = pirq;
+		unmap_irq.domid = DOMID_SELF;

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
+		if (rc)
+			pr_warn("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n",
+				gsi, pirq, irq, rc);
+	}
+
+	mutex_unlock(&irq_mapping_update_lock);
+}
+EXPORT_SYMBOL_GPL(xen_unmap_all_pirqs);

Why the EXPORT? Is this used by modules?

Quoted 63 lines
drivers/xen/events/events_base.c +36 -0
+
+static void xen_startup_unmap_pirqs(void)
+{
+	struct evtchn_status status;
+	int port, rc = -ENOENT;
+	struct physdev_unmap_pirq unmap_irq;
+	struct evtchn_close close;
+
+	memset(&status, 0, sizeof(status));
+	for (port = 0; port < xen_evtchn_max_channels(); port++) {
+		status.dom = DOMID_SELF;
+		status.port = port;
+		rc = HYPERVISOR_event_channel_op(EVTCHNOP_status, &status);
+		if (rc < 0)
+			continue;
+		if (status.status == EVTCHNSTAT_pirq) {
+			close.port = port;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("xen: failed to close evtchn %d\n",
+					port);
+			unmap_irq.pirq = status.u.pirq;
+			unmap_irq.domid = DOMID_SELF;
+			pr_warn("xen: unmapping previously mapped pirq %d\n",
+				unmap_irq.pirq);
+			if (HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq,
+						  &unmap_irq) != 0)
+				pr_warn("xen: failed to unmap pirq %d\n",
+					unmap_irq.pirq);
+		}
+	}
+}
+
+
 void __init xen_init_IRQ(void)
 {
 	int ret = -EINVAL;
@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)
 		xen_callback_vector();
 
 	if (xen_hvm_domain()) {
+		xen_startup_unmap_pirqs();
+
 		native_init_IRQ();
 		/* pci_xen_hvm_init must be called after native_init_IRQ so that
 		 * __acpi_register_gsi can point at the right function */
include/xen/events.h +3 -0
index 8bee7a7..3f9f428 100644
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* Unmap all PIRQs and close event channels */
+void xen_unmap_all_pirqs(void);
+
 /* initialize Xen IRQ subsystem */
 void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#37 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

Quoted 26 lines · Vitaly Kuznetsov
Quoted 24 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 6 lines
Quoted 2 lines

How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?

Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...

I think for FIFO events we can't do much right now - it would need some new hypercalls to de-allocate or such.

But I was thinking that your code logic could just return out when it detects that it is running with FIFO events (with a TODO comment)
- and also spit out some information to this effect?

Say: "Use xen.fifo=0 in your launching kernel"

(don't know the right name for the kernel in which you do 'kexec -e' in ? Is that launching? Original? Bootstrap kernel?)

Quoted 139 lines
Quoted 136 lines

Thanks.

Quoted 58 lines
---
 arch/x86/xen/smp.c               |  1 +
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
index 35dcf39..e2b4deb 100644
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +30 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;
+
+		pirq = info->u.pirq.pirq;
+		gsi = info->u.pirq.gsi;
+		evtchn = info->evtchn;
+		irq = info->irq;
+
+		pr_debug("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n",
+			gsi, pirq, irq, evtchn);
+
+		if (evtchn > 0) {
+			close.port = evtchn;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("close evtchn %d failed\n", evtchn);
+		}
+
+		unmap_irq.pirq = pirq;
+		unmap_irq.domid = DOMID_SELF;

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
+		if (rc)
+			pr_warn("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n",
+				gsi, pirq, irq, rc);
+	}
+
+	mutex_unlock(&irq_mapping_update_lock);
+}
+EXPORT_SYMBOL_GPL(xen_unmap_all_pirqs);

Why the EXPORT? Is this used by modules?

Quoted 63 lines
drivers/xen/events/events_base.c +36 -0
+
+static void xen_startup_unmap_pirqs(void)
+{
+	struct evtchn_status status;
+	int port, rc = -ENOENT;
+	struct physdev_unmap_pirq unmap_irq;
+	struct evtchn_close close;
+
+	memset(&status, 0, sizeof(status));
+	for (port = 0; port < xen_evtchn_max_channels(); port++) {
+		status.dom = DOMID_SELF;
+		status.port = port;
+		rc = HYPERVISOR_event_channel_op(EVTCHNOP_status, &status);
+		if (rc < 0)
+			continue;
+		if (status.status == EVTCHNSTAT_pirq) {
+			close.port = port;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("xen: failed to close evtchn %d\n",
+					port);
+			unmap_irq.pirq = status.u.pirq;
+			unmap_irq.domid = DOMID_SELF;
+			pr_warn("xen: unmapping previously mapped pirq %d\n",
+				unmap_irq.pirq);
+			if (HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq,
+						  &unmap_irq) != 0)
+				pr_warn("xen: failed to unmap pirq %d\n",
+					unmap_irq.pirq);
+		}
+	}
+}
+
+
 void __init xen_init_IRQ(void)
 {
 	int ret = -EINVAL;
@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)
 		xen_callback_vector();
 
 	if (xen_hvm_domain()) {
+		xen_startup_unmap_pirqs();
+
 		native_init_IRQ();
 		/* pci_xen_hvm_init must be called after native_init_IRQ so that
 		 * __acpi_register_gsi can point at the right function */
include/xen/events.h +3 -0
index 8bee7a7..3f9f428 100644
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* Unmap all PIRQs and close event channels */
+void xen_unmap_all_pirqs(void);
+
 /* initialize Xen IRQ subsystem */
 void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#38 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to Vitaly Kuznetsov
Quoted 26 lines · Vitaly Kuznetsov
Quoted 24 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 6 lines
Quoted 2 lines

How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?

Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...

I think for FIFO events we can't do much right now - it would need some new hypercalls to de-allocate or such.

But I was thinking that your code logic could just return out when it detects that it is running with FIFO events (with a TODO comment)
- and also spit out some information to this effect?

Say: "Use xen.fifo=0 in your launching kernel"

(don't know the right name for the kernel in which you do 'kexec -e' in ? Is that launching? Original? Bootstrap kernel?)

Quoted 139 lines
Quoted 136 lines

Thanks.

Quoted 58 lines
---
 arch/x86/xen/smp.c               |  1 +
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
index 35dcf39..e2b4deb 100644
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +30 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;
+
+		pirq = info->u.pirq.pirq;
+		gsi = info->u.pirq.gsi;
+		evtchn = info->evtchn;
+		irq = info->irq;
+
+		pr_debug("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n",
+			gsi, pirq, irq, evtchn);
+
+		if (evtchn > 0) {
+			close.port = evtchn;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("close evtchn %d failed\n", evtchn);
+		}
+
+		unmap_irq.pirq = pirq;
+		unmap_irq.domid = DOMID_SELF;

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
+		if (rc)
+			pr_warn("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n",
+				gsi, pirq, irq, rc);
+	}
+
+	mutex_unlock(&irq_mapping_update_lock);
+}
+EXPORT_SYMBOL_GPL(xen_unmap_all_pirqs);

Why the EXPORT? Is this used by modules?

Quoted 63 lines
drivers/xen/events/events_base.c +36 -0
+
+static void xen_startup_unmap_pirqs(void)
+{
+	struct evtchn_status status;
+	int port, rc = -ENOENT;
+	struct physdev_unmap_pirq unmap_irq;
+	struct evtchn_close close;
+
+	memset(&status, 0, sizeof(status));
+	for (port = 0; port < xen_evtchn_max_channels(); port++) {
+		status.dom = DOMID_SELF;
+		status.port = port;
+		rc = HYPERVISOR_event_channel_op(EVTCHNOP_status, &status);
+		if (rc < 0)
+			continue;
+		if (status.status == EVTCHNSTAT_pirq) {
+			close.port = port;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("xen: failed to close evtchn %d\n",
+					port);
+			unmap_irq.pirq = status.u.pirq;
+			unmap_irq.domid = DOMID_SELF;
+			pr_warn("xen: unmapping previously mapped pirq %d\n",
+				unmap_irq.pirq);
+			if (HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq,
+						  &unmap_irq) != 0)
+				pr_warn("xen: failed to unmap pirq %d\n",
+					unmap_irq.pirq);
+		}
+	}
+}
+
+
 void __init xen_init_IRQ(void)
 {
 	int ret = -EINVAL;
@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)
 		xen_callback_vector();
 
 	if (xen_hvm_domain()) {
+		xen_startup_unmap_pirqs();
+
 		native_init_IRQ();
 		/* pci_xen_hvm_init must be called after native_init_IRQ so that
 		 * __acpi_register_gsi can point at the right function */
include/xen/events.h +3 -0
index 8bee7a7..3f9f428 100644
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* Unmap all PIRQs and close event channels */
+void xen_unmap_all_pirqs(void);
+
 /* initialize Xen IRQ subsystem */
 void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#39 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

Quoted 30 lines · Konrad Rzeszutek Wilk
Quoted 29 lines · Vitaly Kuznetsov
Quoted 26 lines · Konrad Rzeszutek Wilk
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?

Quoted 7 lines
Quoted 3 lines

Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...

I think for FIFO events we can't do much right now - it would need some new hypercalls to de-allocate or such.

Yeah, you're probably right. I tried wrapping evtchn_fifo_destroy() into 'EVTCHNOP_fifo_destroy' hypercall but it seems some other actions are required as well..

Quoted 3 lines

But I was thinking that your code logic could just return out when it detects that it is running with FIFO events (with a TODO comment)
- and also spit out some information to this effect?

Sure, having TODO here is a good idea.

Quoted 2 lines

Say: "Use xen.fifo=0 in your launching kernel"

s,xen.fifo,xen.fifo_events,

Quoted 3 lines

(don't know the right name for the kernel in which you do 'kexec -e' in ? Is that launching? Original? Bootstrap kernel?)

Yes, if under Xen-4.4 I boot original kernel with "xen.fifo_events=0" I'm able to do kexec with "xen.fifo_events=0" and even without it (but only once :-). Once kernel is booted with FIFO-based event channels enabled no kexec is possible, new kernel gets stuck (I guess vcpuop timer doesn't work..). My patch series brings no difference here..

Thanks,

Quoted 140 lines
Quoted 140 lines
Quoted 137 lines

Thanks.

Quoted 58 lines
---
 arch/x86/xen/smp.c               |  1 +
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
index 35dcf39..e2b4deb 100644
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +30 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;
+
+		pirq = info->u.pirq.pirq;
+		gsi = info->u.pirq.gsi;
+		evtchn = info->evtchn;
+		irq = info->irq;
+
+		pr_debug("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n",
+			gsi, pirq, irq, evtchn);
+
+		if (evtchn > 0) {
+			close.port = evtchn;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("close evtchn %d failed\n", evtchn);
+		}
+
+		unmap_irq.pirq = pirq;
+		unmap_irq.domid = DOMID_SELF;

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
+		if (rc)
+			pr_warn("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n",
+				gsi, pirq, irq, rc);
+	}
+
+	mutex_unlock(&irq_mapping_update_lock);
+}
+EXPORT_SYMBOL_GPL(xen_unmap_all_pirqs);

Why the EXPORT? Is this used by modules?

Quoted 63 lines
drivers/xen/events/events_base.c +36 -0
+
+static void xen_startup_unmap_pirqs(void)
+{
+	struct evtchn_status status;
+	int port, rc = -ENOENT;
+	struct physdev_unmap_pirq unmap_irq;
+	struct evtchn_close close;
+
+	memset(&status, 0, sizeof(status));
+	for (port = 0; port < xen_evtchn_max_channels(); port++) {
+		status.dom = DOMID_SELF;
+		status.port = port;
+		rc = HYPERVISOR_event_channel_op(EVTCHNOP_status, &status);
+		if (rc < 0)
+			continue;
+		if (status.status == EVTCHNSTAT_pirq) {
+			close.port = port;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("xen: failed to close evtchn %d\n",
+					port);
+			unmap_irq.pirq = status.u.pirq;
+			unmap_irq.domid = DOMID_SELF;
+			pr_warn("xen: unmapping previously mapped pirq %d\n",
+				unmap_irq.pirq);
+			if (HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq,
+						  &unmap_irq) != 0)
+				pr_warn("xen: failed to unmap pirq %d\n",
+					unmap_irq.pirq);
+		}
+	}
+}
+
+
 void __init xen_init_IRQ(void)
 {
 	int ret = -EINVAL;
@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)
 		xen_callback_vector();
 
 	if (xen_hvm_domain()) {
+		xen_startup_unmap_pirqs();
+
 		native_init_IRQ();
 		/* pci_xen_hvm_init must be called after native_init_IRQ so that
 		 * __acpi_register_gsi can point at the right function */
include/xen/events.h +3 -0
index 8bee7a7..3f9f428 100644
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* Unmap all PIRQs and close event channels */
+void xen_unmap_all_pirqs(void);
+
 /* initialize Xen IRQ subsystem */
 void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
#40 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to Konrad Rzeszutek Wilk
Quoted 30 lines · Konrad Rzeszutek Wilk
Quoted 29 lines · Vitaly Kuznetsov
Quoted 26 lines · Konrad Rzeszutek Wilk
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?

Quoted 7 lines
Quoted 3 lines

Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...

I think for FIFO events we can't do much right now - it would need some new hypercalls to de-allocate or such.

Yeah, you're probably right. I tried wrapping evtchn_fifo_destroy() into 'EVTCHNOP_fifo_destroy' hypercall but it seems some other actions are required as well..

Quoted 3 lines

But I was thinking that your code logic could just return out when it detects that it is running with FIFO events (with a TODO comment)
- and also spit out some information to this effect?

Sure, having TODO here is a good idea.

Quoted 2 lines

Say: "Use xen.fifo=0 in your launching kernel"

s,xen.fifo,xen.fifo_events,

Quoted 3 lines

(don't know the right name for the kernel in which you do 'kexec -e' in ? Is that launching? Original? Bootstrap kernel?)

Yes, if under Xen-4.4 I boot original kernel with "xen.fifo_events=0" I'm able to do kexec with "xen.fifo_events=0" and even without it (but only once :-). Once kernel is booted with FIFO-based event channels enabled no kexec is possible, new kernel gets stuck (I guess vcpuop timer doesn't work..). My patch series brings no difference here..

Thanks,

Quoted 140 lines
Quoted 140 lines
Quoted 137 lines

Thanks.

Quoted 58 lines
---
 arch/x86/xen/smp.c               |  1 +
 drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++
 include/xen/events.h             |  3 ++
 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
index 35dcf39..e2b4deb 100644
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +30 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;
+
+		pirq = info->u.pirq.pirq;
+		gsi = info->u.pirq.gsi;
+		evtchn = info->evtchn;
+		irq = info->irq;
+
+		pr_debug("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n",
+			gsi, pirq, irq, evtchn);
+
+		if (evtchn > 0) {
+			close.port = evtchn;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("close evtchn %d failed\n", evtchn);
+		}
+
+		unmap_irq.pirq = pirq;
+		unmap_irq.domid = DOMID_SELF;

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+
+		rc = HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
+		if (rc)
+			pr_warn("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n",
+				gsi, pirq, irq, rc);
+	}
+
+	mutex_unlock(&irq_mapping_update_lock);
+}
+EXPORT_SYMBOL_GPL(xen_unmap_all_pirqs);

Why the EXPORT? Is this used by modules?

Quoted 63 lines
drivers/xen/events/events_base.c +36 -0
+
+static void xen_startup_unmap_pirqs(void)
+{
+	struct evtchn_status status;
+	int port, rc = -ENOENT;
+	struct physdev_unmap_pirq unmap_irq;
+	struct evtchn_close close;
+
+	memset(&status, 0, sizeof(status));
+	for (port = 0; port < xen_evtchn_max_channels(); port++) {
+		status.dom = DOMID_SELF;
+		status.port = port;
+		rc = HYPERVISOR_event_channel_op(EVTCHNOP_status, &status);
+		if (rc < 0)
+			continue;
+		if (status.status == EVTCHNSTAT_pirq) {
+			close.port = port;
+			if (HYPERVISOR_event_channel_op(EVTCHNOP_close,
+							&close) != 0)
+				pr_warn("xen: failed to close evtchn %d\n",
+					port);
+			unmap_irq.pirq = status.u.pirq;
+			unmap_irq.domid = DOMID_SELF;
+			pr_warn("xen: unmapping previously mapped pirq %d\n",
+				unmap_irq.pirq);
+			if (HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq,
+						  &unmap_irq) != 0)
+				pr_warn("xen: failed to unmap pirq %d\n",
+					unmap_irq.pirq);
+		}
+	}
+}
+
+
 void __init xen_init_IRQ(void)
 {
 	int ret = -EINVAL;
@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)
 		xen_callback_vector();
 
 	if (xen_hvm_domain()) {
+		xen_startup_unmap_pirqs();
+
 		native_init_IRQ();
 		/* pci_xen_hvm_init must be called after native_init_IRQ so that
 		 * __acpi_register_gsi can point at the right function */
include/xen/events.h +3 -0
index 8bee7a7..3f9f428 100644
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);
 /* Determine whether to ignore this IRQ if it is passed to a guest. */
 int xen_test_irq_shared(int irq);
 
+/* Unmap all PIRQs and close event channels */
+void xen_unmap_all_pirqs(void);
+
 /* initialize Xen IRQ subsystem */
 void xen_init_IRQ(void);
 #endif	/* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
#41 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to Vitaly Kuznetsov
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).

Quoted 20 lines
arch/x86/xen/smp.c +1 -0
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +2 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1657

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;

I think you need to do this by querying Xen state rather than relying on potentially bad guest state. Particularly since you may crash while holding irq_mapping_update_lock.

EVTCHNOP_status gets you the info you need I think.

David

#42 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 3 lines

I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).

Thank you for your comments!

The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call evtchn_fifo_destroy() so next EVTCHNOP_init_control won't fail. I was intended to put evtchn_fifo_destroy() in EVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest with EVTCHNOP_bind_interdomain (if we remember where they were mapped before) but we can't do it after we did evtchn_fifo_destroy() and we can't rebind them after kexec and performing EVTCHNOP_init_control as we can't remember where these channels were mapped to after kexec/kdump.

I see the following possible solutions:
1) We put evtchn_fifo_destroy() in EVTCHNOP_init_control so EVTCHNOP_init_control can be called twice. No EVTCHNOP_reset is required in that case.

2) Introduce special (e.g. 'EVTCHNOP_fifo_destroy') hypercall to do evtchn_fifo_destroy() without closing all channels. Alternatively we can avoid closing all channels in EVTCHNOP_reset when called with DOMID_SELF (as this mode is not being used atm) -- but that would look unobvious.

3) Keep evtchn_fifo_destroy() in EVTCHNOP_reset but keep console/store channels -- I saw your concerns it is not safe, some sort of additional blocking will be required.

4) Do the remapping boot time (query for store/console channels -> perform EVTCHNOP_reset -> rebind with EVTCHNOP_bind_interdomain).

There is an additional problem: EVTCHNOP_bind_interdomain operation has local port as OUT parameter so we can't guarantee that remapping store/console channels will remap them to the same local channel they were mapped before EVTCHNOP_reset (and we have this information in hvm
info: HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN, ...). Not sure how to deal with that in case we go with remapping.

Your thoughts would be very appreciated. Thank you again,

Quoted 38 lines
Quoted 20 lines
arch/x86/xen/smp.c +1 -0
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +2 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1657

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;

I think you need to do this by querying Xen state rather than relying on potentially bad guest state. Particularly since you may crash while holding irq_mapping_update_lock.

EVTCHNOP_status gets you the info you need I think.

David

Signature · 1 line
  Vitaly
#43 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to David Vrabel
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 3 lines

I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).

Thank you for your comments!

The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call evtchn_fifo_destroy() so next EVTCHNOP_init_control won't fail. I was intended to put evtchn_fifo_destroy() in EVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest with EVTCHNOP_bind_interdomain (if we remember where they were mapped before) but we can't do it after we did evtchn_fifo_destroy() and we can't rebind them after kexec and performing EVTCHNOP_init_control as we can't remember where these channels were mapped to after kexec/kdump.

I see the following possible solutions:
1) We put evtchn_fifo_destroy() in EVTCHNOP_init_control so EVTCHNOP_init_control can be called twice. No EVTCHNOP_reset is required in that case.

2) Introduce special (e.g. 'EVTCHNOP_fifo_destroy') hypercall to do evtchn_fifo_destroy() without closing all channels. Alternatively we can avoid closing all channels in EVTCHNOP_reset when called with DOMID_SELF (as this mode is not being used atm) -- but that would look unobvious.

3) Keep evtchn_fifo_destroy() in EVTCHNOP_reset but keep console/store channels -- I saw your concerns it is not safe, some sort of additional blocking will be required.

4) Do the remapping boot time (query for store/console channels -> perform EVTCHNOP_reset -> rebind with EVTCHNOP_bind_interdomain).

There is an additional problem: EVTCHNOP_bind_interdomain operation has local port as OUT parameter so we can't guarantee that remapping store/console channels will remap them to the same local channel they were mapped before EVTCHNOP_reset (and we have this information in hvm
info: HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN, ...). Not sure how to deal with that in case we go with remapping.

Your thoughts would be very appreciated. Thank you again,

Quoted 38 lines
Quoted 20 lines
arch/x86/xen/smp.c +1 -0
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +2 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1657

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;

I think you need to do this by querying Xen state rather than relying on potentially bad guest state. Particularly since you may crash while holding irq_mapping_update_lock.

EVTCHNOP_status gets you the info you need I think.

David

Signature · 1 line
  Vitaly
#44 lore raw

Re: [Xen-devel] [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 3 lines

I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).

Thank you for your comments!

The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call evtchn_fifo_destroy() so next EVTCHNOP_init_control won't fail. I was intended to put evtchn_fifo_destroy() in EVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest with EVTCHNOP_bind_interdomain (if we remember where they were mapped before) but we can't do it after we did evtchn_fifo_destroy() and we can't rebind them after kexec and performing EVTCHNOP_init_control as we can't remember where these channels were mapped to after kexec/kdump.

Quoted 4 lines

I see the following possible solutions:
1) We put evtchn_fifo_destroy() in EVTCHNOP_init_control so EVTCHNOP_init_control can be called twice. No EVTCHNOP_reset is required in that case.

EVTCHNOP_init_control is called for each VCPU so I can't see how this would work.

Quoted 4 lines

2) Introduce special (e.g. 'EVTCHNOP_fifo_destroy') hypercall to do evtchn_fifo_destroy() without closing all channels. Alternatively we can avoid closing all channels in EVTCHNOP_reset when called with DOMID_SELF (as this mode is not being used atm) -- but that would look unobvious.

I would try this. The guest prior to kexec would then:

1. Use EVTCHNOP_status to query remote end of console and xenstore event channels.

2. Loop for all event channels:

a. unmap pirq (if required)
b. EVTCHNOP_close

3. EVTCHNOP_fifo_destroy (the implementation of which must verify that no channels are bound).

4. EVTCHNOP_bind_interdomain to rebind the console and xenstore channels.

David

#45 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

Quoted 43 lines · David Vrabel
Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 3 lines

I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).

Thank you for your comments!

The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call evtchn_fifo_destroy() so next EVTCHNOP_init_control won't fail. I was intended to put evtchn_fifo_destroy() in EVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest with EVTCHNOP_bind_interdomain (if we remember where they were mapped before) but we can't do it after we did evtchn_fifo_destroy() and we can't rebind them after kexec and performing EVTCHNOP_init_control as we can't remember where these channels were mapped to after kexec/kdump.

Quoted 7 lines
Quoted 4 lines

I see the following possible solutions:
1) We put evtchn_fifo_destroy() in EVTCHNOP_init_control so EVTCHNOP_init_control can be called twice. No EVTCHNOP_reset is required in that case.

EVTCHNOP_init_control is called for each VCPU so I can't see how this would work.

Right, forgot about that...

Quoted 13 lines
Quoted 5 lines

2) Introduce special (e.g. 'EVTCHNOP_fifo_destroy') hypercall to do evtchn_fifo_destroy() without closing all channels. Alternatively we can avoid closing all channels in EVTCHNOP_reset when called with DOMID_SELF (as this mode is not being used atm) -- but that would look unobvious.

I would try this. The guest prior to kexec would then:

1. Use EVTCHNOP_status to query remote end of console and xenstore event channels.

2. Loop for all event channels:

Quoted 8 lines

a. unmap pirq (if required)
b. EVTCHNOP_close

3. EVTCHNOP_fifo_destroy (the implementation of which must verify that no channels are bound).

4. EVTCHNOP_bind_interdomain to rebind the console and xenstore channels.

Yea, that's what I have now when I put evtchn_fifo_destroy() in EVTCHNOP_reset. The problem here is: we can't do EVTCHNOP_bind_interdomain after we did evtchn_fifo_destroy(), we need to call EVTCHNOP_init_control first. And we'll do that only after kexec so we won't remember what we need to remap.. The second issue is the fact that EVTCHNOP_bind_interdomain will remap store/storage channels to *some* local ports, not necessary matching hvm info (HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN)..

Would it be safe is instead of closing interdomain channels on EVTCHNOP_fifo_destroy we switch evtchn_port_ops to evtchn_port_ops_2l (so on EVTCHNOP_init_control after kexec we switch back)? I'll try prototyping this.

Thank you for your comments,

Quoted 1 line

David

Signature · 1 line
  Vitaly
#46 lore raw

Re: [Xen-devel] [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to David Vrabel
Quoted 43 lines · David Vrabel
Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 3 lines

I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).

Thank you for your comments!

The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call evtchn_fifo_destroy() so next EVTCHNOP_init_control won't fail. I was intended to put evtchn_fifo_destroy() in EVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest with EVTCHNOP_bind_interdomain (if we remember where they were mapped before) but we can't do it after we did evtchn_fifo_destroy() and we can't rebind them after kexec and performing EVTCHNOP_init_control as we can't remember where these channels were mapped to after kexec/kdump.

Quoted 7 lines
Quoted 4 lines

I see the following possible solutions:
1) We put evtchn_fifo_destroy() in EVTCHNOP_init_control so EVTCHNOP_init_control can be called twice. No EVTCHNOP_reset is required in that case.

EVTCHNOP_init_control is called for each VCPU so I can't see how this would work.

Right, forgot about that...

Quoted 13 lines
Quoted 5 lines

2) Introduce special (e.g. 'EVTCHNOP_fifo_destroy') hypercall to do evtchn_fifo_destroy() without closing all channels. Alternatively we can avoid closing all channels in EVTCHNOP_reset when called with DOMID_SELF (as this mode is not being used atm) -- but that would look unobvious.

I would try this. The guest prior to kexec would then:

1. Use EVTCHNOP_status to query remote end of console and xenstore event channels.

2. Loop for all event channels:

Quoted 8 lines

a. unmap pirq (if required)
b. EVTCHNOP_close

3. EVTCHNOP_fifo_destroy (the implementation of which must verify that no channels are bound).

4. EVTCHNOP_bind_interdomain to rebind the console and xenstore channels.

Yea, that's what I have now when I put evtchn_fifo_destroy() in EVTCHNOP_reset. The problem here is: we can't do EVTCHNOP_bind_interdomain after we did evtchn_fifo_destroy(), we need to call EVTCHNOP_init_control first. And we'll do that only after kexec so we won't remember what we need to remap.. The second issue is the fact that EVTCHNOP_bind_interdomain will remap store/storage channels to *some* local ports, not necessary matching hvm info (HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN)..

Would it be safe is instead of closing interdomain channels on EVTCHNOP_fifo_destroy we switch evtchn_port_ops to evtchn_port_ops_2l (so on EVTCHNOP_init_control after kexec we switch back)? I'll try prototyping this.

Thank you for your comments,

Quoted 1 line

David

Signature · 1 line
  Vitaly
#47 lore raw

Re: [Xen-devel] [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

Quoted 77 lines · Vitaly Kuznetsov
Quoted 43 lines · David Vrabel
Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 3 lines

I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).

Thank you for your comments!

The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call evtchn_fifo_destroy() so next EVTCHNOP_init_control won't fail. I was intended to put evtchn_fifo_destroy() in EVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest with EVTCHNOP_bind_interdomain (if we remember where they were mapped before) but we can't do it after we did evtchn_fifo_destroy() and we can't rebind them after kexec and performing EVTCHNOP_init_control as we can't remember where these channels were mapped to after kexec/kdump.

Quoted 7 lines
Quoted 4 lines

I see the following possible solutions:
1) We put evtchn_fifo_destroy() in EVTCHNOP_init_control so EVTCHNOP_init_control can be called twice. No EVTCHNOP_reset is required in that case.

EVTCHNOP_init_control is called for each VCPU so I can't see how this would work.

Right, forgot about that...

Quoted 13 lines
Quoted 5 lines

2) Introduce special (e.g. 'EVTCHNOP_fifo_destroy') hypercall to do evtchn_fifo_destroy() without closing all channels. Alternatively we can avoid closing all channels in EVTCHNOP_reset when called with DOMID_SELF (as this mode is not being used atm) -- but that would look unobvious.

I would try this. The guest prior to kexec would then:

1. Use EVTCHNOP_status to query remote end of console and xenstore event channels.

2. Loop for all event channels:

Quoted 8 lines

a. unmap pirq (if required)
b. EVTCHNOP_close

3. EVTCHNOP_fifo_destroy (the implementation of which must verify that no channels are bound).

4. EVTCHNOP_bind_interdomain to rebind the console and xenstore channels.

Quoted 8 lines

Yea, that's what I have now when I put evtchn_fifo_destroy() in EVTCHNOP_reset. The problem here is: we can't do EVTCHNOP_bind_interdomain after we did evtchn_fifo_destroy(), we need to call EVTCHNOP_init_control first. And we'll do that only after kexec so we won't remember what we need to remap.. The second issue is the fact that EVTCHNOP_bind_interdomain will remap store/storage channels to *some* local ports, not necessary matching hvm info (HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN)..

You can set the HVM params to match.

Quoted 4 lines

Would it be safe is instead of closing interdomain channels on EVTCHNOP_fifo_destroy we switch evtchn_port_ops to evtchn_port_ops_2l (so on EVTCHNOP_init_control after kexec we switch back)? I'll try prototyping this.

Switching the ops back to the 2l ones is an essential part of fifo_destroy (the execed kernel might not have fifo support). I assumed you were already doing this.

David

#48 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to Vitaly Kuznetsov
Quoted 77 lines · Vitaly Kuznetsov
Quoted 43 lines · David Vrabel
Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 3 lines

I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).

Thank you for your comments!

The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call evtchn_fifo_destroy() so next EVTCHNOP_init_control won't fail. I was intended to put evtchn_fifo_destroy() in EVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest with EVTCHNOP_bind_interdomain (if we remember where they were mapped before) but we can't do it after we did evtchn_fifo_destroy() and we can't rebind them after kexec and performing EVTCHNOP_init_control as we can't remember where these channels were mapped to after kexec/kdump.

Quoted 7 lines
Quoted 4 lines

I see the following possible solutions:
1) We put evtchn_fifo_destroy() in EVTCHNOP_init_control so EVTCHNOP_init_control can be called twice. No EVTCHNOP_reset is required in that case.

EVTCHNOP_init_control is called for each VCPU so I can't see how this would work.

Right, forgot about that...

Quoted 13 lines
Quoted 5 lines

2) Introduce special (e.g. 'EVTCHNOP_fifo_destroy') hypercall to do evtchn_fifo_destroy() without closing all channels. Alternatively we can avoid closing all channels in EVTCHNOP_reset when called with DOMID_SELF (as this mode is not being used atm) -- but that would look unobvious.

I would try this. The guest prior to kexec would then:

1. Use EVTCHNOP_status to query remote end of console and xenstore event channels.

2. Loop for all event channels:

Quoted 8 lines

a. unmap pirq (if required)
b. EVTCHNOP_close

3. EVTCHNOP_fifo_destroy (the implementation of which must verify that no channels are bound).

4. EVTCHNOP_bind_interdomain to rebind the console and xenstore channels.

Quoted 8 lines

Yea, that's what I have now when I put evtchn_fifo_destroy() in EVTCHNOP_reset. The problem here is: we can't do EVTCHNOP_bind_interdomain after we did evtchn_fifo_destroy(), we need to call EVTCHNOP_init_control first. And we'll do that only after kexec so we won't remember what we need to remap.. The second issue is the fact that EVTCHNOP_bind_interdomain will remap store/storage channels to *some* local ports, not necessary matching hvm info (HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN)..

You can set the HVM params to match.

Quoted 4 lines

Would it be safe is instead of closing interdomain channels on EVTCHNOP_fifo_destroy we switch evtchn_port_ops to evtchn_port_ops_2l (so on EVTCHNOP_init_control after kexec we switch back)? I'll try prototyping this.

Switching the ops back to the 2l ones is an essential part of fifo_destroy (the execed kernel might not have fifo support). I assumed you were already doing this.

David

#49 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to Vitaly Kuznetsov
Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

Quoted 3 lines

I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).

Thank you for your comments!

The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call evtchn_fifo_destroy() so next EVTCHNOP_init_control won't fail. I was intended to put evtchn_fifo_destroy() in EVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest with EVTCHNOP_bind_interdomain (if we remember where they were mapped before) but we can't do it after we did evtchn_fifo_destroy() and we can't rebind them after kexec and performing EVTCHNOP_init_control as we can't remember where these channels were mapped to after kexec/kdump.

Quoted 4 lines

I see the following possible solutions:
1) We put evtchn_fifo_destroy() in EVTCHNOP_init_control so EVTCHNOP_init_control can be called twice. No EVTCHNOP_reset is required in that case.

EVTCHNOP_init_control is called for each VCPU so I can't see how this would work.

Quoted 4 lines

2) Introduce special (e.g. 'EVTCHNOP_fifo_destroy') hypercall to do evtchn_fifo_destroy() without closing all channels. Alternatively we can avoid closing all channels in EVTCHNOP_reset when called with DOMID_SELF (as this mode is not being used atm) -- but that would look unobvious.

I would try this. The guest prior to kexec would then:

1. Use EVTCHNOP_status to query remote end of console and xenstore event channels.

2. Loop for all event channels:

a. unmap pirq (if required)
b. EVTCHNOP_close

3. EVTCHNOP_fifo_destroy (the implementation of which must verify that no channels are bound).

4. EVTCHNOP_bind_interdomain to rebind the console and xenstore channels.

David

#50 lore raw

Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown

In reply to Vitaly Kuznetsov
Quoted 13 lines · Vitaly Kuznetsov

When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:

(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...

and the following in new kernel's dmesg:

 [   92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines

The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).

I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).

Quoted 20 lines
arch/x86/xen/smp.c +1 -0
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)
 #ifdef CONFIG_KEXEC
 	if (!kexec_in_progress)
 		return;
+	xen_unmap_all_pirqs();
 #endif
 }
 
drivers/xen/events/events_base.c +2 -0
index c919d3d..7701c7f 100644
@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}
 static bool fifo_events = true;
 module_param(fifo_events, bool, 0);
 
+void xen_unmap_all_pirqs(void)
+{

drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1657

Quoted 10 lines
drivers/xen/events/events_base.c +10 -0
+	int pirq, rc, gsi, irq, evtchn;
+	struct physdev_unmap_pirq unmap_irq;
+	struct irq_info *info;
+	struct evtchn_close close;
+
+	mutex_lock(&irq_mapping_update_lock);
+
+	list_for_each_entry(info, &xen_irq_list_head, list) {
+		if (info->type != IRQT_PIRQ)
+			continue;

I think you need to do this by querying Xen state rather than relying on potentially bad guest state. Particularly since you may crash while holding irq_mapping_update_lock.

EVTCHNOP_status gets you the info you need I think.

David

#51 lore raw

RFC PATCH 4/4 xen/pvhvm: Make MSI IRQs work after kexec

In reply to Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
#52 lore raw

RFC PATCH 4/4 xen/pvhvm: Make MSI IRQs work after kexec

In reply to Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
#53 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

receive

How come '__write_msi_msg' does not result in new MSI IRQs?

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
#54 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

In reply to Vitaly Kuznetsov
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

receive

How come '__write_msi_msg' does not result in new MSI IRQs?

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
#55 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

receive

Thanks for your comments!

Quoted 2 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

Quoted 4 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are some cases when we see 'crap' instead of pirq here?

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

Thanks for your comments again!

Quoted 23 lines
Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#56 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

In reply to Konrad Rzeszutek Wilk
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

receive

Thanks for your comments!

Quoted 2 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

Quoted 4 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are some cases when we see 'crap' instead of pirq here?

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

Thanks for your comments again!

Quoted 23 lines
Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#57 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

Quoted 4 lines
Quoted 2 lines

receive

Thanks for your comments!

Thank you for quick turnaround with the answers!

Quoted 30 lines
Quoted 3 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

Quoted 7 lines

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

I am wondering if that is related to in QEMU traditional:

qemu-xen-trad: free all the pirqs for msi/msix when driver unloads

(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)

If you have that patch in, is the PIRQ value correctly updated?

Quoted 23 lines
Quoted 5 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 5 lines
Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are

Good point.

Quoted 1 line

some cases when we see 'crap' instead of pirq here?

For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)

Quoted 3 lines

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

I concur.

Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?

Quoted 29 lines

Thanks for your comments again!

Quoted 23 lines
Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#58 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

In reply to Vitaly Kuznetsov
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

Quoted 4 lines
Quoted 2 lines

receive

Thanks for your comments!

Thank you for quick turnaround with the answers!

Quoted 30 lines
Quoted 3 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

Quoted 7 lines

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

I am wondering if that is related to in QEMU traditional:

qemu-xen-trad: free all the pirqs for msi/msix when driver unloads

(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)

If you have that patch in, is the PIRQ value correctly updated?

Quoted 23 lines
Quoted 5 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 5 lines
Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are

Good point.

Quoted 1 line

some cases when we see 'crap' instead of pirq here?

For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)

Quoted 3 lines

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

I concur.

Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?

Quoted 29 lines

Thanks for your comments again!

Quoted 23 lines
Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
#59 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

Quoted 54 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

Quoted 4 lines
Quoted 2 lines

receive

Thanks for your comments!

Thank you for quick turnaround with the answers!

Quoted 30 lines
Quoted 3 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

Quoted 7 lines

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

Quoted 8 lines

I am wondering if that is related to in QEMU traditional:

qemu-xen-trad: free all the pirqs for msi/msix when driver unloads

(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)

If you have that patch in, is the PIRQ value correctly updated?

Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?

Quoted 29 lines
Quoted 23 lines
Quoted 5 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 5 lines
Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are

Quoted 7 lines

Good point.

Quoted 1 line

some cases when we see 'crap' instead of pirq here?

For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)

AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.

Quoted 39 lines
Quoted 4 lines

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

I concur.

Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?

Quoted 29 lines

Thanks for your comments again!

Quoted 23 lines
Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
#60 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

In reply to Konrad Rzeszutek Wilk
Quoted 54 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

Quoted 4 lines
Quoted 2 lines

receive

Thanks for your comments!

Thank you for quick turnaround with the answers!

Quoted 30 lines
Quoted 3 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

Quoted 7 lines

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

Quoted 8 lines

I am wondering if that is related to in QEMU traditional:

qemu-xen-trad: free all the pirqs for msi/msix when driver unloads

(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)

If you have that patch in, is the PIRQ value correctly updated?

Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?

Quoted 29 lines
Quoted 23 lines
Quoted 5 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 5 lines
Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are

Quoted 7 lines

Good point.

Quoted 1 line

some cases when we see 'crap' instead of pirq here?

For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)

AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.

Quoted 39 lines
Quoted 4 lines

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

I concur.

Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?

Quoted 29 lines

Thanks for your comments again!

Quoted 23 lines
Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
#61 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

Quoted 60 lines · Vitaly Kuznetsov
Quoted 58 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

Quoted 4 lines
Quoted 2 lines

receive

Thanks for your comments!

Thank you for quick turnaround with the answers!

Quoted 30 lines
Quoted 3 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

Quoted 7 lines

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

I am wondering if that is related to in QEMU traditional:

qemu-xen-trad: free all the pirqs for msi/msix when driver unloads

Quoted 7 lines
Quoted 4 lines

(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)

If you have that patch in, is the PIRQ value correctly updated?

Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?

Without your patch on older QEMU's with PCI passthrough we won't get any more interrupts after we kexec in the guest right?

As in, this issue happens _only_ with PCI passthrough devices that use MSI or MSI-X?

Still need to get Stefano's view on this.

Quoted 83 lines
Quoted 30 lines
Quoted 24 lines
Quoted 6 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 5 lines
Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are

Quoted 7 lines

Good point.

Quoted 1 line

some cases when we see 'crap' instead of pirq here?

For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)

AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.

Quoted 39 lines
Quoted 4 lines

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

I concur.

Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?

Quoted 29 lines

Thanks for your comments again!

Quoted 23 lines
Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
#62 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

In reply to Vitaly Kuznetsov
Quoted 60 lines · Vitaly Kuznetsov
Quoted 58 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

Quoted 4 lines
Quoted 2 lines

receive

Thanks for your comments!

Thank you for quick turnaround with the answers!

Quoted 30 lines
Quoted 3 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

Quoted 7 lines

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

I am wondering if that is related to in QEMU traditional:

qemu-xen-trad: free all the pirqs for msi/msix when driver unloads

Quoted 7 lines
Quoted 4 lines

(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)

If you have that patch in, is the PIRQ value correctly updated?

Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?

Without your patch on older QEMU's with PCI passthrough we won't get any more interrupts after we kexec in the guest right?

As in, this issue happens _only_ with PCI passthrough devices that use MSI or MSI-X?

Still need to get Stefano's view on this.

Quoted 83 lines
Quoted 30 lines
Quoted 24 lines
Quoted 6 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 5 lines
Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are

Quoted 7 lines

Good point.

Quoted 1 line

some cases when we see 'crap' instead of pirq here?

For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)

AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.

Quoted 39 lines
Quoted 4 lines

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

I concur.

Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?

Quoted 29 lines

Thanks for your comments again!

Quoted 23 lines
Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
#63 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

Quoted 63 lines · Konrad Rzeszutek Wilk
Quoted 62 lines · Vitaly Kuznetsov
Quoted 60 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

Quoted 4 lines
Quoted 2 lines

receive

Thanks for your comments!

Thank you for quick turnaround with the answers!

Quoted 30 lines
Quoted 3 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

Quoted 7 lines

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

I am wondering if that is related to in QEMU traditional:

qemu-xen-trad: free all the pirqs for msi/msix when driver unloads

(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

If you have that patch in, is the PIRQ value correctly updated?

Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?

Without your patch on older QEMU's with PCI passthrough we won't get any more interrupts after we kexec in the guest right?

Correct.

Quoted 2 lines

As in, this issue happens _only_ with PCI passthrough devices that use MSI or MSI-X?

I haven't tested MSI-X but in theory yes, only MSI and MSI-X passthrough-ed devices are affected.

Quoted 3 lines

Still need to get Stefano's view on this.

Sure, thanks!

Quoted 83 lines
Quoted 83 lines
Quoted 30 lines
Quoted 24 lines
Quoted 6 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 5 lines
Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are

Quoted 7 lines

Good point.

Quoted 1 line

some cases when we see 'crap' instead of pirq here?

For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)

AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.

Quoted 39 lines
Quoted 4 lines

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

I concur.

Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?

Quoted 29 lines

Thanks for your comments again!

Quoted 23 lines
Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
#64 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

In reply to Konrad Rzeszutek Wilk
Quoted 63 lines · Konrad Rzeszutek Wilk
Quoted 62 lines · Vitaly Kuznetsov
Quoted 60 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

Quoted 4 lines
Quoted 2 lines

receive

Thanks for your comments!

Thank you for quick turnaround with the answers!

Quoted 30 lines
Quoted 3 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

Quoted 7 lines

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

I am wondering if that is related to in QEMU traditional:

qemu-xen-trad: free all the pirqs for msi/msix when driver unloads

(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)

Quoted 9 lines
Quoted 5 lines
Quoted 2 lines

If you have that patch in, is the PIRQ value correctly updated?

Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?

Without your patch on older QEMU's with PCI passthrough we won't get any more interrupts after we kexec in the guest right?

Correct.

Quoted 2 lines

As in, this issue happens _only_ with PCI passthrough devices that use MSI or MSI-X?

I haven't tested MSI-X but in theory yes, only MSI and MSI-X passthrough-ed devices are affected.

Quoted 3 lines

Still need to get Stefano's view on this.

Sure, thanks!

Quoted 83 lines
Quoted 83 lines
Quoted 30 lines
Quoted 24 lines
Quoted 6 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 5 lines
Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are

Quoted 7 lines

Good point.

Quoted 1 line

some cases when we see 'crap' instead of pirq here?

For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)

AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.

Quoted 39 lines
Quoted 4 lines

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

I concur.

Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?

Quoted 29 lines

Thanks for your comments again!

Quoted 23 lines
Quoted 23 lines
---
 arch/x86/pci/xen.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
index 905956f..685e8f1 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA || pirq <= 0) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
Signature · 1 line
1.9.3
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
Signature · 1 line
  Vitaly
#65 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

In reply to Vitaly Kuznetsov
Quoted 3 lines · Vitaly Kuznetsov

Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?

I don't think so.

David

#66 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

In reply to Vitaly Kuznetsov
Quoted 3 lines · Vitaly Kuznetsov

Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?

I don't think so.

David

#67 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

In reply to Konrad Rzeszutek Wilk
Quoted 100 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

Quoted 4 lines
Quoted 2 lines

receive

Thanks for your comments!

Thank you for quick turnaround with the answers!

Quoted 30 lines
Quoted 3 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

Quoted 7 lines

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

I am wondering if that is related to in QEMU traditional:

qemu-xen-trad: free all the pirqs for msi/msix when driver unloads

(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)

If you have that patch in, is the PIRQ value correctly updated?

Quoted 23 lines
Quoted 5 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 5 lines
Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are

Good point.

Quoted 1 line

some cases when we see 'crap' instead of pirq here?

For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

I concur.

Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?

If I recall correctly the problem is that pirq == 0 is a valid pirq number. So the check pirq <= 0 is wrong.

Can we rely on the fact that msg.data is always 0 on first read? If so, then we could simply:

arch/x86/pci/xen.c +1 -2
index 905956f..d824743 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
#68 lore raw

Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec

In reply to Konrad Rzeszutek Wilk
Quoted 100 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov

When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.

Quoted 4 lines
Quoted 2 lines

receive

Thanks for your comments!

Thank you for quick turnaround with the answers!

Quoted 30 lines
Quoted 3 lines

How come '__write_msi_msg' does not result in new MSI IRQs?

Actually that was the hidden question in my RFC :-)

Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():

__read_msi_msg()
pirq -> 0

then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54

and we have the following mapping:
xen: msi --> pirq=54 --> irq=72

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),

After kexec we see the following:
__read_msi_msg()
pirq -> 54

but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55

Quoted 7 lines

and we have the following mapping:
xen: msi --> pirq=55 --> irq=75

However (afaict) mapping in xen wasn't updated:

in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),

I am wondering if that is related to in QEMU traditional:

qemu-xen-trad: free all the pirqs for msi/msix when driver unloads

(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)

If you have that patch in, is the PIRQ value correctly updated?

Quoted 23 lines
Quoted 5 lines

Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?

Yea, 'always trust the device'.

Quoted 10 lines
Quoted 6 lines

RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.

We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.

I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.

Quoted 5 lines
Quoted 3 lines

In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.

But if we're called twice we'll see the same pirq, right? Or there are

Good point.

Quoted 1 line

some cases when we see 'crap' instead of pirq here?

For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)

Quoted 1 line
Quoted 7 lines
Quoted 2 lines

I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.

I concur.

Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?

If I recall correctly the problem is that pirq == 0 is a valid pirq number. So the check pirq <= 0 is wrong.

Can we rely on the fact that msg.data is always 0 on first read? If so, then we could simply:

arch/x86/pci/xen.c +1 -2
index 905956f..d824743 100644
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
 		__read_msi_msg(msidesc, &msg);
 		pirq = MSI_ADDR_EXT_DEST_ID(msg.address_hi) |
 			((msg.address_lo >> MSI_ADDR_DEST_ID_SHIFT) & 0xff);
-		if (msg.data != XEN_PIRQ_MSI_DATA ||
-		    xen_irq_from_pirq(pirq) < 0) {
+		if (msg.data != XEN_PIRQ_MSI_DATA) {
 			pirq = xen_allocate_pirq_msi(dev, msidesc);
 			if (pirq < 0) {
 				irq = -ENODEV;
#69 lore raw
In reply to Vitaly Kuznetsov
Quoted 8 lines · Vitaly Kuznetsov

With this patch series I'm trying to address several issues with kexec on pvhvm:
- shared_info issue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)

This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.

In addition to the fixes for multi-VCPU guests, what else remains?

What's the plan for handling granted pages?

I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.

David

#70 lore raw
In reply to Vitaly Kuznetsov
Quoted 8 lines · Vitaly Kuznetsov

With this patch series I'm trying to address several issues with kexec on pvhvm:
- shared_info issue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)

This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.

In addition to the fixes for multi-VCPU guests, what else remains?

What's the plan for handling granted pages?

I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.

David

#71 lore raw
Quoted 7 lines · David Vrabel
Quoted 6 lines · Vitaly Kuznetsov

With this patch series I'm trying to address several issues with kexec on pvhvm:
- shared_info issue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)

Quoted 5 lines
Quoted 2 lines

This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.

In addition to the fixes for multi-VCPU guests, what else remains?

I'm aware of grants and ballooned out pages.

Quoted 2 lines

What's the plan for handling granted pages?

(if I got the design right) we have two issues:

1) Pages we grant access to other domains. We have the list so we can try doing gnttab_end_foreign_access for all unmapped grants but there is nothing we can do with mapped ones from guest. We can either assume that all such usages are short-term and try waiting for them to finish or we need to do something like force-unmap from hypervisor side.

2) Pages we mapped from other domains. There is no easy way to collect all grant handles from different places in kernel atm so I can see two possible solutions:
- we keep track of all handles with new kernel structure in guest and unmap them all on kexec/kdump.
- we introduce new GNTTABOP_reset which does something similar to gnttab_release_mappings().

There is nothing we need to do with transferred grants (and I don't see transfer usages in kernel).

Please correct me if I'm wrong.

Quoted 2 lines

I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.

*I think* we can leave ballooned out pages out of scope for now.

Thanks,

Quoted 2 lines

David

Signature · 1 line
  Vitaly
#72 lore raw
In reply to David Vrabel
Quoted 7 lines · David Vrabel
Quoted 6 lines · Vitaly Kuznetsov

With this patch series I'm trying to address several issues with kexec on pvhvm:
- shared_info issue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)

Quoted 5 lines
Quoted 2 lines

This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.

In addition to the fixes for multi-VCPU guests, what else remains?

I'm aware of grants and ballooned out pages.

Quoted 2 lines

What's the plan for handling granted pages?

(if I got the design right) we have two issues:

1) Pages we grant access to other domains. We have the list so we can try doing gnttab_end_foreign_access for all unmapped grants but there is nothing we can do with mapped ones from guest. We can either assume that all such usages are short-term and try waiting for them to finish or we need to do something like force-unmap from hypervisor side.

2) Pages we mapped from other domains. There is no easy way to collect all grant handles from different places in kernel atm so I can see two possible solutions:
- we keep track of all handles with new kernel structure in guest and unmap them all on kexec/kdump.
- we introduce new GNTTABOP_reset which does something similar to gnttab_release_mappings().

There is nothing we need to do with transferred grants (and I don't see transfer usages in kernel).

Please correct me if I'm wrong.

Quoted 2 lines

I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.

*I think* we can leave ballooned out pages out of scope for now.

Thanks,

Quoted 2 lines

David

Signature · 1 line
  Vitaly
#73 lore raw
Quoted 20 lines · Vitaly Kuznetsov
Quoted 7 lines · David Vrabel
Quoted 6 lines · Vitaly Kuznetsov

With this patch series I'm trying to address several issues with kexec on pvhvm:
- shared_info issue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)

Quoted 5 lines
Quoted 2 lines

This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.

In addition to the fixes for multi-VCPU guests, what else remains?

I'm aware of grants and ballooned out pages.

Quoted 2 lines

What's the plan for handling granted pages?

Quoted 7 lines

(if I got the design right) we have two issues:

1) Pages we grant access to other domains. We have the list so we can try doing gnttab_end_foreign_access for all unmapped grants but there is nothing we can do with mapped ones from guest. We can either assume that all such usages are short-term and try waiting for them to finish or we need to do something like force-unmap from hypervisor side.

Shared rings and persistent grants (used in blkfront) remain mapped for long periods so just waiting won't work.

Force unmap by the hypervisor might be a possibility but the hypervisor needs to atomically replace the grant mapping with a different valid mapping, or the force unmap would cause the backend that was accesses the pages to fault.

Every writable mapping would have to be replaced with a mapping to a unique page (to prevent information leaking between different granted pages). Read-only mappings could be replaces with a read-only mapping to shared zero page safely.

The only way I can see how to do this requires co-operation from the backend kernel -- it would need to provide replacement frames for every grant map. Xen would then use this frame when force-unmapping (revoking) the mapping.

Quoted 7 lines

2) Pages we mapped from other domains. There is no easy way to collect all grant handles from different places in kernel atm so I can see two possible solutions:
- we keep track of all handles with new kernel structure in guest and unmap them all on kexec/kdump.
- we introduce new GNTTABOP_reset which does something similar to gnttab_release_mappings().

I think you can ignore this for now -- frontend drivers do not grant map, but see suggestion about kexec_is_safe below.

Quoted 2 lines

There is nothing we need to do with transferred grants (and I don't see transfer usages in kernel).

Agreed.

Quoted 4 lines
Quoted 2 lines

I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.

*I think* we can leave ballooned out pages out of scope for now.

I'm thinking we want a global flag (kexec_is_safe) that is cleared even any unsafe operation (e.g., ballooning, grant mapping) is done by the guest. kexec load and exec can then be made to fail if this flag is clear.

This would allow you to fix only the use cases you're interested in without introducing subtle failures if someone tries an unsupported configurations.

David

#74 lore raw
Quoted 42 lines · David Vrabel
Quoted 20 lines · Vitaly Kuznetsov
Quoted 7 lines · David Vrabel
Quoted 6 lines · Vitaly Kuznetsov

With this patch series I'm trying to address several issues with kexec on pvhvm:
- shared_info issue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)

Quoted 5 lines
Quoted 2 lines

This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.

In addition to the fixes for multi-VCPU guests, what else remains?

I'm aware of grants and ballooned out pages.

Quoted 2 lines

What's the plan for handling granted pages?

Quoted 7 lines

(if I got the design right) we have two issues:

1) Pages we grant access to other domains. We have the list so we can try doing gnttab_end_foreign_access for all unmapped grants but there is nothing we can do with mapped ones from guest. We can either assume that all such usages are short-term and try waiting for them to finish or we need to do something like force-unmap from hypervisor side.

Shared rings and persistent grants (used in blkfront) remain mapped for long periods so just waiting won't work.

Force unmap by the hypervisor might be a possibility but the hypervisor needs to atomically replace the grant mapping with a different valid mapping, or the force unmap would cause the backend that was accesses the pages to fault.

Every writable mapping would have to be replaced with a mapping to a unique page (to prevent information leaking between different granted pages). Read-only mappings could be replaces with a read-only mapping to shared zero page safely.

Quoted 4 lines

The only way I can see how to do this requires co-operation from the backend kernel -- it would need to provide replacement frames for every grant map. Xen would then use this frame when force-unmapping (revoking) the mapping.

We can introduce something like GNTTABOP_safe_map_grant_ref op with such replacement frames and use then in 'force unmap' but to be honest I'd like to avoid that.. What if xen could allocate and provide replacement pages on its own? I understand we need to take some precautions against malicious domains hogging all memory with such approach..

Actually if we narrow down our case to backend/frontend usage only we can think that grants do not introduce any issue for kexec/kdump because:
- frontends are being closed on kexec so backends unmap corresponding grants. That's not true for persistent grant case as persistent grants are being unmaped only in xen_blkif_free() but not in xen_blkif_disconnect() atm. The following patch helps:

(quoted diff) +0 -0

b/drivers/block/xen-blkback/xenbus.c
index 3a8b810..54f4089 100644

drivers/block/xen-blkback/xenbus.c +0 -0
@@ -270,6 +270,9 @@ static int xen_blkif_disconnect(struct xen_blkif

*blkif)

                blkif->blk_rings.common.sring = NULL;
        }

+ /* Remove all persistent grants and the cache of ballooned
pages. */
+ xen_blkbk_free_caches(blkif);
+

        return 0;

}

@@ -281,9 +284,6 @@ static void xen_blkif_free(struct xen_blkif *blkif)

        xen_blkif_disconnect(blkif);
        xen_vbd_free(&blkif->vbd);

- /* Remove all persistent grants and the cache of ballooned

        pages. */
-       xen_blkbk_free_caches(blkif);
-
        /* Make sure everything is drained before shutting down */
        BUG_ON(blkif->persistent_gnt_c != 0);
        BUG_ON(atomic_read(&blkif->persistent_gnt_in_use) != 0);

- in kdump case our new kernel lives in its own space so we won't hit granted pages (we can read some crap when collecting memory dump but it's not any better in case these pages were unmapped)

- frontends are being reinitialized on new kernel startup, that also causes the unmap (with the patch from above applied).

So I think we need to do something with grants in two cases: they were used outside frontends/backends and we have no idea if they are going to be unmapped (can be solved with kexec_is_safe flag) and when backend got stuck and refused to close/unmap (should be rare).

Quoted 22 lines
Quoted 8 lines

2) Pages we mapped from other domains. There is no easy way to collect all grant handles from different places in kernel atm so I can see two possible solutions:
- we keep track of all handles with new kernel structure in guest and unmap them all on kexec/kdump.
- we introduce new GNTTABOP_reset which does something similar to gnttab_release_mappings().

I think you can ignore this for now -- frontend drivers do not grant map, but see suggestion about kexec_is_safe below.

Quoted 2 lines

There is nothing we need to do with transferred grants (and I don't see transfer usages in kernel).

Agreed.

Quoted 4 lines
Quoted 2 lines

I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.

*I think* we can leave ballooned out pages out of scope for now.

Quoted 7 lines

I'm thinking we want a global flag (kexec_is_safe) that is cleared even any unsafe operation (e.g., ballooning, grant mapping) is done by the guest. kexec load and exec can then be made to fail if this flag is clear.

This would allow you to fix only the use cases you're interested in without introducing subtle failures if someone tries an unsupported configurations.

Makes sense but it would also be nice to have this flag supported in kexec utility itself (so it will report something like "kexec is not safe, use '-f' to override...).

Thanks,

Quoted 2 lines

David

Signature · 1 line
  Vitaly
#75 lore raw
In reply to Vitaly Kuznetsov
Quoted 20 lines · Vitaly Kuznetsov
Quoted 7 lines · David Vrabel
Quoted 6 lines · Vitaly Kuznetsov

With this patch series I'm trying to address several issues with kexec on pvhvm:
- shared_info issue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)

Quoted 5 lines
Quoted 2 lines

This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.

In addition to the fixes for multi-VCPU guests, what else remains?

I'm aware of grants and ballooned out pages.

Quoted 2 lines

What's the plan for handling granted pages?

Quoted 7 lines

(if I got the design right) we have two issues:

1) Pages we grant access to other domains. We have the list so we can try doing gnttab_end_foreign_access for all unmapped grants but there is nothing we can do with mapped ones from guest. We can either assume that all such usages are short-term and try waiting for them to finish or we need to do something like force-unmap from hypervisor side.

Shared rings and persistent grants (used in blkfront) remain mapped for long periods so just waiting won't work.

Force unmap by the hypervisor might be a possibility but the hypervisor needs to atomically replace the grant mapping with a different valid mapping, or the force unmap would cause the backend that was accesses the pages to fault.

Every writable mapping would have to be replaced with a mapping to a unique page (to prevent information leaking between different granted pages). Read-only mappings could be replaces with a read-only mapping to shared zero page safely.

The only way I can see how to do this requires co-operation from the backend kernel -- it would need to provide replacement frames for every grant map. Xen would then use this frame when force-unmapping (revoking) the mapping.

Quoted 7 lines

2) Pages we mapped from other domains. There is no easy way to collect all grant handles from different places in kernel atm so I can see two possible solutions:
- we keep track of all handles with new kernel structure in guest and unmap them all on kexec/kdump.
- we introduce new GNTTABOP_reset which does something similar to gnttab_release_mappings().

I think you can ignore this for now -- frontend drivers do not grant map, but see suggestion about kexec_is_safe below.

Quoted 2 lines

There is nothing we need to do with transferred grants (and I don't see transfer usages in kernel).

Agreed.

Quoted 4 lines
Quoted 2 lines

I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.

*I think* we can leave ballooned out pages out of scope for now.

I'm thinking we want a global flag (kexec_is_safe) that is cleared even any unsafe operation (e.g., ballooning, grant mapping) is done by the guest. kexec load and exec can then be made to fail if this flag is clear.

This would allow you to fix only the use cases you're interested in without introducing subtle failures if someone tries an unsupported configurations.

David