xen/pvhvm: fix shared_info and pirq issues with kexec
Subsystems (7)
cover xen/pvhvm: fix shared_info and pirq issues with kexec
With this patch series I'm trying to address several issues with kexec on pvhvm:
- shared_info issue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)
This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.
Olaf Hering (1):
xen PVonHVM: use E820_Reserved area for shared_info
Vitaly Kuznetsov (3):
xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown() xen/pvhvm: Unmap all PIRQs on startup and shutdown xen/pvhvm: Make MSI IRQs work after kexec
arch/x86/pci/xen.c | 3 +- arch/x86/xen/enlighten.c | 83 +++++++++++++++++++++++++++++++--------- arch/x86/xen/smp.c | 10 +++++ arch/x86/xen/smp.h | 1 + arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 8 files changed, 158 insertions(+), 22 deletions(-)
Signature · 1 line
1.9.3
RFC PATCH 1/4 xen PVonHVM: use E820_Reserved area for shared_info
From: Olaf Hering <[email protected]>
This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").
Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.
The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.
[On resume we need to reset the xen_vcpu_info, which the original patch did not do]
--- arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------ arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL +static struct shared_info *xen_hvm_shared_info; +static unsigned long xen_hvm_sip_phys; +static int xen_major, xen_minor; + +static void xen_hvm_connect_shared_info (unsigned long pfn) { -int cpu;struct xen_add_to_physmap xatp; -static struct shared_info *shared_info_page =0 ; -if (!shared_info_page) - shared_info_page = (struct shared_info *) -extend_brk (PAGE_SIZE ,PAGE_SIZE ); xatp.domid =DOMID_SELF ; xatp.idx =0 ; xatp.space = XENMAPSPACE_shared_info; - xatp.gpfn =__pa (shared_info_page) >>PAGE_SHIFT ; + xatp.gpfn = pfn;if (HYPERVISOR_memory_op (XENMEM_add_to_physmap, &xatp))BUG (); - HYPERVISOR_shared_info = (struct shared_info *)shared_info_page; +} +static void __init xen_hvm_set_shared_info (struct shared_info *sip) +{ +int cpu; + + HYPERVISOR_shared_info = sip;/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void) } } -#ifdef CONFIG_XEN_PVHVM +/* Reconnect the shared_info pfn to a (new) mfn */ +void xen_hvm_resume_shared_info (void ) +{ +xen_hvm_connect_shared_info (xen_hvm_sip_phys >>PAGE_SHIFT ); +xen_hvm_set_shared_info (xen_hvm_shared_info); +} + +/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage. +* On these old tools the shared info page will be placed in E820_Ram. +* Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects +* that nothing is mapped up to HVM_SHARED_INFO_ADDR. +* Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used +* here for the shared info page. */ +static void __init xen_hvm_init_shared_info (void ) +{ +if (xen_major <4 ) { + xen_hvm_shared_info =extend_brk (PAGE_SIZE ,PAGE_SIZE ); + xen_hvm_sip_phys =__pa (xen_hvm_shared_info); + }else { + xen_hvm_sip_phys =HVM_SHARED_INFO_ADDR ; +set_fixmap (FIX_PARAVIRT_BOOTMAP , xen_hvm_sip_phys); + xen_hvm_shared_info = + (struct shared_info *)fix_to_virt (FIX_PARAVIRT_BOOTMAP ); + } +xen_hvm_resume_shared_info (); +} +static void __init init_hvm_pv_info (void ) { -int major, minor; -uint32_t eax, ebx, ecx, edx, pages, msr, base; +uint32_t ecx, edx, pages, msr, base;u64 pfn; base =xen_cpuid_base (); -cpuid (base +1 , &eax, &ebx, &ecx, &edx); - - major = eax >>16 ; - minor = eax &0xffff ; -printk (KERN_INFO "Xen version %d.%d.\n" , major, minor); -cpuid (base +2 , &pages, &msr, &ecx, &edx); pfn =__pa (hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base) +return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ; }bool xen_hvm_need_lapic (void )
arch/x86/xen/suspend.c +1 -1
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled) {#ifdef CONFIG_XEN_PVHVM int cpu; -xen_hvm_init_shared_info (); +xen_hvm_resume_shared_info ();xen_callback_vector ();xen_unplug_emulated_devices ();if (xen_feature (XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);void xen_vcpu_restore (void );void xen_callback_vector (void ); -void xen_hvm_init_shared_info (void ); +void xen_hvm_resume_shared_info (void );void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void );
Signature · 1 line
1.9.3
RFC PATCH 1/4 xen PVonHVM: use E820_Reserved area for shared_info
In reply to Vitaly KuznetsovFrom: Olaf Hering <[email protected]>
This is a respin of 00e37bdb0113a98408de42db85be002f21dbffd3 ("xen PVonHVM: move shared_info to MMIO before kexec").
Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the shared_info is located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.
The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the shared_info page will be placed in RAM. As a result kexec can not be used.
[On resume we need to reset the xen_vcpu_info, which the original patch did not do]
--- arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------ arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- 3 files changed, 58 insertions(+), 20 deletions(-)
arch/x86/xen/enlighten.c +56 -18
@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL +static struct shared_info *xen_hvm_shared_info; +static unsigned long xen_hvm_sip_phys; +static int xen_major, xen_minor; + +static void xen_hvm_connect_shared_info (unsigned long pfn) { -int cpu;struct xen_add_to_physmap xatp; -static struct shared_info *shared_info_page =0 ; -if (!shared_info_page) - shared_info_page = (struct shared_info *) -extend_brk (PAGE_SIZE ,PAGE_SIZE ); xatp.domid =DOMID_SELF ; xatp.idx =0 ; xatp.space = XENMAPSPACE_shared_info; - xatp.gpfn =__pa (shared_info_page) >>PAGE_SHIFT ; + xatp.gpfn = pfn;if (HYPERVISOR_memory_op (XENMEM_add_to_physmap, &xatp))BUG (); - HYPERVISOR_shared_info = (struct shared_info *)shared_info_page; +} +static void __init xen_hvm_set_shared_info (struct shared_info *sip) +{ +int cpu; + + HYPERVISOR_shared_info = sip;/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info * page, we use it in the event channel upcall and in some pvclock
@@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void) } } -#ifdef CONFIG_XEN_PVHVM +/* Reconnect the shared_info pfn to a (new) mfn */ +void xen_hvm_resume_shared_info (void ) +{ +xen_hvm_connect_shared_info (xen_hvm_sip_phys >>PAGE_SHIFT ); +xen_hvm_set_shared_info (xen_hvm_shared_info); +} + +/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage. +* On these old tools the shared info page will be placed in E820_Ram. +* Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects +* that nothing is mapped up to HVM_SHARED_INFO_ADDR. +* Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used +* here for the shared info page. */ +static void __init xen_hvm_init_shared_info (void ) +{ +if (xen_major <4 ) { + xen_hvm_shared_info =extend_brk (PAGE_SIZE ,PAGE_SIZE ); + xen_hvm_sip_phys =__pa (xen_hvm_shared_info); + }else { + xen_hvm_sip_phys =HVM_SHARED_INFO_ADDR ; +set_fixmap (FIX_PARAVIRT_BOOTMAP , xen_hvm_sip_phys); + xen_hvm_shared_info = + (struct shared_info *)fix_to_virt (FIX_PARAVIRT_BOOTMAP ); + } +xen_hvm_resume_shared_info (); +} +static void __init init_hvm_pv_info (void ) { -int major, minor; -uint32_t eax, ebx, ecx, edx, pages, msr, base; +uint32_t ecx, edx, pages, msr, base;u64 pfn; base =xen_cpuid_base (); -cpuid (base +1 , &eax, &ebx, &ecx, &edx); - - major = eax >>16 ; - minor = eax &0xffff ; -printk (KERN_INFO "Xen version %d.%d.\n" , major, minor); -cpuid (base +2 , &pages, &msr, &ecx, &edx); pfn =__pa (hypercall_page);
@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base) +return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ; }bool xen_hvm_need_lapic (void )
arch/x86/xen/suspend.c +1 -1
@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled) {#ifdef CONFIG_XEN_PVHVM int cpu; -xen_hvm_init_shared_info (); +xen_hvm_resume_shared_info ();xen_callback_vector ();xen_unplug_emulated_devices ();if (xen_feature (XENFEAT_hvm_safe_pvclock)) {
arch/x86/xen/xen-ops.h +1 -1
@@ -43,7 +43,7 @@ void xen_enable_syscall(void);void xen_vcpu_restore (void );void xen_callback_vector (void ); -void xen_hvm_init_shared_info (void ); +void xen_hvm_resume_shared_info (void );void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void );
Signature · 1 line
1.9.3
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.
So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9
This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.
We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.
Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Quoted 160 lines
Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)[On resume we need to reset the
xen_vcpu_info, which the original patch did not do]Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------ arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- 3 files changed, 58 insertions(+), 20 deletions(-)arch/x86/xen/enlighten.c +56
-18@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL +static struct shared_info *xen_hvm_shared_info; +static unsigned long xen_hvm_sip_phys; +static int xen_major, xen_minor; + +static void xen_hvm_connect_shared_info (unsigned long pfn) { -int cpu;struct xen_add_to_physmap xatp; -static struct shared_info *shared_info_page =0 ; -if (!shared_info_page) - shared_info_page = (struct shared_info *) -extend_brk (PAGE_SIZE ,PAGE_SIZE ); xatp.domid =DOMID_SELF ; xatp.idx =0 ; xatp.space = XENMAPSPACE_shared_info; - xatp.gpfn =__pa (shared_info_page) >>PAGE_SHIFT ; + xatp.gpfn = pfn;if (HYPERVISOR_memory_op (XENMEM_add_to_physmap, &xatp))BUG (); - HYPERVISOR_shared_info = (struct shared_info *)shared_info_page; +} +static void __init xen_hvm_set_shared_info (struct shared_info *sip) +{ +int cpu; + + HYPERVISOR_shared_info = sip;/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info * page, we use it in the event channel upcall and in some pvclock @@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void) } } -#ifdef CONFIG_XEN_PVHVM +/* Reconnect the shared_info pfn to a (new) mfn */ +void xen_hvm_resume_shared_info (void ) +{ +xen_hvm_connect_shared_info (xen_hvm_sip_phys >>PAGE_SHIFT ); +xen_hvm_set_shared_info (xen_hvm_shared_info); +} + +/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage. +* On these old tools the shared info page will be placed in E820_Ram. +* Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects +* that nothing is mapped up to HVM_SHARED_INFO_ADDR. +* Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used +* here for the shared info page. */ +static void __init xen_hvm_init_shared_info (void ) +{ +if (xen_major <4 ) { + xen_hvm_shared_info =extend_brk (PAGE_SIZE ,PAGE_SIZE ); + xen_hvm_sip_phys =__pa (xen_hvm_shared_info); + }else { + xen_hvm_sip_phys =HVM_SHARED_INFO_ADDR ; +set_fixmap (FIX_PARAVIRT_BOOTMAP , xen_hvm_sip_phys); + xen_hvm_shared_info = + (struct shared_info *)fix_to_virt (FIX_PARAVIRT_BOOTMAP ); + } +xen_hvm_resume_shared_info (); +} +static void __init init_hvm_pv_info (void ) { -int major, minor; -uint32_t eax, ebx, ecx, edx, pages, msr, base; +uint32_t ecx, edx, pages, msr, base;u64 pfn; base =xen_cpuid_base (); -cpuid (base +1 , &eax, &ebx, &ecx, &edx); - - major = eax >>16 ; - minor = eax &0xffff ; -printk (KERN_INFO "Xen version %d.%d.\n" , major, minor); -cpuid (base +2 , &pages, &msr, &ecx, &edx); pfn =__pa (hypercall_page);@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base) +return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ; }bool xen_hvm_need_lapic (void )arch/x86/xen/suspend.c +1
-1@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled) {#ifdef CONFIG_XEN_PVHVM int cpu; -xen_hvm_init_shared_info (); +xen_hvm_resume_shared_info ();xen_callback_vector ();xen_unplug_emulated_devices ();if (xen_feature (XENFEAT_hvm_safe_pvclock)) {arch/x86/xen/xen-ops.h +1
-1@@ -43,7 +43,7 @@ void xen_enable_syscall(void);void xen_vcpu_restore (void );void xen_callback_vector (void ); -void xen_hvm_init_shared_info (void ); +void xen_hvm_resume_shared_info (void );void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void );Signature · 1 line
1.9.3
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
In reply to Vitaly KuznetsovQuoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.
So this looks right, the one thing that we really need to check is e9daff24a266307943457086533041bd971d0ef9
This reverts commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.
We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.
Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Quoted 160 lines
Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)[On resume we need to reset the
xen_vcpu_info, which the original patch did not do]Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------ arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- 3 files changed, 58 insertions(+), 20 deletions(-)arch/x86/xen/enlighten.c +56
-18@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL +static struct shared_info *xen_hvm_shared_info; +static unsigned long xen_hvm_sip_phys; +static int xen_major, xen_minor; + +static void xen_hvm_connect_shared_info (unsigned long pfn) { -int cpu;struct xen_add_to_physmap xatp; -static struct shared_info *shared_info_page =0 ; -if (!shared_info_page) - shared_info_page = (struct shared_info *) -extend_brk (PAGE_SIZE ,PAGE_SIZE ); xatp.domid =DOMID_SELF ; xatp.idx =0 ; xatp.space = XENMAPSPACE_shared_info; - xatp.gpfn =__pa (shared_info_page) >>PAGE_SHIFT ; + xatp.gpfn = pfn;if (HYPERVISOR_memory_op (XENMEM_add_to_physmap, &xatp))BUG (); - HYPERVISOR_shared_info = (struct shared_info *)shared_info_page; +} +static void __init xen_hvm_set_shared_info (struct shared_info *sip) +{ +int cpu; + + HYPERVISOR_shared_info = sip;/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info * page, we use it in the event channel upcall and in some pvclock @@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void) } } -#ifdef CONFIG_XEN_PVHVM +/* Reconnect the shared_info pfn to a (new) mfn */ +void xen_hvm_resume_shared_info (void ) +{ +xen_hvm_connect_shared_info (xen_hvm_sip_phys >>PAGE_SHIFT ); +xen_hvm_set_shared_info (xen_hvm_shared_info); +} + +/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage. +* On these old tools the shared info page will be placed in E820_Ram. +* Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects +* that nothing is mapped up to HVM_SHARED_INFO_ADDR. +* Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used +* here for the shared info page. */ +static void __init xen_hvm_init_shared_info (void ) +{ +if (xen_major <4 ) { + xen_hvm_shared_info =extend_brk (PAGE_SIZE ,PAGE_SIZE ); + xen_hvm_sip_phys =__pa (xen_hvm_shared_info); + }else { + xen_hvm_sip_phys =HVM_SHARED_INFO_ADDR ; +set_fixmap (FIX_PARAVIRT_BOOTMAP , xen_hvm_sip_phys); + xen_hvm_shared_info = + (struct shared_info *)fix_to_virt (FIX_PARAVIRT_BOOTMAP ); + } +xen_hvm_resume_shared_info (); +} +static void __init init_hvm_pv_info (void ) { -int major, minor; -uint32_t eax, ebx, ecx, edx, pages, msr, base; +uint32_t ecx, edx, pages, msr, base;u64 pfn; base =xen_cpuid_base (); -cpuid (base +1 , &eax, &ebx, &ecx, &edx); - - major = eax >>16 ; - minor = eax &0xffff ; -printk (KERN_INFO "Xen version %d.%d.\n" , major, minor); -cpuid (base +2 , &pages, &msr, &ecx, &edx); pfn =__pa (hypercall_page);@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base) +return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ; }bool xen_hvm_need_lapic (void )arch/x86/xen/suspend.c +1
-1@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled) {#ifdef CONFIG_XEN_PVHVM int cpu; -xen_hvm_init_shared_info (); +xen_hvm_resume_shared_info ();xen_callback_vector ();xen_unplug_emulated_devices ();if (xen_feature (XENFEAT_hvm_safe_pvclock)) {arch/x86/xen/xen-ops.h +1
-1@@ -43,7 +43,7 @@ void xen_enable_syscall(void);void xen_vcpu_restore (void );void xen_callback_vector (void ); -void xen_hvm_init_shared_info (void ); +void xen_hvm_resume_shared_info (void );void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void );Signature · 1 line
1.9.3
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.So this looks right, the one thing that we really need to check is
e9daff24a266307943457086533041bd971d0ef9This reverts commit
9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.
Quoted 9 lines
We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the
start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.
Quoted 160 lines
Quoted 160 lines
Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)[On resume we need to reset the
xen_vcpu_info, which the original patch did not do]Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------ arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- 3 files changed, 58 insertions(+), 20 deletions(-)arch/x86/xen/enlighten.c +56
-18@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL +static struct shared_info *xen_hvm_shared_info; +static unsigned long xen_hvm_sip_phys; +static int xen_major, xen_minor; + +static void xen_hvm_connect_shared_info (unsigned long pfn) { -int cpu;struct xen_add_to_physmap xatp; -static struct shared_info *shared_info_page =0 ; -if (!shared_info_page) - shared_info_page = (struct shared_info *) -extend_brk (PAGE_SIZE ,PAGE_SIZE ); xatp.domid =DOMID_SELF ; xatp.idx =0 ; xatp.space = XENMAPSPACE_shared_info; - xatp.gpfn =__pa (shared_info_page) >>PAGE_SHIFT ; + xatp.gpfn = pfn;if (HYPERVISOR_memory_op (XENMEM_add_to_physmap, &xatp))BUG (); - HYPERVISOR_shared_info = (struct shared_info *)shared_info_page; +} +static void __init xen_hvm_set_shared_info (struct shared_info *sip) +{ +int cpu; + + HYPERVISOR_shared_info = sip;/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info * page, we use it in the event channel upcall and in some pvclock @@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void) } } -#ifdef CONFIG_XEN_PVHVM +/* Reconnect the shared_info pfn to a (new) mfn */ +void xen_hvm_resume_shared_info (void ) +{ +xen_hvm_connect_shared_info (xen_hvm_sip_phys >>PAGE_SHIFT ); +xen_hvm_set_shared_info (xen_hvm_shared_info); +} + +/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage. +* On these old tools the shared info page will be placed in E820_Ram. +* Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects +* that nothing is mapped up to HVM_SHARED_INFO_ADDR. +* Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used +* here for the shared info page. */ +static void __init xen_hvm_init_shared_info (void ) +{ +if (xen_major <4 ) { + xen_hvm_shared_info =extend_brk (PAGE_SIZE ,PAGE_SIZE ); + xen_hvm_sip_phys =__pa (xen_hvm_shared_info); + }else { + xen_hvm_sip_phys =HVM_SHARED_INFO_ADDR ; +set_fixmap (FIX_PARAVIRT_BOOTMAP , xen_hvm_sip_phys); + xen_hvm_shared_info = + (struct shared_info *)fix_to_virt (FIX_PARAVIRT_BOOTMAP ); + } +xen_hvm_resume_shared_info (); +} +static void __init init_hvm_pv_info (void ) { -int major, minor; -uint32_t eax, ebx, ecx, edx, pages, msr, base; +uint32_t ecx, edx, pages, msr, base;u64 pfn; base =xen_cpuid_base (); -cpuid (base +1 , &eax, &ebx, &ecx, &edx); - - major = eax >>16 ; - minor = eax &0xffff ; -printk (KERN_INFO "Xen version %d.%d.\n" , major, minor); -cpuid (base +2 , &pages, &msr, &ecx, &edx); pfn =__pa (hypercall_page);@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base) +return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ; }bool xen_hvm_need_lapic (void )arch/x86/xen/suspend.c +1
-1@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled) {#ifdef CONFIG_XEN_PVHVM int cpu; -xen_hvm_init_shared_info (); +xen_hvm_resume_shared_info ();xen_callback_vector ();xen_unplug_emulated_devices ();if (xen_feature (XENFEAT_hvm_safe_pvclock)) {arch/x86/xen/xen-ops.h +1
-1@@ -43,7 +43,7 @@ void xen_enable_syscall(void);void xen_vcpu_restore (void );void xen_callback_vector (void ); -void xen_hvm_init_shared_info (void ); +void xen_hvm_resume_shared_info (void );void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void );Signature · 1 line
1.9.3
Signature · 1 line
Vitaly
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
In reply to Konrad Rzeszutek WilkQuoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.So this looks right, the one thing that we really need to check is
e9daff24a266307943457086533041bd971d0ef9This reverts commit
9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.
Quoted 9 lines
We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the
start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.
Quoted 160 lines
Quoted 160 lines
Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)[On resume we need to reset the
xen_vcpu_info, which the original patch did not do]Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------ arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- 3 files changed, 58 insertions(+), 20 deletions(-)arch/x86/xen/enlighten.c +56
-18@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL +static struct shared_info *xen_hvm_shared_info; +static unsigned long xen_hvm_sip_phys; +static int xen_major, xen_minor; + +static void xen_hvm_connect_shared_info (unsigned long pfn) { -int cpu;struct xen_add_to_physmap xatp; -static struct shared_info *shared_info_page =0 ; -if (!shared_info_page) - shared_info_page = (struct shared_info *) -extend_brk (PAGE_SIZE ,PAGE_SIZE ); xatp.domid =DOMID_SELF ; xatp.idx =0 ; xatp.space = XENMAPSPACE_shared_info; - xatp.gpfn =__pa (shared_info_page) >>PAGE_SHIFT ; + xatp.gpfn = pfn;if (HYPERVISOR_memory_op (XENMEM_add_to_physmap, &xatp))BUG (); - HYPERVISOR_shared_info = (struct shared_info *)shared_info_page; +} +static void __init xen_hvm_set_shared_info (struct shared_info *sip) +{ +int cpu; + + HYPERVISOR_shared_info = sip;/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info * page, we use it in the event channel upcall and in some pvclock @@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void) } } -#ifdef CONFIG_XEN_PVHVM +/* Reconnect the shared_info pfn to a (new) mfn */ +void xen_hvm_resume_shared_info (void ) +{ +xen_hvm_connect_shared_info (xen_hvm_sip_phys >>PAGE_SHIFT ); +xen_hvm_set_shared_info (xen_hvm_shared_info); +} + +/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage. +* On these old tools the shared info page will be placed in E820_Ram. +* Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects +* that nothing is mapped up to HVM_SHARED_INFO_ADDR. +* Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used +* here for the shared info page. */ +static void __init xen_hvm_init_shared_info (void ) +{ +if (xen_major <4 ) { + xen_hvm_shared_info =extend_brk (PAGE_SIZE ,PAGE_SIZE ); + xen_hvm_sip_phys =__pa (xen_hvm_shared_info); + }else { + xen_hvm_sip_phys =HVM_SHARED_INFO_ADDR ; +set_fixmap (FIX_PARAVIRT_BOOTMAP , xen_hvm_sip_phys); + xen_hvm_shared_info = + (struct shared_info *)fix_to_virt (FIX_PARAVIRT_BOOTMAP ); + } +xen_hvm_resume_shared_info (); +} +static void __init init_hvm_pv_info (void ) { -int major, minor; -uint32_t eax, ebx, ecx, edx, pages, msr, base; +uint32_t ecx, edx, pages, msr, base;u64 pfn; base =xen_cpuid_base (); -cpuid (base +1 , &eax, &ebx, &ecx, &edx); - - major = eax >>16 ; - minor = eax &0xffff ; -printk (KERN_INFO "Xen version %d.%d.\n" , major, minor); -cpuid (base +2 , &pages, &msr, &ecx, &edx); pfn =__pa (hypercall_page);@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base) +return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ; }bool xen_hvm_need_lapic (void )arch/x86/xen/suspend.c +1
-1@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled) {#ifdef CONFIG_XEN_PVHVM int cpu; -xen_hvm_init_shared_info (); +xen_hvm_resume_shared_info ();xen_callback_vector ();xen_unplug_emulated_devices ();if (xen_feature (XENFEAT_hvm_safe_pvclock)) {arch/x86/xen/xen-ops.h +1
-1@@ -43,7 +43,7 @@ void xen_enable_syscall(void);void xen_vcpu_restore (void );void xen_callback_vector (void ); -void xen_hvm_init_shared_info (void ); +void xen_hvm_resume_shared_info (void );void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void );Signature · 1 line
1.9.3
Signature · 1 line
Vitaly
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
Quoted 40 lines · Vitaly Kuznetsov
Quoted 38 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.So this looks right, the one thing that we really need to check is
e9daff24a266307943457086533041bd971d0ef9This reverts commit
9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the
start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.
Quoted 8 lines
Quoted 3 lines
Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.
Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.
Thank you!
Quoted 164 lines
Quoted 161 lines
Quoted 161 lines
Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)[On resume we need to reset the
xen_vcpu_info, which the original patch did not do]Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------ arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- 3 files changed, 58 insertions(+), 20 deletions(-)arch/x86/xen/enlighten.c +56
-18@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL +static struct shared_info *xen_hvm_shared_info; +static unsigned long xen_hvm_sip_phys; +static int xen_major, xen_minor; + +static void xen_hvm_connect_shared_info (unsigned long pfn) { -int cpu;struct xen_add_to_physmap xatp; -static struct shared_info *shared_info_page =0 ; -if (!shared_info_page) - shared_info_page = (struct shared_info *) -extend_brk (PAGE_SIZE ,PAGE_SIZE ); xatp.domid =DOMID_SELF ; xatp.idx =0 ; xatp.space = XENMAPSPACE_shared_info; - xatp.gpfn =__pa (shared_info_page) >>PAGE_SHIFT ; + xatp.gpfn = pfn;if (HYPERVISOR_memory_op (XENMEM_add_to_physmap, &xatp))BUG (); - HYPERVISOR_shared_info = (struct shared_info *)shared_info_page; +} +static void __init xen_hvm_set_shared_info (struct shared_info *sip) +{ +int cpu; + + HYPERVISOR_shared_info = sip;/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info * page, we use it in the event channel upcall and in some pvclock @@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void) } } -#ifdef CONFIG_XEN_PVHVM +/* Reconnect the shared_info pfn to a (new) mfn */ +void xen_hvm_resume_shared_info (void ) +{ +xen_hvm_connect_shared_info (xen_hvm_sip_phys >>PAGE_SHIFT ); +xen_hvm_set_shared_info (xen_hvm_shared_info); +} + +/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage. +* On these old tools the shared info page will be placed in E820_Ram. +* Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects +* that nothing is mapped up to HVM_SHARED_INFO_ADDR. +* Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used +* here for the shared info page. */ +static void __init xen_hvm_init_shared_info (void ) +{ +if (xen_major <4 ) { + xen_hvm_shared_info =extend_brk (PAGE_SIZE ,PAGE_SIZE ); + xen_hvm_sip_phys =__pa (xen_hvm_shared_info); + }else { + xen_hvm_sip_phys =HVM_SHARED_INFO_ADDR ; +set_fixmap (FIX_PARAVIRT_BOOTMAP , xen_hvm_sip_phys); + xen_hvm_shared_info = + (struct shared_info *)fix_to_virt (FIX_PARAVIRT_BOOTMAP ); + } +xen_hvm_resume_shared_info (); +} +static void __init init_hvm_pv_info (void ) { -int major, minor; -uint32_t eax, ebx, ecx, edx, pages, msr, base; +uint32_t ecx, edx, pages, msr, base;u64 pfn; base =xen_cpuid_base (); -cpuid (base +1 , &eax, &ebx, &ecx, &edx); - - major = eax >>16 ; - minor = eax &0xffff ; -printk (KERN_INFO "Xen version %d.%d.\n" , major, minor); -cpuid (base +2 , &pages, &msr, &ecx, &edx); pfn =__pa (hypercall_page);@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base) +return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ; }bool xen_hvm_need_lapic (void )arch/x86/xen/suspend.c +1
-1@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled) {#ifdef CONFIG_XEN_PVHVM int cpu; -xen_hvm_init_shared_info (); +xen_hvm_resume_shared_info ();xen_callback_vector ();xen_unplug_emulated_devices ();if (xen_feature (XENFEAT_hvm_safe_pvclock)) {arch/x86/xen/xen-ops.h +1
-1@@ -43,7 +43,7 @@ void xen_enable_syscall(void);void xen_vcpu_restore (void );void xen_callback_vector (void ); -void xen_hvm_init_shared_info (void ); +void xen_hvm_resume_shared_info (void );void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void );Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
In reply to Vitaly KuznetsovQuoted 40 lines · Vitaly Kuznetsov
Quoted 38 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.So this looks right, the one thing that we really need to check is
e9daff24a266307943457086533041bd971d0ef9This reverts commit
9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the
start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.
Quoted 8 lines
Quoted 3 lines
Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.
Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.
Thank you!
Quoted 164 lines
Quoted 161 lines
Quoted 161 lines
Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)[On resume we need to reset the
xen_vcpu_info, which the original patch did not do]Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------ arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- 3 files changed, 58 insertions(+), 20 deletions(-)arch/x86/xen/enlighten.c +56
-18@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL +static struct shared_info *xen_hvm_shared_info; +static unsigned long xen_hvm_sip_phys; +static int xen_major, xen_minor; + +static void xen_hvm_connect_shared_info (unsigned long pfn) { -int cpu;struct xen_add_to_physmap xatp; -static struct shared_info *shared_info_page =0 ; -if (!shared_info_page) - shared_info_page = (struct shared_info *) -extend_brk (PAGE_SIZE ,PAGE_SIZE ); xatp.domid =DOMID_SELF ; xatp.idx =0 ; xatp.space = XENMAPSPACE_shared_info; - xatp.gpfn =__pa (shared_info_page) >>PAGE_SHIFT ; + xatp.gpfn = pfn;if (HYPERVISOR_memory_op (XENMEM_add_to_physmap, &xatp))BUG (); - HYPERVISOR_shared_info = (struct shared_info *)shared_info_page; +} +static void __init xen_hvm_set_shared_info (struct shared_info *sip) +{ +int cpu; + + HYPERVISOR_shared_info = sip;/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info * page, we use it in the event channel upcall and in some pvclock @@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void) } } -#ifdef CONFIG_XEN_PVHVM +/* Reconnect the shared_info pfn to a (new) mfn */ +void xen_hvm_resume_shared_info (void ) +{ +xen_hvm_connect_shared_info (xen_hvm_sip_phys >>PAGE_SHIFT ); +xen_hvm_set_shared_info (xen_hvm_shared_info); +} + +/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage. +* On these old tools the shared info page will be placed in E820_Ram. +* Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects +* that nothing is mapped up to HVM_SHARED_INFO_ADDR. +* Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used +* here for the shared info page. */ +static void __init xen_hvm_init_shared_info (void ) +{ +if (xen_major <4 ) { + xen_hvm_shared_info =extend_brk (PAGE_SIZE ,PAGE_SIZE ); + xen_hvm_sip_phys =__pa (xen_hvm_shared_info); + }else { + xen_hvm_sip_phys =HVM_SHARED_INFO_ADDR ; +set_fixmap (FIX_PARAVIRT_BOOTMAP , xen_hvm_sip_phys); + xen_hvm_shared_info = + (struct shared_info *)fix_to_virt (FIX_PARAVIRT_BOOTMAP ); + } +xen_hvm_resume_shared_info (); +} +static void __init init_hvm_pv_info (void ) { -int major, minor; -uint32_t eax, ebx, ecx, edx, pages, msr, base; +uint32_t ecx, edx, pages, msr, base;u64 pfn; base =xen_cpuid_base (); -cpuid (base +1 , &eax, &ebx, &ecx, &edx); - - major = eax >>16 ; - minor = eax &0xffff ; -printk (KERN_INFO "Xen version %d.%d.\n" , major, minor); -cpuid (base +2 , &pages, &msr, &ecx, &edx); pfn =__pa (hypercall_page);@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base) +return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ; }bool xen_hvm_need_lapic (void )arch/x86/xen/suspend.c +1
-1@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled) {#ifdef CONFIG_XEN_PVHVM int cpu; -xen_hvm_init_shared_info (); +xen_hvm_resume_shared_info ();xen_callback_vector ();xen_unplug_emulated_devices ();if (xen_feature (XENFEAT_hvm_safe_pvclock)) {arch/x86/xen/xen-ops.h +1
-1@@ -43,7 +43,7 @@ void xen_enable_syscall(void);void xen_vcpu_restore (void );void xen_callback_vector (void ); -void xen_hvm_init_shared_info (void ); +void xen_hvm_resume_shared_info (void );void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void );Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
Quoted 45 lines · Konrad Rzeszutek Wilk
Quoted 44 lines · Vitaly Kuznetsov
Quoted 41 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.So this looks right, the one thing that we really need to check is
e9daff24a266307943457086533041bd971d0ef9This reverts commit
9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the
start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Quoted 8 lines
Quoted 4 lines
Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.
Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.
The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce that on old patch version and everything works fine with your fix (calling xen_hvm_set_shared_info() in addition to xen_hvm_connect_shared_info() on resume). We're probably safe to apply it now, thanks!
However I'd like to suggest we remove '__init' from xen_hvm_set_shared_info() as now we call it on resume.
Quoted 165 lines
Thank you!
Quoted 164 lines
Quoted 161 lines
Quoted 161 lines
Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)[On resume we need to reset the
xen_vcpu_info, which the original patch did not do]Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------ arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- 3 files changed, 58 insertions(+), 20 deletions(-)arch/x86/xen/enlighten.c +56
-18@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL +static struct shared_info *xen_hvm_shared_info; +static unsigned long xen_hvm_sip_phys; +static int xen_major, xen_minor; + +static void xen_hvm_connect_shared_info (unsigned long pfn) { -int cpu;struct xen_add_to_physmap xatp; -static struct shared_info *shared_info_page =0 ; -if (!shared_info_page) - shared_info_page = (struct shared_info *) -extend_brk (PAGE_SIZE ,PAGE_SIZE ); xatp.domid =DOMID_SELF ; xatp.idx =0 ; xatp.space = XENMAPSPACE_shared_info; - xatp.gpfn =__pa (shared_info_page) >>PAGE_SHIFT ; + xatp.gpfn = pfn;if (HYPERVISOR_memory_op (XENMEM_add_to_physmap, &xatp))BUG (); - HYPERVISOR_shared_info = (struct shared_info *)shared_info_page; +} +static void __init xen_hvm_set_shared_info (struct shared_info *sip) +{ +int cpu; + + HYPERVISOR_shared_info = sip;/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info * page, we use it in the event channel upcall and in some pvclock @@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void) } } -#ifdef CONFIG_XEN_PVHVM +/* Reconnect the shared_info pfn to a (new) mfn */ +void xen_hvm_resume_shared_info (void ) +{ +xen_hvm_connect_shared_info (xen_hvm_sip_phys >>PAGE_SHIFT ); +xen_hvm_set_shared_info (xen_hvm_shared_info); +} + +/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage. +* On these old tools the shared info page will be placed in E820_Ram. +* Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects +* that nothing is mapped up to HVM_SHARED_INFO_ADDR. +* Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used +* here for the shared info page. */ +static void __init xen_hvm_init_shared_info (void ) +{ +if (xen_major <4 ) { + xen_hvm_shared_info =extend_brk (PAGE_SIZE ,PAGE_SIZE ); + xen_hvm_sip_phys =__pa (xen_hvm_shared_info); + }else { + xen_hvm_sip_phys =HVM_SHARED_INFO_ADDR ; +set_fixmap (FIX_PARAVIRT_BOOTMAP , xen_hvm_sip_phys); + xen_hvm_shared_info = + (struct shared_info *)fix_to_virt (FIX_PARAVIRT_BOOTMAP ); + } +xen_hvm_resume_shared_info (); +} +static void __init init_hvm_pv_info (void ) { -int major, minor; -uint32_t eax, ebx, ecx, edx, pages, msr, base; +uint32_t ecx, edx, pages, msr, base;u64 pfn; base =xen_cpuid_base (); -cpuid (base +1 , &eax, &ebx, &ecx, &edx); - - major = eax >>16 ; - minor = eax &0xffff ; -printk (KERN_INFO "Xen version %d.%d.\n" , major, minor); -cpuid (base +2 , &pages, &msr, &ecx, &edx); pfn =__pa (hypercall_page);@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base) +return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ; }bool xen_hvm_need_lapic (void )arch/x86/xen/suspend.c +1
-1@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled) {#ifdef CONFIG_XEN_PVHVM int cpu; -xen_hvm_init_shared_info (); +xen_hvm_resume_shared_info ();xen_callback_vector ();xen_unplug_emulated_devices ();if (xen_feature (XENFEAT_hvm_safe_pvclock)) {arch/x86/xen/xen-ops.h +1
-1@@ -43,7 +43,7 @@ void xen_enable_syscall(void);void xen_vcpu_restore (void );void xen_callback_vector (void ); -void xen_hvm_init_shared_info (void ); +void xen_hvm_resume_shared_info (void );void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void );Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Signature · 1 line
Vitaly
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
In reply to Konrad Rzeszutek WilkQuoted 45 lines · Konrad Rzeszutek Wilk
Quoted 44 lines · Vitaly Kuznetsov
Quoted 41 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.So this looks right, the one thing that we really need to check is
e9daff24a266307943457086533041bd971d0ef9This reverts commit
9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the
start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Quoted 8 lines
Quoted 4 lines
Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.
Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.
The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce that on old patch version and everything works fine with your fix (calling xen_hvm_set_shared_info() in addition to xen_hvm_connect_shared_info() on resume). We're probably safe to apply it now, thanks!
However I'd like to suggest we remove '__init' from xen_hvm_set_shared_info() as now we call it on resume.
Quoted 165 lines
Thank you!
Quoted 164 lines
Quoted 161 lines
Quoted 161 lines
Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)[On resume we need to reset the
xen_vcpu_info, which the original patch did not do]Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 74 ++++++++++++++++++++++++++++++++++++------------ arch/x86/xen/suspend.c | 2 +- arch/x86/xen/xen-ops.h | 2 +- 3 files changed, 58 insertions(+), 20 deletions(-)arch/x86/xen/enlighten.c +56
-18@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL +static struct shared_info *xen_hvm_shared_info; +static unsigned long xen_hvm_sip_phys; +static int xen_major, xen_minor; + +static void xen_hvm_connect_shared_info (unsigned long pfn) { -int cpu;struct xen_add_to_physmap xatp; -static struct shared_info *shared_info_page =0 ; -if (!shared_info_page) - shared_info_page = (struct shared_info *) -extend_brk (PAGE_SIZE ,PAGE_SIZE ); xatp.domid =DOMID_SELF ; xatp.idx =0 ; xatp.space = XENMAPSPACE_shared_info; - xatp.gpfn =__pa (shared_info_page) >>PAGE_SHIFT ; + xatp.gpfn = pfn;if (HYPERVISOR_memory_op (XENMEM_add_to_physmap, &xatp))BUG (); - HYPERVISOR_shared_info = (struct shared_info *)shared_info_page; +} +static void __init xen_hvm_set_shared_info (struct shared_info *sip) +{ +int cpu; + + HYPERVISOR_shared_info = sip;/* xen_vcpu is a pointer to the vcpu_info struct in the shared_info * page, we use it in the event channel upcall and in some pvclock @@ -1760,20 +1766,39 @@ void __ref xen_hvm_init_shared_info(void) } } -#ifdef CONFIG_XEN_PVHVM +/* Reconnect the shared_info pfn to a (new) mfn */ +void xen_hvm_resume_shared_info (void ) +{ +xen_hvm_connect_shared_info (xen_hvm_sip_phys >>PAGE_SHIFT ); +xen_hvm_set_shared_info (xen_hvm_shared_info); +} + +/* Xen tools prior to Xen 4 do not provide a E820_Reserved area for guest usage. +* On these old tools the shared info page will be placed in E820_Ram. +* Xen 4 provides a E820_Reserved area at 0xFC000000, and this code expects +* that nothing is mapped up to HVM_SHARED_INFO_ADDR. +* Xen 4.3+ provides an explicit 1MB area at HVM_SHARED_INFO_ADDR which is used +* here for the shared info page. */ +static void __init xen_hvm_init_shared_info (void ) +{ +if (xen_major <4 ) { + xen_hvm_shared_info =extend_brk (PAGE_SIZE ,PAGE_SIZE ); + xen_hvm_sip_phys =__pa (xen_hvm_shared_info); + }else { + xen_hvm_sip_phys =HVM_SHARED_INFO_ADDR ; +set_fixmap (FIX_PARAVIRT_BOOTMAP , xen_hvm_sip_phys); + xen_hvm_shared_info = + (struct shared_info *)fix_to_virt (FIX_PARAVIRT_BOOTMAP ); + } +xen_hvm_resume_shared_info (); +} +static void __init init_hvm_pv_info (void ) { -int major, minor; -uint32_t eax, ebx, ecx, edx, pages, msr, base; +uint32_t ecx, edx, pages, msr, base;u64 pfn; base =xen_cpuid_base (); -cpuid (base +1 , &eax, &ebx, &ecx, &edx); - - major = eax >>16 ; - minor = eax &0xffff ; -printk (KERN_INFO "Xen version %d.%d.\n" , major, minor); -cpuid (base +2 , &pages, &msr, &ecx, &edx); pfn =__pa (hypercall_page);@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base) +return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ; }bool xen_hvm_need_lapic (void )arch/x86/xen/suspend.c +1
-1@@ -32,7 +32,7 @@ static void xen_hvm_post_suspend(int suspend_cancelled) {#ifdef CONFIG_XEN_PVHVM int cpu; -xen_hvm_init_shared_info (); +xen_hvm_resume_shared_info ();xen_callback_vector ();xen_unplug_emulated_devices ();if (xen_feature (XENFEAT_hvm_safe_pvclock)) {arch/x86/xen/xen-ops.h +1
-1@@ -43,7 +43,7 @@ void xen_enable_syscall(void);void xen_vcpu_restore (void );void xen_callback_vector (void ); -void xen_hvm_init_shared_info (void ); +void xen_hvm_resume_shared_info (void );void xen_unplug_emulated_devices (void );void __init xen_build_dynamic_phys_to_machine (void );Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Signature · 1 line
Vitaly
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
Quoted 52 lines · Vitaly Kuznetsov
Quoted 50 lines · Konrad Rzeszutek Wilk
Quoted 48 lines · Vitaly Kuznetsov
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.So this looks right, the one thing that we really need to check is
e9daff24a266307943457086533041bd971d0ef9This reverts commit
9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.Quoted 9 lines
We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the
start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.
Quoted 6 lines
Quoted 3 lines
Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.
The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce
Yes, that is it.
Quoted 4 lines
that on old patch version and everything works fine with your fix (calling
xen_hvm_set_shared_info()in addition toxen_hvm_connect_shared_info()on resume). We're probably safe to apply it now, thanks!
Woot! Could you include that tidbit of information in the commit please?
Quoted 3 lines
However I'd like to suggest we remove '
__init' fromxen_hvm_set_shared_info()as now we call it on resume.
Good idea.
Lets wait until Stefano responds (for the MSI PIRQ one), and if he does not have anything special to say, then repost the whole patchset including this tiny __init fix and the updated comment?
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
In reply to Vitaly KuznetsovQuoted 52 lines · Vitaly Kuznetsov
Quoted 50 lines · Konrad Rzeszutek Wilk
Quoted 48 lines · Vitaly Kuznetsov
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.So this looks right, the one thing that we really need to check is
e9daff24a266307943457086533041bd971d0ef9This reverts commit
9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.Quoted 9 lines
We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the
start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.
Quoted 6 lines
Quoted 3 lines
Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.
The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce
Yes, that is it.
Quoted 4 lines
that on old patch version and everything works fine with your fix (calling
xen_hvm_set_shared_info()in addition toxen_hvm_connect_shared_info()on resume). We're probably safe to apply it now, thanks!
Woot! Could you include that tidbit of information in the commit please?
Quoted 3 lines
However I'd like to suggest we remove '
__init' fromxen_hvm_set_shared_info()as now we call it on resume.
Good idea.
Lets wait until Stefano responds (for the MSI PIRQ one), and if he does not have anything special to say, then repost the whole patchset including this tiny __init fix and the updated comment?
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
Quoted 60 lines · Konrad Rzeszutek Wilk
Quoted 52 lines · Vitaly Kuznetsov
Quoted 50 lines · Konrad Rzeszutek Wilk
Quoted 48 lines · Vitaly Kuznetsov
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.So this looks right, the one thing that we really need to check is
e9daff24a266307943457086533041bd971d0ef9This reverts commit
9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.Quoted 9 lines
We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the
start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.
Quoted 6 lines
Quoted 3 lines
Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.
The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce
Quoted 9 lines
Yes, that is it.
Quoted 4 lines
that on old patch version and everything works fine with your fix (calling
xen_hvm_set_shared_info()in addition toxen_hvm_connect_shared_info()on resume). We're probably safe to apply it now, thanks!Woot! Could you include that tidbit of information in the commit please?
Sure,
Quoted 10 lines
Quoted 3 lines
However I'd like to suggest we remove '
__init' fromxen_hvm_set_shared_info()as now we call it on resume.Good idea.
Lets wait until Stefano responds (for the MSI PIRQ one), and if he does not have anything special to say, then repost the whole patchset including this tiny
__initfix and the updated comment?
Deal :-) Please take a look at my '[PATCH RFC] evtchn: introduce EVTCHNOP_fifo_destroy hypercall'. In case that works we can fix FIFO case at the same time, no TODO required.
I'll be able to return to this work at the end of next week.
Signature · 1 line
Vitaly
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
In reply to Konrad Rzeszutek WilkQuoted 60 lines · Konrad Rzeszutek Wilk
Quoted 52 lines · Vitaly Kuznetsov
Quoted 50 lines · Konrad Rzeszutek Wilk
Quoted 48 lines · Vitaly Kuznetsov
Quoted 32 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 3 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.So this looks right, the one thing that we really need to check is
e9daff24a266307943457086533041bd971d0ef9This reverts commit
9d02b43dee0d7fb18dfb13a00915550b1a3daa9f.Quoted 9 lines
We are doing this b/c on 32-bit PVonHVM with older hypervisors (Xen 4.1) it ends up bothing up the
start_info. This is bad b/c we use it for the time keeping, and the timekeeping code loops forever - as the version field never changes. Olaf says to revert it, so lets do that.Could you kindly test that the migration on 32-bit PVHVM guests on older hypervisors works?
Sure, will do! Was there anything special about the setup or any 32-bit pvhvm guest migration (on 64-bit hypervisor I suppose) would fail? I can try checking both current and old versions to make sure the issue was acutually fixed.
Quoted 6 lines
Quoted 3 lines
Nothing fancy (well, it was SMP, so 4 CPUs). I did the 'save'/'restore' and the guest would not restore properly.
The symptoms you saw were: after the resume guest appears to be frozen, all vcpus except for the first one spin at 100%? I was able to reproduce
Quoted 9 lines
Yes, that is it.
Quoted 4 lines
that on old patch version and everything works fine with your fix (calling
xen_hvm_set_shared_info()in addition toxen_hvm_connect_shared_info()on resume). We're probably safe to apply it now, thanks!Woot! Could you include that tidbit of information in the commit please?
Sure,
Quoted 10 lines
Quoted 3 lines
However I'd like to suggest we remove '
__init' fromxen_hvm_set_shared_info()as now we call it on resume.Good idea.
Lets wait until Stefano responds (for the MSI PIRQ one), and if he does not have anything special to say, then repost the whole patchset including this tiny
__initfix and the updated comment?
Deal :-) Please take a look at my '[PATCH RFC] evtchn: introduce EVTCHNOP_fifo_destroy hypercall'. In case that works we can fix FIFO case at the same time, no TODO required.
I'll be able to return to this work at the end of next week.
Signature · 1 line
Vitaly
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
In reply to Vitaly KuznetsovQuoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 7 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)
Is this a useful commit to give? Isn't it one of Konrad's random trees?
arch/x86/xen/enlighten.c asmlinkage __visible void __init xen_start_kernel(void) · line 1730
Quoted 9 lines
arch/x86/xen/enlighten.c +2
-1@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL
I would like clearer documentation for this magic address. Perhaps referring to some toolstack documentation?
Quoted 12 lines
arch/x86/xen/enlighten.c +4
-1@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base)
arch/x86/xen/enlighten.c static void __init xen_hvm_guest_init(void) · line 1872
Quoted 10 lines
arch/x86/xen/enlighten.c +10
-0+return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ;
You appear to have changed to return value here? Did you mean to return base here?
David
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
In reply to Vitaly KuznetsovQuoted 22 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
Quoted 7 lines
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)
Is this a useful commit to give? Isn't it one of Konrad's random trees?
arch/x86/xen/enlighten.c asmlinkage __visible void __init xen_start_kernel(void) · line 1730
Quoted 9 lines
arch/x86/xen/enlighten.c +2
-1@@ -1726,23 +1726,29 @@ asmlinkage __visible void __init xen_start_kernel(void)#endif } -void __ref xen_hvm_init_shared_info (void ) +#ifdef CONFIG_XEN_PVHVM +#define HVM_SHARED_INFO_ADDR 0xFE700000UL
I would like clearer documentation for this magic address. Perhaps referring to some toolstack documentation?
Quoted 12 lines
arch/x86/xen/enlighten.c +4
-1@@ -1828,10 +1853,23 @@ static void __init xen_hvm_guest_init(void)static uint32_t __init xen_hvm_platform (void ) { +uint32_t eax, ebx, ecx, edx, base; +if (xen_pv_domain ())return 0 ; -return xen_cpuid_base (); + base =xen_cpuid_base (); +if (!base)
arch/x86/xen/enlighten.c static void __init xen_hvm_guest_init(void) · line 1872
Quoted 10 lines
arch/x86/xen/enlighten.c +10
-0+return 0 ; + +cpuid (base +1 , &eax, &ebx, &ecx, &edx); + + xen_major = eax >>16 ; + xen_minor = eax &0xffff ; + +printk (KERN_INFO "Xen version %d.%d.\n" , xen_major, xen_minor); + +return 1 ;
You appear to have changed to return value here? Did you mean to return base here?
David
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
Quoted 27 lines · David Vrabel
Quoted 26 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.
Quoted 5 lines
Quoted 3 lines
Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)Is this a useful commit to give? Isn't it one of Konrad's random trees?
Actually it is from upstream. That patch was reverted at certain point.
Perhaps just mention that it was added in, then reverted, and now after a year or so, added back in with the fix.
Re: [PATCH RFC 1/4] xen PVonHVM: use E820_Reserved area for shared_info
In reply to David VrabelQuoted 27 lines · David Vrabel
Quoted 26 lines · Vitaly Kuznetsov
From: Olaf Hering <[email protected]>
This is a respin of
00e37bdb0113a98408de42db85be002f21dbffd3("xen PVonHVM: moveshared_infoto MMIO before kexec").Currently kexec in a PVonHVM guest fails with a triple fault because the new kernel overwrites the shared info page. The exact failure depends on the size of the kernel image. This patch moves the pfn from RAM into an E820 reserved memory area.
The pfn containing the
shared_infois located somewhere in RAM. This will cause trouble if the current kernel is doing a kexec boot into a new kernel. The new kernel (and its startup code) can not know where the pfn is, so it can not reserve the page. The hypervisor will continue to update the pfn, and as a result memory corruption occours in the new kernel.The toolstack marks the memory area FC000000-FFFFFFFF as reserved in the E820 map. Within that range newer toolstacks (4.3+) will keep 1MB starting from FE700000 as reserved for guest use. Older Xen4 toolstacks will usually not allocate areas up to FE700000, so FE700000 is expected to work also with older toolstacks.
In Xen3 there is no reserved area at a fixed location. If the guest is started on such old hosts the
shared_infopage will be placed in RAM. As a result kexec can not be used.
Quoted 5 lines
Quoted 3 lines
Signed-off-by Olaf Hering <[email protected]>Signed-off-by Konrad Rzeszutek Wilk <[email protected]>(cherry picked from commit 9d02b43dee0d7fb18dfb13a00915550b1a3daa9f)Is this a useful commit to give? Isn't it one of Konrad's random trees?
Actually it is from upstream. That patch was reverted at certain point.
Perhaps just mention that it was added in, then reverted, and now after a year or so, added back in with the fix.
RFC PATCH 2/4 xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
In reply to Vitaly KuznetsovPVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().
--- arch/x86/xen/enlighten.c | 9 +++++++++ arch/x86/xen/smp.c | 9 +++++++++ arch/x86/xen/smp.h | 1 + 3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +9 -0
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown (); +native_machine_shutdown (); +} +static void __init xen_hvm_guest_init (void ) {init_hvm_pv_info ();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void) x86_init.irqs.intr_init = xen_init_IRQ;xen_hvm_init_time_ops ();xen_hvm_init_mmu_ops (); +#ifdef CONFIG_KEXEC + machine_ops.shutdown = xen_pvhvm_kexec_shutdown; +#endif }static uint32_t __init xen_hvm_platform (void )
arch/x86/xen/smp.c +9 -0
@@ -18,6 +18,7 @@#include <linux/smp.h> #include <linux/irq_work.h> #include <linux/tick.h> +#include <linux/kexec.h> #include <asm/paravirt.h> #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)native_cpu_die (cpu); } +void xen_kexec_shutdown (void ) +{ +#ifdef CONFIG_KEXEC +if (!kexec_in_progress) +return ; +#endif +} +void __init xen_hvm_smp_init (void ) {if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);extern void xen_send_IPI_all (int vector);extern void xen_send_IPI_self (int vector); +extern void xen_kexec_shutdown (void );#endif
Signature · 1 line
1.9.3
RFC PATCH 2/4 xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
In reply to Vitaly KuznetsovPVHVM guest requires special actions before kexec. Register specific xen_pvhvm_kexec_shutdown() handler for machine_ops.shutdown().
--- arch/x86/xen/enlighten.c | 9 +++++++++ arch/x86/xen/smp.c | 9 +++++++++ arch/x86/xen/smp.h | 1 + 3 files changed, 19 insertions(+)
arch/x86/xen/enlighten.c +9 -0
@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown (); +native_machine_shutdown (); +} +static void __init xen_hvm_guest_init (void ) {init_hvm_pv_info ();
@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void) x86_init.irqs.intr_init = xen_init_IRQ;xen_hvm_init_time_ops ();xen_hvm_init_mmu_ops (); +#ifdef CONFIG_KEXEC + machine_ops.shutdown = xen_pvhvm_kexec_shutdown; +#endif }static uint32_t __init xen_hvm_platform (void )
arch/x86/xen/smp.c +9 -0
@@ -18,6 +18,7 @@#include <linux/smp.h> #include <linux/irq_work.h> #include <linux/tick.h> +#include <linux/kexec.h> #include <asm/paravirt.h> #include <asm/desc.h>
@@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)native_cpu_die (cpu); } +void xen_kexec_shutdown (void ) +{ +#ifdef CONFIG_KEXEC +if (!kexec_in_progress) +return ; +#endif +} +void __init xen_hvm_smp_init (void ) {if (!xen_have_vector_callback)
arch/x86/xen/smp.h +1 -0
@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);extern void xen_send_IPI_all (int vector);extern void xen_send_IPI_self (int vector); +extern void xen_kexec_shutdown (void );#endif
Signature · 1 line
1.9.3
Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
Quoted 3 lines · Vitaly Kuznetsov
PVHVM guest requires special actions before kexec. Register specific
xen_pvhvm_kexec_shutdown()handler formachine_ops.shutdown().
This looks close to what I had sent as an RFC to you?
Quoted 74 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 9 +++++++++ arch/x86/xen/smp.c | 9 +++++++++ arch/x86/xen/smp.h | 1 + 3 files changed, 19 insertions(+)arch/x86/xen/enlighten.c +9
-0@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown (); +native_machine_shutdown (); +} +static void __init xen_hvm_guest_init (void ) {init_hvm_pv_info ();@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void) x86_init.irqs.intr_init = xen_init_IRQ;xen_hvm_init_time_ops ();xen_hvm_init_mmu_ops (); +#ifdef CONFIG_KEXEC + machine_ops.shutdown = xen_pvhvm_kexec_shutdown; +#endif }static uint32_t __init xen_hvm_platform (void )arch/x86/xen/smp.c +9
-0@@ -18,6 +18,7 @@#include <linux/smp.h> #include <linux/irq_work.h> #include <linux/tick.h> +#include <linux/kexec.h> #include <asm/paravirt.h> #include <asm/desc.h> @@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)native_cpu_die (cpu); } +void xen_kexec_shutdown (void ) +{ +#ifdef CONFIG_KEXEC +if (!kexec_in_progress) +return ; +#endif +} +void __init xen_hvm_smp_init (void ) {if (!xen_have_vector_callback)arch/x86/xen/smp.h +1
-0@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);extern void xen_send_IPI_all (int vector);extern void xen_send_IPI_self (int vector); +extern void xen_kexec_shutdown (void );#endif Signature · 1 line
1.9.3
Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
In reply to Vitaly KuznetsovQuoted 3 lines · Vitaly Kuznetsov
PVHVM guest requires special actions before kexec. Register specific
xen_pvhvm_kexec_shutdown()handler formachine_ops.shutdown().
This looks close to what I had sent as an RFC to you?
Quoted 74 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 9 +++++++++ arch/x86/xen/smp.c | 9 +++++++++ arch/x86/xen/smp.h | 1 + 3 files changed, 19 insertions(+)arch/x86/xen/enlighten.c +9
-0@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown (); +native_machine_shutdown (); +} +static void __init xen_hvm_guest_init (void ) {init_hvm_pv_info ();@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void) x86_init.irqs.intr_init = xen_init_IRQ;xen_hvm_init_time_ops ();xen_hvm_init_mmu_ops (); +#ifdef CONFIG_KEXEC + machine_ops.shutdown = xen_pvhvm_kexec_shutdown; +#endif }static uint32_t __init xen_hvm_platform (void )arch/x86/xen/smp.c +9
-0@@ -18,6 +18,7 @@#include <linux/smp.h> #include <linux/irq_work.h> #include <linux/tick.h> +#include <linux/kexec.h> #include <asm/paravirt.h> #include <asm/desc.h> @@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)native_cpu_die (cpu); } +void xen_kexec_shutdown (void ) +{ +#ifdef CONFIG_KEXEC +if (!kexec_in_progress) +return ; +#endif +} +void __init xen_hvm_smp_init (void ) {if (!xen_have_vector_callback)arch/x86/xen/smp.h +1
-0@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);extern void xen_send_IPI_all (int vector);extern void xen_send_IPI_self (int vector); +extern void xen_kexec_shutdown (void );#endif Signature · 1 line
1.9.3
Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Vitaly Kuznetsov
PVHVM guest requires special actions before kexec. Register specific
xen_pvhvm_kexec_shutdown()handler formachine_ops.shutdown().This looks close to what I had sent as an RFC to you?
Yes, I stole that part from your "RFC: VCPU_reset_cpu_info" patch to call xen_unmap_all_pirqs() on shutdown. I'm looking at SMP issues your patches address as well.
Quoted 74 lines
Quoted 74 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 9 +++++++++ arch/x86/xen/smp.c | 9 +++++++++ arch/x86/xen/smp.h | 1 + 3 files changed, 19 insertions(+)arch/x86/xen/enlighten.c +9
-0@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown (); +native_machine_shutdown (); +} +static void __init xen_hvm_guest_init (void ) {init_hvm_pv_info ();@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void) x86_init.irqs.intr_init = xen_init_IRQ;xen_hvm_init_time_ops ();xen_hvm_init_mmu_ops (); +#ifdef CONFIG_KEXEC + machine_ops.shutdown = xen_pvhvm_kexec_shutdown; +#endif }static uint32_t __init xen_hvm_platform (void )arch/x86/xen/smp.c +9
-0@@ -18,6 +18,7 @@#include <linux/smp.h> #include <linux/irq_work.h> #include <linux/tick.h> +#include <linux/kexec.h> #include <asm/paravirt.h> #include <asm/desc.h> @@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)native_cpu_die (cpu); } +void xen_kexec_shutdown (void ) +{ +#ifdef CONFIG_KEXEC +if (!kexec_in_progress) +return ; +#endif +} +void __init xen_hvm_smp_init (void ) {if (!xen_have_vector_callback)arch/x86/xen/smp.h +1
-0@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);extern void xen_send_IPI_all (int vector);extern void xen_send_IPI_self (int vector); +extern void xen_kexec_shutdown (void );#endif Signature · 1 line
1.9.3
Signature · 1 line
Vitaly
Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
In reply to Konrad Rzeszutek WilkQuoted 7 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Vitaly Kuznetsov
PVHVM guest requires special actions before kexec. Register specific
xen_pvhvm_kexec_shutdown()handler formachine_ops.shutdown().This looks close to what I had sent as an RFC to you?
Yes, I stole that part from your "RFC: VCPU_reset_cpu_info" patch to call xen_unmap_all_pirqs() on shutdown. I'm looking at SMP issues your patches address as well.
Quoted 74 lines
Quoted 74 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 9 +++++++++ arch/x86/xen/smp.c | 9 +++++++++ arch/x86/xen/smp.h | 1 + 3 files changed, 19 insertions(+)arch/x86/xen/enlighten.c +9
-0@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown (); +native_machine_shutdown (); +} +static void __init xen_hvm_guest_init (void ) {init_hvm_pv_info ();@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void) x86_init.irqs.intr_init = xen_init_IRQ;xen_hvm_init_time_ops ();xen_hvm_init_mmu_ops (); +#ifdef CONFIG_KEXEC + machine_ops.shutdown = xen_pvhvm_kexec_shutdown; +#endif }static uint32_t __init xen_hvm_platform (void )arch/x86/xen/smp.c +9
-0@@ -18,6 +18,7 @@#include <linux/smp.h> #include <linux/irq_work.h> #include <linux/tick.h> +#include <linux/kexec.h> #include <asm/paravirt.h> #include <asm/desc.h> @@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)native_cpu_die (cpu); } +void xen_kexec_shutdown (void ) +{ +#ifdef CONFIG_KEXEC +if (!kexec_in_progress) +return ; +#endif +} +void __init xen_hvm_smp_init (void ) {if (!xen_have_vector_callback)arch/x86/xen/smp.h +1
-0@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);extern void xen_send_IPI_all (int vector);extern void xen_send_IPI_self (int vector); +extern void xen_kexec_shutdown (void );#endif Signature · 1 line
1.9.3
Signature · 1 line
Vitaly
Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
Quoted 7 lines · Vitaly Kuznetsov
Quoted 5 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Vitaly Kuznetsov
PVHVM guest requires special actions before kexec. Register specific
xen_pvhvm_kexec_shutdown()handler formachine_ops.shutdown().
Quoted 6 lines
Quoted 2 lines
This looks close to what I had sent as an RFC to you?
Yes, I stole that part from your "RFC:
VCPU_reset_cpu_info" patch to callxen_unmap_all_pirqs()on shutdown. I'm looking at SMP issues your patches address as well.
It is customary to attribute that in the patch. Such as:
"Original idea/patch from Konrad Rzeszutek Wilk" is suffice.
Thanks!
Quoted 78 lines
Quoted 75 lines
Quoted 75 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 9 +++++++++ arch/x86/xen/smp.c | 9 +++++++++ arch/x86/xen/smp.h | 1 + 3 files changed, 19 insertions(+)arch/x86/xen/enlighten.c +9
-0@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown (); +native_machine_shutdown (); +} +static void __init xen_hvm_guest_init (void ) {init_hvm_pv_info ();@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void) x86_init.irqs.intr_init = xen_init_IRQ;xen_hvm_init_time_ops ();xen_hvm_init_mmu_ops (); +#ifdef CONFIG_KEXEC + machine_ops.shutdown = xen_pvhvm_kexec_shutdown; +#endif }static uint32_t __init xen_hvm_platform (void )arch/x86/xen/smp.c +9
-0@@ -18,6 +18,7 @@#include <linux/smp.h> #include <linux/irq_work.h> #include <linux/tick.h> +#include <linux/kexec.h> #include <asm/paravirt.h> #include <asm/desc.h> @@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)native_cpu_die (cpu); } +void xen_kexec_shutdown (void ) +{ +#ifdef CONFIG_KEXEC +if (!kexec_in_progress) +return ; +#endif +} +void __init xen_hvm_smp_init (void ) {if (!xen_have_vector_callback)arch/x86/xen/smp.h +1
-0@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);extern void xen_send_IPI_all (int vector);extern void xen_send_IPI_self (int vector); +extern void xen_kexec_shutdown (void );#endif Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
In reply to Vitaly KuznetsovQuoted 7 lines · Vitaly Kuznetsov
Quoted 5 lines · Konrad Rzeszutek Wilk
Quoted 3 lines · Vitaly Kuznetsov
PVHVM guest requires special actions before kexec. Register specific
xen_pvhvm_kexec_shutdown()handler formachine_ops.shutdown().
Quoted 6 lines
Quoted 2 lines
This looks close to what I had sent as an RFC to you?
Yes, I stole that part from your "RFC:
VCPU_reset_cpu_info" patch to callxen_unmap_all_pirqs()on shutdown. I'm looking at SMP issues your patches address as well.
It is customary to attribute that in the patch. Such as:
"Original idea/patch from Konrad Rzeszutek Wilk" is suffice.
Thanks!
Quoted 78 lines
Quoted 75 lines
Quoted 75 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 9 +++++++++ arch/x86/xen/smp.c | 9 +++++++++ arch/x86/xen/smp.h | 1 + 3 files changed, 19 insertions(+)arch/x86/xen/enlighten.c +9
-0@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown (); +native_machine_shutdown (); +} +static void __init xen_hvm_guest_init (void ) {init_hvm_pv_info ();@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void) x86_init.irqs.intr_init = xen_init_IRQ;xen_hvm_init_time_ops ();xen_hvm_init_mmu_ops (); +#ifdef CONFIG_KEXEC + machine_ops.shutdown = xen_pvhvm_kexec_shutdown; +#endif }static uint32_t __init xen_hvm_platform (void )arch/x86/xen/smp.c +9
-0@@ -18,6 +18,7 @@#include <linux/smp.h> #include <linux/irq_work.h> #include <linux/tick.h> +#include <linux/kexec.h> #include <asm/paravirt.h> #include <asm/desc.h> @@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)native_cpu_die (cpu); } +void xen_kexec_shutdown (void ) +{ +#ifdef CONFIG_KEXEC +if (!kexec_in_progress) +return ; +#endif +} +void __init xen_hvm_smp_init (void ) {if (!xen_have_vector_callback)arch/x86/xen/smp.h +1
-0@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);extern void xen_send_IPI_all (int vector);extern void xen_send_IPI_self (int vector); +extern void xen_kexec_shutdown (void );#endif Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Boris Ostrovsky
reviewerRe: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
In reply to Vitaly KuznetsovQuoted 15 lines · Vitaly Kuznetsov
PVHVM guest requires special actions before kexec. Register specific
xen_pvhvm_kexec_shutdown()handler formachine_ops.shutdown().Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 9 +++++++++ arch/x86/xen/smp.c | 9 +++++++++ arch/x86/xen/smp.h | 1 + 3 files changed, 19 insertions(+)arch/x86/xen/enlighten.c +0
-0
arch/x86/xen/enlighten.c static struct notifier_block xen_hvm_cpu_notifier = { · line 1841
Quoted 9 lines
arch/x86/xen/enlighten.c +6
-0@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown (); +native_machine_shutdown (); +} +
This should probably be under ifdef CONFIG_KEXEC ?
Quoted 27 lines
arch/x86/xen/enlighten.c +3
-0static void __init xen_hvm_guest_init (void ) {init_hvm_pv_info ();@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void) x86_init.irqs.intr_init = xen_init_IRQ;xen_hvm_init_time_ops ();xen_hvm_init_mmu_ops (); +#ifdef CONFIG_KEXEC + machine_ops.shutdown = xen_pvhvm_kexec_shutdown; +#endif }static uint32_t __init xen_hvm_platform (void )arch/x86/xen/smp.c +1
-0@@ -18,6 +18,7 @@#include <linux/smp.h> #include <linux/irq_work.h> #include <linux/tick.h> +#include <linux/kexec.h> #include <asm/paravirt.h> #include <asm/desc.h> @@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)native_cpu_die (cpu);
arch/x86/xen/smp.c static void xen_hvm_cpu_die(unsigned int cpu) · line 773
Quoted 10 lines
arch/x86/xen/smp.c +8
-0} +void xen_kexec_shutdown (void ) +{ +#ifdef CONFIG_KEXEC +if (!kexec_in_progress) +return ; +#endif +} +
and this one too (whole routine)? And "#include" above as well.
Quoted 13 lines
arch/x86/xen/smp.c +0
-0void __init xen_hvm_smp_init (void ) {if (!xen_have_vector_callback)arch/x86/xen/smp.h +1
-0@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);extern void xen_send_IPI_all (int vector);extern void xen_send_IPI_self (int vector); +extern void xen_kexec_shutdown (void );#endif
Boris Ostrovsky
reviewerRe: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
In reply to Vitaly KuznetsovQuoted 15 lines · Vitaly Kuznetsov
PVHVM guest requires special actions before kexec. Register specific
xen_pvhvm_kexec_shutdown()handler formachine_ops.shutdown().Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/enlighten.c | 9 +++++++++ arch/x86/xen/smp.c | 9 +++++++++ arch/x86/xen/smp.h | 1 + 3 files changed, 19 insertions(+)arch/x86/xen/enlighten.c +0
-0
arch/x86/xen/enlighten.c static struct notifier_block xen_hvm_cpu_notifier = { · line 1841
Quoted 9 lines
arch/x86/xen/enlighten.c +6
-0@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown (); +native_machine_shutdown (); +} +
This should probably be under ifdef CONFIG_KEXEC ?
Quoted 27 lines
arch/x86/xen/enlighten.c +3
-0static void __init xen_hvm_guest_init (void ) {init_hvm_pv_info ();@@ -1849,6 +1855,9 @@ static void __init xen_hvm_guest_init(void) x86_init.irqs.intr_init = xen_init_IRQ;xen_hvm_init_time_ops ();xen_hvm_init_mmu_ops (); +#ifdef CONFIG_KEXEC + machine_ops.shutdown = xen_pvhvm_kexec_shutdown; +#endif }static uint32_t __init xen_hvm_platform (void )arch/x86/xen/smp.c +1
-0@@ -18,6 +18,7 @@#include <linux/smp.h> #include <linux/irq_work.h> #include <linux/tick.h> +#include <linux/kexec.h> #include <asm/paravirt.h> #include <asm/desc.h> @@ -762,6 +763,14 @@ static void xen_hvm_cpu_die(unsigned int cpu)native_cpu_die (cpu);
arch/x86/xen/smp.c static void xen_hvm_cpu_die(unsigned int cpu) · line 773
Quoted 10 lines
arch/x86/xen/smp.c +8
-0} +void xen_kexec_shutdown (void ) +{ +#ifdef CONFIG_KEXEC +if (!kexec_in_progress) +return ; +#endif +} +
and this one too (whole routine)? And "#include" above as well.
Quoted 13 lines
arch/x86/xen/smp.c +0
-0void __init xen_hvm_smp_init (void ) {if (!xen_have_vector_callback)arch/x86/xen/smp.h +1
-0@@ -8,4 +8,5 @@ extern void xen_send_IPI_allbutself(int vector);extern void xen_send_IPI_all (int vector);extern void xen_send_IPI_self (int vector); +extern void xen_kexec_shutdown (void );#endif
Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
In reply to Vitaly KuznetsovQuoted 2 lines · Vitaly Kuznetsov
PVHVM guest requires special actions before kexec. Register specific
xen_pvhvm_kexec_shutdown()handler formachine_ops.shutdown().
[...]
arch/x86/xen/enlighten.c static struct notifier_block xen_hvm_cpu_notifier = { · line 1838
Quoted 9 lines
arch/x86/xen/enlighten.c +3
-0@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown ();
I think it would be preferrable to have
if (kexec_in_progress)
xen_kexec_shutdown();David
Re: [PATCH RFC 2/4] xen/pvhvm: Introduce xen_pvhvm_kexec_shutdown()
In reply to Vitaly KuznetsovQuoted 2 lines · Vitaly Kuznetsov
PVHVM guest requires special actions before kexec. Register specific
xen_pvhvm_kexec_shutdown()handler formachine_ops.shutdown().
[...]
arch/x86/xen/enlighten.c static struct notifier_block xen_hvm_cpu_notifier = { · line 1838
Quoted 9 lines
arch/x86/xen/enlighten.c +3
-0@@ -1833,6 +1833,12 @@ static struct notifier_block xen_hvm_cpu_notifier = { .notifier_call = xen_hvm_cpu_notify, }; +static void xen_pvhvm_kexec_shutdown (void ) +{ +xen_kexec_shutdown ();
I think it would be preferrable to have
if (kexec_in_progress)
xen_kexec_shutdown();David
RFC PATCH 3/4 xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to Vitaly KuznetsovWhen kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...
and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).
--- arch/x86/xen/smp.c | 1 + drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }
drivers/xen/events/events_base.c +76 -0
88 lines not on this page · raw message
include/xen/events.h +3 -0
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);/* Determine whether to ignore this IRQ if it is passed to a guest. */ int xen_test_irq_shared (int irq); +/* Unmap all PIRQs and close event channels */ +void xen_unmap_all_pirqs (void ); +/* initialize Xen IRQ subsystem */ void xen_init_IRQ (void );#endif /* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
RFC PATCH 3/4 xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to Vitaly KuznetsovWhen kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...
and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but no PHYSDEVOP_unmap_pirq is being performed).
--- arch/x86/xen/smp.c | 1 + drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 3 files changed, 80 insertions(+)
arch/x86/xen/smp.c +1 -0
@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }
drivers/xen/events/events_base.c +76 -0
88 lines not on this page · raw message
include/xen/events.h +3 -0
@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);/* Determine whether to ignore this IRQ if it is passed to a guest. */ int xen_test_irq_shared (int irq); +/* Unmap all PIRQs and close event channels */ +void xen_unmap_all_pirqs (void ); +/* initialize Xen IRQ subsystem */ void xen_init_IRQ (void );#endif /* _XEN_EVENTS_H */
Signature · 1 line
1.9.3
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).
How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?
Thanks.
Quoted 58 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/smp.c | 1 + drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 3 files changed, 80 insertions(+)arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +30
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{ +int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ; + + pirq = info->u.pirq.pirq; + gsi = info->u.pirq.gsi; + evtchn = info->evtchn; + irq = info->irq; + +pr_debug ("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n" , + gsi, pirq, irq, evtchn); + +if (evtchn >0 ) { + close.port = evtchn; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("close evtchn %d failed\n" , evtchn); + } + + unmap_irq.pirq = pirq; + unmap_irq.domid =DOMID_SELF ;
drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+ + rc =HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, &unmap_irq); +if (rc) +pr_warn ("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n" , + gsi, pirq, irq, rc); + } + +mutex_unlock (&irq_mapping_update_lock); +} +EXPORT_SYMBOL_GPL (xen_unmap_all_pirqs);
Why the EXPORT? Is this used by modules?
Quoted 63 lines
drivers/xen/events/events_base.c +36
-0+ +static void xen_startup_unmap_pirqs (void ) +{ +struct evtchn_status status; +int port, rc = -ENOENT ; +struct physdev_unmap_pirq unmap_irq; +struct evtchn_close close; + +memset (&status,0 ,sizeof (status)); +for (port =0 ; port <xen_evtchn_max_channels (); port++) { + status.dom =DOMID_SELF ; + status.port = port; + rc =HYPERVISOR_event_channel_op (EVTCHNOP_status, &status); +if (rc <0 ) +continue ; +if (status.status == EVTCHNSTAT_pirq) { + close.port = port; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("xen: failed to close evtchn %d\n" , + port); + unmap_irq.pirq = status.u.pirq; + unmap_irq.domid =DOMID_SELF ; +pr_warn ("xen: unmapping previously mapped pirq %d\n" , + unmap_irq.pirq); +if (HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, + &unmap_irq) !=0 ) +pr_warn ("xen: failed to unmap pirq %d\n" , + unmap_irq.pirq); + } + } +} + +void __init xen_init_IRQ (void ) {int ret = -EINVAL ;@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)xen_callback_vector ();if (xen_hvm_domain ()) { +xen_startup_unmap_pirqs (); +native_init_IRQ ();/* pci_xen_hvm_init must be called after native_init_IRQ so that * __acpi_register_gsi can point at the right function */ include/xen/events.h +3
-0@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);/* Determine whether to ignore this IRQ if it is passed to a guest. */ int xen_test_irq_shared (int irq); +/* Unmap all PIRQs and close event channels */ +void xen_unmap_all_pirqs (void ); +/* initialize Xen IRQ subsystem */ void xen_init_IRQ (void );#endif /* _XEN_EVENTS_H */ Signature · 1 line
1.9.3
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to Vitaly KuznetsovQuoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).
How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?
Thanks.
Quoted 58 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/smp.c | 1 + drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 3 files changed, 80 insertions(+)arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +30
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{ +int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ; + + pirq = info->u.pirq.pirq; + gsi = info->u.pirq.gsi; + evtchn = info->evtchn; + irq = info->irq; + +pr_debug ("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n" , + gsi, pirq, irq, evtchn); + +if (evtchn >0 ) { + close.port = evtchn; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("close evtchn %d failed\n" , evtchn); + } + + unmap_irq.pirq = pirq; + unmap_irq.domid =DOMID_SELF ;
drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+ + rc =HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, &unmap_irq); +if (rc) +pr_warn ("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n" , + gsi, pirq, irq, rc); + } + +mutex_unlock (&irq_mapping_update_lock); +} +EXPORT_SYMBOL_GPL (xen_unmap_all_pirqs);
Why the EXPORT? Is this used by modules?
Quoted 63 lines
drivers/xen/events/events_base.c +36
-0+ +static void xen_startup_unmap_pirqs (void ) +{ +struct evtchn_status status; +int port, rc = -ENOENT ; +struct physdev_unmap_pirq unmap_irq; +struct evtchn_close close; + +memset (&status,0 ,sizeof (status)); +for (port =0 ; port <xen_evtchn_max_channels (); port++) { + status.dom =DOMID_SELF ; + status.port = port; + rc =HYPERVISOR_event_channel_op (EVTCHNOP_status, &status); +if (rc <0 ) +continue ; +if (status.status == EVTCHNSTAT_pirq) { + close.port = port; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("xen: failed to close evtchn %d\n" , + port); + unmap_irq.pirq = status.u.pirq; + unmap_irq.domid =DOMID_SELF ; +pr_warn ("xen: unmapping previously mapped pirq %d\n" , + unmap_irq.pirq); +if (HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, + &unmap_irq) !=0 ) +pr_warn ("xen: failed to unmap pirq %d\n" , + unmap_irq.pirq); + } + } +} + +void __init xen_init_IRQ (void ) {int ret = -EINVAL ;@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)xen_callback_vector ();if (xen_hvm_domain ()) { +xen_startup_unmap_pirqs (); +native_init_IRQ ();/* pci_xen_hvm_init must be called after native_init_IRQ so that * __acpi_register_gsi can point at the right function */ include/xen/events.h +3
-0@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);/* Determine whether to ignore this IRQ if it is passed to a guest. */ int xen_test_irq_shared (int irq); +/* Unmap all PIRQs and close event channels */ +void xen_unmap_all_pirqs (void ); +/* initialize Xen IRQ subsystem */ void xen_init_IRQ (void );#endif /* _XEN_EVENTS_H */ Signature · 1 line
1.9.3
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
Quoted 24 lines · Konrad Rzeszutek Wilk
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).
Quoted 2 lines
How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?
Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...
Quoted 135 lines
Thanks.
Quoted 58 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/smp.c | 1 + drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 3 files changed, 80 insertions(+)arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +30
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{ +int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ; + + pirq = info->u.pirq.pirq; + gsi = info->u.pirq.gsi; + evtchn = info->evtchn; + irq = info->irq; + +pr_debug ("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n" , + gsi, pirq, irq, evtchn); + +if (evtchn >0 ) { + close.port = evtchn; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("close evtchn %d failed\n" , evtchn); + } + + unmap_irq.pirq = pirq; + unmap_irq.domid =DOMID_SELF ;drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+ + rc =HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, &unmap_irq); +if (rc) +pr_warn ("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n" , + gsi, pirq, irq, rc); + } + +mutex_unlock (&irq_mapping_update_lock); +} +EXPORT_SYMBOL_GPL (xen_unmap_all_pirqs);Why the EXPORT? Is this used by modules?
Quoted 63 lines
drivers/xen/events/events_base.c +36
-0+ +static void xen_startup_unmap_pirqs (void ) +{ +struct evtchn_status status; +int port, rc = -ENOENT ; +struct physdev_unmap_pirq unmap_irq; +struct evtchn_close close; + +memset (&status,0 ,sizeof (status)); +for (port =0 ; port <xen_evtchn_max_channels (); port++) { + status.dom =DOMID_SELF ; + status.port = port; + rc =HYPERVISOR_event_channel_op (EVTCHNOP_status, &status); +if (rc <0 ) +continue ; +if (status.status == EVTCHNSTAT_pirq) { + close.port = port; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("xen: failed to close evtchn %d\n" , + port); + unmap_irq.pirq = status.u.pirq; + unmap_irq.domid =DOMID_SELF ; +pr_warn ("xen: unmapping previously mapped pirq %d\n" , + unmap_irq.pirq); +if (HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, + &unmap_irq) !=0 ) +pr_warn ("xen: failed to unmap pirq %d\n" , + unmap_irq.pirq); + } + } +} + +void __init xen_init_IRQ (void ) {int ret = -EINVAL ;@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)xen_callback_vector ();if (xen_hvm_domain ()) { +xen_startup_unmap_pirqs (); +native_init_IRQ ();/* pci_xen_hvm_init must be called after native_init_IRQ so that * __acpi_register_gsi can point at the right function */ include/xen/events.h +3
-0@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);/* Determine whether to ignore this IRQ if it is passed to a guest. */ int xen_test_irq_shared (int irq); +/* Unmap all PIRQs and close event channels */ +void xen_unmap_all_pirqs (void ); +/* initialize Xen IRQ subsystem */ void xen_init_IRQ (void );#endif /* _XEN_EVENTS_H */ Signature · 1 line
1.9.3
Signature · 1 line
Vitaly
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to Konrad Rzeszutek WilkQuoted 24 lines · Konrad Rzeszutek Wilk
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).
Quoted 2 lines
How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?
Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...
Quoted 135 lines
Thanks.
Quoted 58 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/smp.c | 1 + drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 3 files changed, 80 insertions(+)arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +30
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{ +int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ; + + pirq = info->u.pirq.pirq; + gsi = info->u.pirq.gsi; + evtchn = info->evtchn; + irq = info->irq; + +pr_debug ("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n" , + gsi, pirq, irq, evtchn); + +if (evtchn >0 ) { + close.port = evtchn; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("close evtchn %d failed\n" , evtchn); + } + + unmap_irq.pirq = pirq; + unmap_irq.domid =DOMID_SELF ;drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+ + rc =HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, &unmap_irq); +if (rc) +pr_warn ("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n" , + gsi, pirq, irq, rc); + } + +mutex_unlock (&irq_mapping_update_lock); +} +EXPORT_SYMBOL_GPL (xen_unmap_all_pirqs);Why the EXPORT? Is this used by modules?
Quoted 63 lines
drivers/xen/events/events_base.c +36
-0+ +static void xen_startup_unmap_pirqs (void ) +{ +struct evtchn_status status; +int port, rc = -ENOENT ; +struct physdev_unmap_pirq unmap_irq; +struct evtchn_close close; + +memset (&status,0 ,sizeof (status)); +for (port =0 ; port <xen_evtchn_max_channels (); port++) { + status.dom =DOMID_SELF ; + status.port = port; + rc =HYPERVISOR_event_channel_op (EVTCHNOP_status, &status); +if (rc <0 ) +continue ; +if (status.status == EVTCHNSTAT_pirq) { + close.port = port; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("xen: failed to close evtchn %d\n" , + port); + unmap_irq.pirq = status.u.pirq; + unmap_irq.domid =DOMID_SELF ; +pr_warn ("xen: unmapping previously mapped pirq %d\n" , + unmap_irq.pirq); +if (HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, + &unmap_irq) !=0 ) +pr_warn ("xen: failed to unmap pirq %d\n" , + unmap_irq.pirq); + } + } +} + +void __init xen_init_IRQ (void ) {int ret = -EINVAL ;@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)xen_callback_vector ();if (xen_hvm_domain ()) { +xen_startup_unmap_pirqs (); +native_init_IRQ ();/* pci_xen_hvm_init must be called after native_init_IRQ so that * __acpi_register_gsi can point at the right function */ include/xen/events.h +3
-0@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);/* Determine whether to ignore this IRQ if it is passed to a guest. */ int xen_test_irq_shared (int irq); +/* Unmap all PIRQs and close event channels */ +void xen_unmap_all_pirqs (void ); +/* initialize Xen IRQ subsystem */ void xen_init_IRQ (void );#endif /* _XEN_EVENTS_H */ Signature · 1 line
1.9.3
Signature · 1 line
Vitaly
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
Quoted 26 lines · Vitaly Kuznetsov
Quoted 24 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).
Quoted 6 lines
Quoted 2 lines
How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?
Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...
I think for FIFO events we can't do much right now - it would need some new hypercalls to de-allocate or such.
But I was thinking that your code logic could just return out when it detects that it is running with FIFO events (with a TODO comment)
- and also spit out some information to this effect?
Say: "Use xen.fifo=0 in your launching kernel"
(don't know the right name for the kernel in which you do 'kexec -e' in ? Is that launching? Original? Bootstrap kernel?)
Quoted 139 lines
Quoted 136 lines
Thanks.
Quoted 58 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/smp.c | 1 + drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 3 files changed, 80 insertions(+)arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +30
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{ +int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ; + + pirq = info->u.pirq.pirq; + gsi = info->u.pirq.gsi; + evtchn = info->evtchn; + irq = info->irq; + +pr_debug ("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n" , + gsi, pirq, irq, evtchn); + +if (evtchn >0 ) { + close.port = evtchn; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("close evtchn %d failed\n" , evtchn); + } + + unmap_irq.pirq = pirq; + unmap_irq.domid =DOMID_SELF ;drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+ + rc =HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, &unmap_irq); +if (rc) +pr_warn ("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n" , + gsi, pirq, irq, rc); + } + +mutex_unlock (&irq_mapping_update_lock); +} +EXPORT_SYMBOL_GPL (xen_unmap_all_pirqs);Why the EXPORT? Is this used by modules?
Quoted 63 lines
drivers/xen/events/events_base.c +36
-0+ +static void xen_startup_unmap_pirqs (void ) +{ +struct evtchn_status status; +int port, rc = -ENOENT ; +struct physdev_unmap_pirq unmap_irq; +struct evtchn_close close; + +memset (&status,0 ,sizeof (status)); +for (port =0 ; port <xen_evtchn_max_channels (); port++) { + status.dom =DOMID_SELF ; + status.port = port; + rc =HYPERVISOR_event_channel_op (EVTCHNOP_status, &status); +if (rc <0 ) +continue ; +if (status.status == EVTCHNSTAT_pirq) { + close.port = port; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("xen: failed to close evtchn %d\n" , + port); + unmap_irq.pirq = status.u.pirq; + unmap_irq.domid =DOMID_SELF ; +pr_warn ("xen: unmapping previously mapped pirq %d\n" , + unmap_irq.pirq); +if (HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, + &unmap_irq) !=0 ) +pr_warn ("xen: failed to unmap pirq %d\n" , + unmap_irq.pirq); + } + } +} + +void __init xen_init_IRQ (void ) {int ret = -EINVAL ;@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)xen_callback_vector ();if (xen_hvm_domain ()) { +xen_startup_unmap_pirqs (); +native_init_IRQ ();/* pci_xen_hvm_init must be called after native_init_IRQ so that * __acpi_register_gsi can point at the right function */ include/xen/events.h +3
-0@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);/* Determine whether to ignore this IRQ if it is passed to a guest. */ int xen_test_irq_shared (int irq); +/* Unmap all PIRQs and close event channels */ +void xen_unmap_all_pirqs (void ); +/* initialize Xen IRQ subsystem */ void xen_init_IRQ (void );#endif /* _XEN_EVENTS_H */ Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to Vitaly KuznetsovQuoted 26 lines · Vitaly Kuznetsov
Quoted 24 lines · Konrad Rzeszutek Wilk
Quoted 22 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).
Quoted 6 lines
Quoted 2 lines
How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?
Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...
I think for FIFO events we can't do much right now - it would need some new hypercalls to de-allocate or such.
But I was thinking that your code logic could just return out when it detects that it is running with FIFO events (with a TODO comment)
- and also spit out some information to this effect?
Say: "Use xen.fifo=0 in your launching kernel"
(don't know the right name for the kernel in which you do 'kexec -e' in ? Is that launching? Original? Bootstrap kernel?)
Quoted 139 lines
Quoted 136 lines
Thanks.
Quoted 58 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/smp.c | 1 + drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 3 files changed, 80 insertions(+)arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +30
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{ +int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ; + + pirq = info->u.pirq.pirq; + gsi = info->u.pirq.gsi; + evtchn = info->evtchn; + irq = info->irq; + +pr_debug ("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n" , + gsi, pirq, irq, evtchn); + +if (evtchn >0 ) { + close.port = evtchn; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("close evtchn %d failed\n" , evtchn); + } + + unmap_irq.pirq = pirq; + unmap_irq.domid =DOMID_SELF ;drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+ + rc =HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, &unmap_irq); +if (rc) +pr_warn ("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n" , + gsi, pirq, irq, rc); + } + +mutex_unlock (&irq_mapping_update_lock); +} +EXPORT_SYMBOL_GPL (xen_unmap_all_pirqs);Why the EXPORT? Is this used by modules?
Quoted 63 lines
drivers/xen/events/events_base.c +36
-0+ +static void xen_startup_unmap_pirqs (void ) +{ +struct evtchn_status status; +int port, rc = -ENOENT ; +struct physdev_unmap_pirq unmap_irq; +struct evtchn_close close; + +memset (&status,0 ,sizeof (status)); +for (port =0 ; port <xen_evtchn_max_channels (); port++) { + status.dom =DOMID_SELF ; + status.port = port; + rc =HYPERVISOR_event_channel_op (EVTCHNOP_status, &status); +if (rc <0 ) +continue ; +if (status.status == EVTCHNSTAT_pirq) { + close.port = port; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("xen: failed to close evtchn %d\n" , + port); + unmap_irq.pirq = status.u.pirq; + unmap_irq.domid =DOMID_SELF ; +pr_warn ("xen: unmapping previously mapped pirq %d\n" , + unmap_irq.pirq); +if (HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, + &unmap_irq) !=0 ) +pr_warn ("xen: failed to unmap pirq %d\n" , + unmap_irq.pirq); + } + } +} + +void __init xen_init_IRQ (void ) {int ret = -EINVAL ;@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)xen_callback_vector ();if (xen_hvm_domain ()) { +xen_startup_unmap_pirqs (); +native_init_IRQ ();/* pci_xen_hvm_init must be called after native_init_IRQ so that * __acpi_register_gsi can point at the right function */ include/xen/events.h +3
-0@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);/* Determine whether to ignore this IRQ if it is passed to a guest. */ int xen_test_irq_shared (int irq); +/* Unmap all PIRQs and close event channels */ +void xen_unmap_all_pirqs (void ); +/* initialize Xen IRQ subsystem */ void xen_init_IRQ (void );#endif /* _XEN_EVENTS_H */ Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
Quoted 30 lines · Konrad Rzeszutek Wilk
Quoted 29 lines · Vitaly Kuznetsov
Quoted 26 lines · Konrad Rzeszutek Wilk
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?
Quoted 7 lines
Quoted 3 lines
Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...
I think for FIFO events we can't do much right now - it would need some new hypercalls to de-allocate or such.
Yeah, you're probably right. I tried wrapping evtchn_fifo_destroy() into 'EVTCHNOP_fifo_destroy' hypercall but it seems some other actions are required as well..
Quoted 3 lines
But I was thinking that your code logic could just return out when it detects that it is running with FIFO events (with a TODO comment)
- and also spit out some information to this effect?
Sure, having TODO here is a good idea.
Quoted 2 lines
Say: "Use xen.fifo=0 in your launching kernel"
s,xen.fifo,xen.fifo_events,
Quoted 3 lines
(don't know the right name for the kernel in which you do 'kexec -e' in ? Is that launching? Original? Bootstrap kernel?)
Yes, if under Xen-4.4 I boot original kernel with "xen.fifo_events=0" I'm able to do kexec with "xen.fifo_events=0" and even without it (but only once :-). Once kernel is booted with FIFO-based event channels enabled no kexec is possible, new kernel gets stuck (I guess vcpuop timer doesn't work..). My patch series brings no difference here..
Thanks,
Quoted 140 lines
Quoted 140 lines
Quoted 137 lines
Thanks.
Quoted 58 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/smp.c | 1 + drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 3 files changed, 80 insertions(+)arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +30
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{ +int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ; + + pirq = info->u.pirq.pirq; + gsi = info->u.pirq.gsi; + evtchn = info->evtchn; + irq = info->irq; + +pr_debug ("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n" , + gsi, pirq, irq, evtchn); + +if (evtchn >0 ) { + close.port = evtchn; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("close evtchn %d failed\n" , evtchn); + } + + unmap_irq.pirq = pirq; + unmap_irq.domid =DOMID_SELF ;drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+ + rc =HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, &unmap_irq); +if (rc) +pr_warn ("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n" , + gsi, pirq, irq, rc); + } + +mutex_unlock (&irq_mapping_update_lock); +} +EXPORT_SYMBOL_GPL (xen_unmap_all_pirqs);Why the EXPORT? Is this used by modules?
Quoted 63 lines
drivers/xen/events/events_base.c +36
-0+ +static void xen_startup_unmap_pirqs (void ) +{ +struct evtchn_status status; +int port, rc = -ENOENT ; +struct physdev_unmap_pirq unmap_irq; +struct evtchn_close close; + +memset (&status,0 ,sizeof (status)); +for (port =0 ; port <xen_evtchn_max_channels (); port++) { + status.dom =DOMID_SELF ; + status.port = port; + rc =HYPERVISOR_event_channel_op (EVTCHNOP_status, &status); +if (rc <0 ) +continue ; +if (status.status == EVTCHNSTAT_pirq) { + close.port = port; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("xen: failed to close evtchn %d\n" , + port); + unmap_irq.pirq = status.u.pirq; + unmap_irq.domid =DOMID_SELF ; +pr_warn ("xen: unmapping previously mapped pirq %d\n" , + unmap_irq.pirq); +if (HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, + &unmap_irq) !=0 ) +pr_warn ("xen: failed to unmap pirq %d\n" , + unmap_irq.pirq); + } + } +} + +void __init xen_init_IRQ (void ) {int ret = -EINVAL ;@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)xen_callback_vector ();if (xen_hvm_domain ()) { +xen_startup_unmap_pirqs (); +native_init_IRQ ();/* pci_xen_hvm_init must be called after native_init_IRQ so that * __acpi_register_gsi can point at the right function */ include/xen/events.h +3
-0@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);/* Determine whether to ignore this IRQ if it is passed to a guest. */ int xen_test_irq_shared (int irq); +/* Unmap all PIRQs and close event channels */ +void xen_unmap_all_pirqs (void ); +/* initialize Xen IRQ subsystem */ void xen_init_IRQ (void );#endif /* _XEN_EVENTS_H */ Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Signature · 1 line
Vitaly
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to Konrad Rzeszutek WilkQuoted 30 lines · Konrad Rzeszutek Wilk
Quoted 29 lines · Vitaly Kuznetsov
Quoted 26 lines · Konrad Rzeszutek Wilk
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).How does this work when you boot the guest under Xen 4.4 where the FIFO events are used? Does it still work correctly?
Quoted 7 lines
Quoted 3 lines
Thanks for pointing that out! I've checked and it doesn't. However patches make no difference - guest kernel gets stuck on boot with and without them. Will try to investigate...
I think for FIFO events we can't do much right now - it would need some new hypercalls to de-allocate or such.
Yeah, you're probably right. I tried wrapping evtchn_fifo_destroy() into 'EVTCHNOP_fifo_destroy' hypercall but it seems some other actions are required as well..
Quoted 3 lines
But I was thinking that your code logic could just return out when it detects that it is running with FIFO events (with a TODO comment)
- and also spit out some information to this effect?
Sure, having TODO here is a good idea.
Quoted 2 lines
Say: "Use xen.fifo=0 in your launching kernel"
s,xen.fifo,xen.fifo_events,
Quoted 3 lines
(don't know the right name for the kernel in which you do 'kexec -e' in ? Is that launching? Original? Bootstrap kernel?)
Yes, if under Xen-4.4 I boot original kernel with "xen.fifo_events=0" I'm able to do kexec with "xen.fifo_events=0" and even without it (but only once :-). Once kernel is booted with FIFO-based event channels enabled no kexec is possible, new kernel gets stuck (I guess vcpuop timer doesn't work..). My patch series brings no difference here..
Thanks,
Quoted 140 lines
Quoted 140 lines
Quoted 137 lines
Thanks.
Quoted 58 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/xen/smp.c | 1 + drivers/xen/events/events_base.c | 76 ++++++++++++++++++++++++++++++++++++++++ include/xen/events.h | 3 ++ 3 files changed, 80 insertions(+)arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +30
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{ +int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ; + + pirq = info->u.pirq.pirq; + gsi = info->u.pirq.gsi; + evtchn = info->evtchn; + irq = info->irq; + +pr_debug ("unmapping pirq gsi=%d pirq=%d irq=%d evtchn=%d\n" , + gsi, pirq, irq, evtchn); + +if (evtchn >0 ) { + close.port = evtchn; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("close evtchn %d failed\n" , evtchn); + } + + unmap_irq.pirq = pirq; + unmap_irq.domid =DOMID_SELF ;drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1685
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+ + rc =HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, &unmap_irq); +if (rc) +pr_warn ("unmap pirq failed gsi=%d pirq=%d irq=%d rc=%d\n" , + gsi, pirq, irq, rc); + } + +mutex_unlock (&irq_mapping_update_lock); +} +EXPORT_SYMBOL_GPL (xen_unmap_all_pirqs);Why the EXPORT? Is this used by modules?
Quoted 63 lines
drivers/xen/events/events_base.c +36
-0+ +static void xen_startup_unmap_pirqs (void ) +{ +struct evtchn_status status; +int port, rc = -ENOENT ; +struct physdev_unmap_pirq unmap_irq; +struct evtchn_close close; + +memset (&status,0 ,sizeof (status)); +for (port =0 ; port <xen_evtchn_max_channels (); port++) { + status.dom =DOMID_SELF ; + status.port = port; + rc =HYPERVISOR_event_channel_op (EVTCHNOP_status, &status); +if (rc <0 ) +continue ; +if (status.status == EVTCHNSTAT_pirq) { + close.port = port; +if (HYPERVISOR_event_channel_op (EVTCHNOP_close, + &close) !=0 ) +pr_warn ("xen: failed to close evtchn %d\n" , + port); + unmap_irq.pirq = status.u.pirq; + unmap_irq.domid =DOMID_SELF ; +pr_warn ("xen: unmapping previously mapped pirq %d\n" , + unmap_irq.pirq); +if (HYPERVISOR_physdev_op (PHYSDEVOP_unmap_pirq, + &unmap_irq) !=0 ) +pr_warn ("xen: failed to unmap pirq %d\n" , + unmap_irq.pirq); + } + } +} + +void __init xen_init_IRQ (void ) {int ret = -EINVAL ;@@ -1671,6 +1745,8 @@ void __init xen_init_IRQ(void)xen_callback_vector ();if (xen_hvm_domain ()) { +xen_startup_unmap_pirqs (); +native_init_IRQ ();/* pci_xen_hvm_init must be called after native_init_IRQ so that * __acpi_register_gsi can point at the right function */ include/xen/events.h +3
-0@@ -122,6 +122,9 @@ int xen_irq_from_gsi(unsigned gsi);/* Determine whether to ignore this IRQ if it is passed to a guest. */ int xen_test_irq_shared (int irq); +/* Unmap all PIRQs and close event channels */ +void xen_unmap_all_pirqs (void ); +/* initialize Xen IRQ subsystem */ void xen_init_IRQ (void );#endif /* _XEN_EVENTS_H */ Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Signature · 1 line
Vitaly
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to Vitaly KuznetsovQuoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).
I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).
Quoted 20 lines
arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +2
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{
drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1657
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ;
I think you need to do this by querying Xen state rather than relying on potentially bad guest state. Particularly since you may crash while holding irq_mapping_update_lock.
EVTCHNOP_status gets you the info you need I think.
David
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).
Quoted 3 lines
I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).
Thank you for your comments!
The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call evtchn_fifo_destroy() so next EVTCHNOP_init_control won't fail. I was intended to put evtchn_fifo_destroy() in EVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest with EVTCHNOP_bind_interdomain (if we remember where they were mapped before) but we can't do it after we did evtchn_fifo_destroy() and we can't rebind them after kexec and performing EVTCHNOP_init_control as we can't remember where these channels were mapped to after kexec/kdump.
I see the following possible solutions:
1) We put evtchn_fifo_destroy() in EVTCHNOP_init_control so EVTCHNOP_init_control can be called twice. No EVTCHNOP_reset is required in that case.
2) Introduce special (e.g. 'EVTCHNOP_fifo_destroy') hypercall to do evtchn_fifo_destroy() without closing all channels. Alternatively we can avoid closing all channels in EVTCHNOP_reset when called with DOMID_SELF (as this mode is not being used atm) -- but that would look unobvious.
3) Keep evtchn_fifo_destroy() in EVTCHNOP_reset but keep console/store channels -- I saw your concerns it is not safe, some sort of additional blocking will be required.
4) Do the remapping boot time (query for store/console channels -> perform EVTCHNOP_reset -> rebind with EVTCHNOP_bind_interdomain).
There is an additional problem: EVTCHNOP_bind_interdomain operation has local port as OUT parameter so we can't guarantee that remapping store/console channels will remap them to the same local channel they were mapped before EVTCHNOP_reset (and we have this information in hvm
info: HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN, ...). Not sure how to deal with that in case we go with remapping.
Your thoughts would be very appreciated. Thank you again,
Quoted 38 lines
Quoted 20 lines
arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +2
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1657
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ;I think you need to do this by querying Xen state rather than relying on potentially bad guest state. Particularly since you may crash while holding
irq_mapping_update_lock.
EVTCHNOP_statusgets you the info you need I think.David
Signature · 1 line
Vitaly
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to David VrabelQuoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).
Quoted 3 lines
I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).
Thank you for your comments!
The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call evtchn_fifo_destroy() so next EVTCHNOP_init_control won't fail. I was intended to put evtchn_fifo_destroy() in EVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest with EVTCHNOP_bind_interdomain (if we remember where they were mapped before) but we can't do it after we did evtchn_fifo_destroy() and we can't rebind them after kexec and performing EVTCHNOP_init_control as we can't remember where these channels were mapped to after kexec/kdump.
I see the following possible solutions:
1) We put evtchn_fifo_destroy() in EVTCHNOP_init_control so EVTCHNOP_init_control can be called twice. No EVTCHNOP_reset is required in that case.
2) Introduce special (e.g. 'EVTCHNOP_fifo_destroy') hypercall to do evtchn_fifo_destroy() without closing all channels. Alternatively we can avoid closing all channels in EVTCHNOP_reset when called with DOMID_SELF (as this mode is not being used atm) -- but that would look unobvious.
3) Keep evtchn_fifo_destroy() in EVTCHNOP_reset but keep console/store channels -- I saw your concerns it is not safe, some sort of additional blocking will be required.
4) Do the remapping boot time (query for store/console channels -> perform EVTCHNOP_reset -> rebind with EVTCHNOP_bind_interdomain).
There is an additional problem: EVTCHNOP_bind_interdomain operation has local port as OUT parameter so we can't guarantee that remapping store/console channels will remap them to the same local channel they were mapped before EVTCHNOP_reset (and we have this information in hvm
info: HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN, ...). Not sure how to deal with that in case we go with remapping.
Your thoughts would be very appreciated. Thank you again,
Quoted 38 lines
Quoted 20 lines
arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +2
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1657
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ;I think you need to do this by querying Xen state rather than relying on potentially bad guest state. Particularly since you may crash while holding
irq_mapping_update_lock.
EVTCHNOP_statusgets you the info you need I think.David
Signature · 1 line
Vitaly
Re: [Xen-devel] [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).Quoted 3 lines
I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).
Thank you for your comments!
The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call
evtchn_fifo_destroy()so nextEVTCHNOP_init_controlwon't fail. I was intended to putevtchn_fifo_destroy()inEVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest withEVTCHNOP_bind_interdomain(if we remember where they were mapped before) but we can't do it after we didevtchn_fifo_destroy()and we can't rebind them after kexec and performingEVTCHNOP_init_controlas we can't remember where these channels were mapped to after kexec/kdump.
Quoted 4 lines
I see the following possible solutions:
1) We putevtchn_fifo_destroy()inEVTCHNOP_init_controlsoEVTCHNOP_init_controlcan be called twice. NoEVTCHNOP_resetis required in that case.
EVTCHNOP_init_control is called for each VCPU so I can't see how this would work.
Quoted 4 lines
2) Introduce special (e.g. '
EVTCHNOP_fifo_destroy') hypercall to doevtchn_fifo_destroy()without closing all channels. Alternatively we can avoid closing all channels inEVTCHNOP_resetwhen called withDOMID_SELF(as this mode is not being used atm) -- but that would look unobvious.
I would try this. The guest prior to kexec would then:
1. Use EVTCHNOP_status to query remote end of console and xenstore event channels.
2. Loop for all event channels:
a. unmap pirq (if required)
b. EVTCHNOP_close
3. EVTCHNOP_fifo_destroy (the implementation of which must verify that no channels are bound).
4. EVTCHNOP_bind_interdomain to rebind the console and xenstore channels.
David
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
Quoted 43 lines · David Vrabel
Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).Quoted 3 lines
I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).
Thank you for your comments!
The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call
evtchn_fifo_destroy()so nextEVTCHNOP_init_controlwon't fail. I was intended to putevtchn_fifo_destroy()inEVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest withEVTCHNOP_bind_interdomain(if we remember where they were mapped before) but we can't do it after we didevtchn_fifo_destroy()and we can't rebind them after kexec and performingEVTCHNOP_init_controlas we can't remember where these channels were mapped to after kexec/kdump.
Quoted 7 lines
Quoted 4 lines
I see the following possible solutions:
1) We putevtchn_fifo_destroy()inEVTCHNOP_init_controlsoEVTCHNOP_init_controlcan be called twice. NoEVTCHNOP_resetis required in that case.
EVTCHNOP_init_controlis called for each VCPU so I can't see how this would work.
Right, forgot about that...
Quoted 13 lines
Quoted 5 lines
2) Introduce special (e.g. '
EVTCHNOP_fifo_destroy') hypercall to doevtchn_fifo_destroy()without closing all channels. Alternatively we can avoid closing all channels inEVTCHNOP_resetwhen called withDOMID_SELF(as this mode is not being used atm) -- but that would look unobvious.I would try this. The guest prior to kexec would then:
1. Use
EVTCHNOP_statusto query remote end of console and xenstore event channels.2. Loop for all event channels:
Quoted 8 lines
a. unmap pirq (if required)
b.EVTCHNOP_close3.
EVTCHNOP_fifo_destroy(the implementation of which must verify that no channels are bound).4.
EVTCHNOP_bind_interdomainto rebind the console and xenstore channels.
Yea, that's what I have now when I put evtchn_fifo_destroy() in EVTCHNOP_reset. The problem here is: we can't do EVTCHNOP_bind_interdomain after we did evtchn_fifo_destroy(), we need to call EVTCHNOP_init_control first. And we'll do that only after kexec so we won't remember what we need to remap.. The second issue is the fact that EVTCHNOP_bind_interdomain will remap store/storage channels to *some* local ports, not necessary matching hvm info (HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN)..
Would it be safe is instead of closing interdomain channels on EVTCHNOP_fifo_destroy we switch evtchn_port_ops to evtchn_port_ops_2l (so on EVTCHNOP_init_control after kexec we switch back)? I'll try prototyping this.
Thank you for your comments,
Quoted 1 line
David
Signature · 1 line
Vitaly
Re: [Xen-devel] [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to David VrabelQuoted 43 lines · David Vrabel
Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).Quoted 3 lines
I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).
Thank you for your comments!
The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call
evtchn_fifo_destroy()so nextEVTCHNOP_init_controlwon't fail. I was intended to putevtchn_fifo_destroy()inEVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest withEVTCHNOP_bind_interdomain(if we remember where they were mapped before) but we can't do it after we didevtchn_fifo_destroy()and we can't rebind them after kexec and performingEVTCHNOP_init_controlas we can't remember where these channels were mapped to after kexec/kdump.
Quoted 7 lines
Quoted 4 lines
I see the following possible solutions:
1) We putevtchn_fifo_destroy()inEVTCHNOP_init_controlsoEVTCHNOP_init_controlcan be called twice. NoEVTCHNOP_resetis required in that case.
EVTCHNOP_init_controlis called for each VCPU so I can't see how this would work.
Right, forgot about that...
Quoted 13 lines
Quoted 5 lines
2) Introduce special (e.g. '
EVTCHNOP_fifo_destroy') hypercall to doevtchn_fifo_destroy()without closing all channels. Alternatively we can avoid closing all channels inEVTCHNOP_resetwhen called withDOMID_SELF(as this mode is not being used atm) -- but that would look unobvious.I would try this. The guest prior to kexec would then:
1. Use
EVTCHNOP_statusto query remote end of console and xenstore event channels.2. Loop for all event channels:
Quoted 8 lines
a. unmap pirq (if required)
b.EVTCHNOP_close3.
EVTCHNOP_fifo_destroy(the implementation of which must verify that no channels are bound).4.
EVTCHNOP_bind_interdomainto rebind the console and xenstore channels.
Yea, that's what I have now when I put evtchn_fifo_destroy() in EVTCHNOP_reset. The problem here is: we can't do EVTCHNOP_bind_interdomain after we did evtchn_fifo_destroy(), we need to call EVTCHNOP_init_control first. And we'll do that only after kexec so we won't remember what we need to remap.. The second issue is the fact that EVTCHNOP_bind_interdomain will remap store/storage channels to *some* local ports, not necessary matching hvm info (HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN)..
Would it be safe is instead of closing interdomain channels on EVTCHNOP_fifo_destroy we switch evtchn_port_ops to evtchn_port_ops_2l (so on EVTCHNOP_init_control after kexec we switch back)? I'll try prototyping this.
Thank you for your comments,
Quoted 1 line
David
Signature · 1 line
Vitaly
Re: [Xen-devel] [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
Quoted 77 lines · Vitaly Kuznetsov
Quoted 43 lines · David Vrabel
Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).Quoted 3 lines
I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).
Thank you for your comments!
The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call
evtchn_fifo_destroy()so nextEVTCHNOP_init_controlwon't fail. I was intended to putevtchn_fifo_destroy()inEVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest withEVTCHNOP_bind_interdomain(if we remember where they were mapped before) but we can't do it after we didevtchn_fifo_destroy()and we can't rebind them after kexec and performingEVTCHNOP_init_controlas we can't remember where these channels were mapped to after kexec/kdump.Quoted 7 lines
Quoted 4 lines
I see the following possible solutions:
1) We putevtchn_fifo_destroy()inEVTCHNOP_init_controlsoEVTCHNOP_init_controlcan be called twice. NoEVTCHNOP_resetis required in that case.
EVTCHNOP_init_controlis called for each VCPU so I can't see how this would work.Right, forgot about that...
Quoted 13 lines
Quoted 5 lines
2) Introduce special (e.g. '
EVTCHNOP_fifo_destroy') hypercall to doevtchn_fifo_destroy()without closing all channels. Alternatively we can avoid closing all channels inEVTCHNOP_resetwhen called withDOMID_SELF(as this mode is not being used atm) -- but that would look unobvious.I would try this. The guest prior to kexec would then:
1. Use
EVTCHNOP_statusto query remote end of console and xenstore event channels.2. Loop for all event channels:
Quoted 8 lines
a. unmap pirq (if required)
b.EVTCHNOP_close3.
EVTCHNOP_fifo_destroy(the implementation of which must verify that no channels are bound).4.
EVTCHNOP_bind_interdomainto rebind the console and xenstore channels.
Quoted 8 lines
Yea, that's what I have now when I put
evtchn_fifo_destroy()inEVTCHNOP_reset. The problem here is: we can't doEVTCHNOP_bind_interdomainafter we didevtchn_fifo_destroy(), we need to callEVTCHNOP_init_controlfirst. And we'll do that only after kexec so we won't remember what we need to remap.. The second issue is the fact thatEVTCHNOP_bind_interdomainwill remap store/storage channels to *some* local ports, not necessary matching hvm info (HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN)..
You can set the HVM params to match.
Quoted 4 lines
Would it be safe is instead of closing interdomain channels on
EVTCHNOP_fifo_destroywe switchevtchn_port_opstoevtchn_port_ops_2l(so onEVTCHNOP_init_controlafter kexec we switch back)? I'll try prototyping this.
Switching the ops back to the 2l ones is an essential part of fifo_destroy (the execed kernel might not have fifo support). I assumed you were already doing this.
David
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to Vitaly KuznetsovQuoted 77 lines · Vitaly Kuznetsov
Quoted 43 lines · David Vrabel
Quoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).Quoted 3 lines
I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).
Thank you for your comments!
The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call
evtchn_fifo_destroy()so nextEVTCHNOP_init_controlwon't fail. I was intended to putevtchn_fifo_destroy()inEVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest withEVTCHNOP_bind_interdomain(if we remember where they were mapped before) but we can't do it after we didevtchn_fifo_destroy()and we can't rebind them after kexec and performingEVTCHNOP_init_controlas we can't remember where these channels were mapped to after kexec/kdump.Quoted 7 lines
Quoted 4 lines
I see the following possible solutions:
1) We putevtchn_fifo_destroy()inEVTCHNOP_init_controlsoEVTCHNOP_init_controlcan be called twice. NoEVTCHNOP_resetis required in that case.
EVTCHNOP_init_controlis called for each VCPU so I can't see how this would work.Right, forgot about that...
Quoted 13 lines
Quoted 5 lines
2) Introduce special (e.g. '
EVTCHNOP_fifo_destroy') hypercall to doevtchn_fifo_destroy()without closing all channels. Alternatively we can avoid closing all channels inEVTCHNOP_resetwhen called withDOMID_SELF(as this mode is not being used atm) -- but that would look unobvious.I would try this. The guest prior to kexec would then:
1. Use
EVTCHNOP_statusto query remote end of console and xenstore event channels.2. Loop for all event channels:
Quoted 8 lines
a. unmap pirq (if required)
b.EVTCHNOP_close3.
EVTCHNOP_fifo_destroy(the implementation of which must verify that no channels are bound).4.
EVTCHNOP_bind_interdomainto rebind the console and xenstore channels.
Quoted 8 lines
Yea, that's what I have now when I put
evtchn_fifo_destroy()inEVTCHNOP_reset. The problem here is: we can't doEVTCHNOP_bind_interdomainafter we didevtchn_fifo_destroy(), we need to callEVTCHNOP_init_controlfirst. And we'll do that only after kexec so we won't remember what we need to remap.. The second issue is the fact thatEVTCHNOP_bind_interdomainwill remap store/storage channels to *some* local ports, not necessary matching hvm info (HVM_PARAM_CONSOLE_EVTCHN/HVM_PARAM_STORE_EVTCHN)..
You can set the HVM params to match.
Quoted 4 lines
Would it be safe is instead of closing interdomain channels on
EVTCHNOP_fifo_destroywe switchevtchn_port_opstoevtchn_port_ops_2l(so onEVTCHNOP_init_controlafter kexec we switch back)? I'll try prototyping this.
Switching the ops back to the 2l ones is an essential part of fifo_destroy (the execed kernel might not have fifo support). I assumed you were already doing this.
David
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to Vitaly KuznetsovQuoted 42 lines · Vitaly Kuznetsov
Quoted 24 lines · David Vrabel
Quoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).Quoted 3 lines
I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).
Thank you for your comments!
The problem I'm fighting wiht atm is: with FIFO-based event channels we need to call
evtchn_fifo_destroy()so nextEVTCHNOP_init_controlwon't fail. I was intended to putevtchn_fifo_destroy()inEVTCHNOP_reset. That introduces a problem: we need to deal with store/console channels. It is possible to remap those from guest withEVTCHNOP_bind_interdomain(if we remember where they were mapped before) but we can't do it after we didevtchn_fifo_destroy()and we can't rebind them after kexec and performingEVTCHNOP_init_controlas we can't remember where these channels were mapped to after kexec/kdump.
Quoted 4 lines
I see the following possible solutions:
1) We putevtchn_fifo_destroy()inEVTCHNOP_init_controlsoEVTCHNOP_init_controlcan be called twice. NoEVTCHNOP_resetis required in that case.
EVTCHNOP_init_control is called for each VCPU so I can't see how this would work.
Quoted 4 lines
2) Introduce special (e.g. '
EVTCHNOP_fifo_destroy') hypercall to doevtchn_fifo_destroy()without closing all channels. Alternatively we can avoid closing all channels inEVTCHNOP_resetwhen called withDOMID_SELF(as this mode is not being used atm) -- but that would look unobvious.
I would try this. The guest prior to kexec would then:
1. Use EVTCHNOP_status to query remote end of console and xenstore event channels.
2. Loop for all event channels:
a. unmap pirq (if required)
b. EVTCHNOP_close
3. EVTCHNOP_fifo_destroy (the implementation of which must verify that no channels are bound).
4. EVTCHNOP_bind_interdomain to rebind the console and xenstore channels.
David
Re: [PATCH RFC 3/4] xen/pvhvm: Unmap all PIRQs on startup and shutdown
In reply to Vitaly KuznetsovQuoted 13 lines · Vitaly Kuznetsov
When kexec is being run PIRQs from Qemu-emulated devices are still mapped to old event channels and new kernel has no information about that. Trying to map them twice results in the following in Xen's dmesg:
(XEN) irq.c:2278: dom7: pirq 24 or emuirq 8 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 12 already mapped (XEN) irq.c:2278: dom7: pirq 24 or emuirq 1 already mapped
...and the following in new kernel's dmesg:
[ 92.286796] xen:events: Failed to obtain physical IRQ 4
Quoted 9 lines
The result is that the new kernel doesn't recieve IRQs for Qemu-emulated devices. Address the issue by unmapping all mapped PIRQs on kernel shutdown when kexec was requested and on every kernel startup. We need to do this twice to deal with the following issues:
- startup-time unmapping is required to make kdump work;
- shutdown-time unmapping is required to support kexec-ing non-fixed kernels;
- shutdown-time unmapping is required to make Qemu-emulated NICs work after kexec (event channel is being closed on shutdown but noPHYSDEVOP_unmap_pirqis being performed).
I think this should be done only in one place -- just prior to exec'ing the new kernel (including kdump kernels).
Quoted 20 lines
arch/x86/xen/smp.c +1
-0@@ -768,6 +768,7 @@ void xen_kexec_shutdown(void)#ifdef CONFIG_KEXEC if (!kexec_in_progress)return ; +xen_unmap_all_pirqs ();#endif }drivers/xen/events/events_base.c +2
-0@@ -1643,6 +1643,80 @@ void xen_callback_vector(void) {}static bool fifo_events = true;module_param (fifo_events, bool,0 ); +void xen_unmap_all_pirqs (void ) +{
drivers/xen/events/events_base.c void xen_callback_vector(void) {} · line 1657
Quoted 10 lines
drivers/xen/events/events_base.c +10
-0+int pirq, rc, gsi, irq, evtchn; +struct physdev_unmap_pirq unmap_irq; +struct irq_info *info; +struct evtchn_close close; + +mutex_lock (&irq_mapping_update_lock); + +list_for_each_entry (info, &xen_irq_list_head, list) { +if (info->type !=IRQT_PIRQ ) +continue ;
I think you need to do this by querying Xen state rather than relying on potentially bad guest state. Particularly since you may crash while holding irq_mapping_update_lock.
EVTCHNOP_status gets you the info you need I think.
David
RFC PATCH 4/4 xen/pvhvm: Make MSI IRQs work after kexec
In reply to Vitaly KuznetsovWhen kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.
RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.
--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;
Signature · 1 line
1.9.3
RFC PATCH 4/4 xen/pvhvm: Make MSI IRQs work after kexec
In reply to Vitaly KuznetsovWhen kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg. xen_irq_from_pirq() fails as we have no IRQ mapping information for that. Requesting for new mapping with __write_msi_msg() does not result in MSI IRQ being remapped so we don't recieve these IRQs.
RFC: I wasn't able to understand why commit af42b8d1 which introduced xen_irq_from_pirq() check in xen_hvm_setup_msi_irqs() is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when __write_msi_msg() with new PIRQ will result in new mapping.
--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/x86/pci/xen.c +1 -2
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;
Signature · 1 line
1.9.3
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.
receive
How come '__write_msi_msg' does not result in new MSI IRQs?
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.
We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.
In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
In reply to Vitaly KuznetsovQuoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.
receive
How come '__write_msi_msg' does not result in new MSI IRQs?
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.
We don't request a new pirq. We end up returning before we call xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.
In regards to why using 'xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.receive
Thanks for your comments!
Quoted 2 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?
Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0
then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54
and we have the following mapping:
xen: msi --> pirq=54 --> irq=72
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),
After kexec we see the following:__read_msi_msg()
pirq -> 54
but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75
However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),
Quoted 4 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.
I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.
But if we're called twice we'll see the same pirq, right? Or there are some cases when we see 'crap' instead of pirq here?
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
Thanks for your comments again!
Quoted 23 lines
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3
Signature · 1 line
Vitaly
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
In reply to Konrad Rzeszutek WilkQuoted 9 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.receive
Thanks for your comments!
Quoted 2 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?
Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0
then we allocate new pirq with
pirq = xen_allocate_pirq_msi()
pirq -> 54
and we have the following mapping:
xen: msi --> pirq=54 --> irq=72
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),
After kexec we see the following:__read_msi_msg()
pirq -> 54
but as xen_irq_from_pirq() fails we follow the same path allocating new pirq:
pirq = xen_allocate_pirq_msi()
pirq -> 55
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75
However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),
Quoted 4 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.
I meant to say that in case we have pirq > 0 from __read_msi_msg() but xen_irq_from_pirq(pirq) fails (kexec-only case?) we always do xen_allocate_pirq_msi() which brings us new pirq.
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.
But if we're called twice we'll see the same pirq, right? Or there are some cases when we see 'crap' instead of pirq here?
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
Thanks for your comments again!
Quoted 23 lines
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3
Signature · 1 line
Vitaly
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.
Quoted 4 lines
Quoted 2 lines
receive
Thanks for your comments!
Thank you for quick turnaround with the answers!
Quoted 30 lines
Quoted 3 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in
xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0then we allocate new pirq with
pirq =xen_allocate_pirq_msi()
pirq -> 54and we have the following mapping:
xen: msi --> pirq=54 --> irq=72in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),After kexec we see the following:
__read_msi_msg()
pirq -> 54but as
xen_irq_from_pirq()fails we follow the same path allocating new pirq:
pirq =xen_allocate_pirq_msi()
pirq -> 55
Quoted 7 lines
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),
I am wondering if that is related to in QEMU traditional:
qemu-xen-trad: free all the pirqs for msi/msix when driver unloads
(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)
If you have that patch in, is the PIRQ value correctly updated?
Quoted 23 lines
Quoted 5 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.I meant to say that in case we have pirq > 0 from
__read_msi_msg()butxen_irq_from_pirq(pirq)fails (kexec-only case?) we always doxen_allocate_pirq_msi()which brings us new pirq.
Quoted 5 lines
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.But if we're called twice we'll see the same pirq, right? Or there are
Good point.
Quoted 1 line
some cases when we see 'crap' instead of pirq here?
For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)
Quoted 3 lines
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
I concur.
Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?
Quoted 29 lines
Thanks for your comments again!
Quoted 23 lines
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
In reply to Vitaly KuznetsovQuoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.
Quoted 4 lines
Quoted 2 lines
receive
Thanks for your comments!
Thank you for quick turnaround with the answers!
Quoted 30 lines
Quoted 3 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in
xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0then we allocate new pirq with
pirq =xen_allocate_pirq_msi()
pirq -> 54and we have the following mapping:
xen: msi --> pirq=54 --> irq=72in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),After kexec we see the following:
__read_msi_msg()
pirq -> 54but as
xen_irq_from_pirq()fails we follow the same path allocating new pirq:
pirq =xen_allocate_pirq_msi()
pirq -> 55
Quoted 7 lines
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),
I am wondering if that is related to in QEMU traditional:
qemu-xen-trad: free all the pirqs for msi/msix when driver unloads
(which in the upstream QEMU is 1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)
If you have that patch in, is the PIRQ value correctly updated?
Quoted 23 lines
Quoted 5 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.I meant to say that in case we have pirq > 0 from
__read_msi_msg()butxen_irq_from_pirq(pirq)fails (kexec-only case?) we always doxen_allocate_pirq_msi()which brings us new pirq.
Quoted 5 lines
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.But if we're called twice we'll see the same pirq, right? Or there are
Good point.
Quoted 1 line
some cases when we see 'crap' instead of pirq here?
For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)
Quoted 3 lines
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
I concur.
Stefano, do you recall why you used xen_irq_from_pirq instead of just trusting the 'pirq' value? Was it to workaround broken QEMU?
Quoted 29 lines
Thanks for your comments again!
Quoted 23 lines
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
Quoted 54 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.Quoted 4 lines
Quoted 2 lines
receive
Thanks for your comments!
Thank you for quick turnaround with the answers!
Quoted 30 lines
Quoted 3 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in
xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0then we allocate new pirq with
pirq =xen_allocate_pirq_msi()
pirq -> 54and we have the following mapping:
xen: msi --> pirq=54 --> irq=72in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),After kexec we see the following:
__read_msi_msg()
pirq -> 54but as
xen_irq_from_pirq()fails we follow the same path allocating new pirq:
pirq =xen_allocate_pirq_msi()
pirq -> 55Quoted 7 lines
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),
Quoted 8 lines
I am wondering if that is related to in QEMU traditional:
qemu-xen-trad: free all the pirqs for msi/msix when driver unloads
(which in the upstream QEMU is
1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)If you have that patch in, is the PIRQ value correctly updated?
Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?
Quoted 29 lines
Quoted 23 lines
Quoted 5 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.I meant to say that in case we have pirq > 0 from
__read_msi_msg()butxen_irq_from_pirq(pirq)fails (kexec-only case?) we always doxen_allocate_pirq_msi()which brings us new pirq.Quoted 5 lines
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.But if we're called twice we'll see the same pirq, right? Or there are
Quoted 7 lines
Good point.
Quoted 1 line
some cases when we see 'crap' instead of pirq here?
For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)
AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.
Quoted 39 lines
Quoted 4 lines
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
I concur.
Stefano, do you recall why you used
xen_irq_from_pirqinstead of just trusting the 'pirq' value? Was it to workaround broken QEMU?Quoted 29 lines
Thanks for your comments again!
Quoted 23 lines
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Signature · 1 line
Vitaly
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
In reply to Konrad Rzeszutek WilkQuoted 54 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.Quoted 4 lines
Quoted 2 lines
receive
Thanks for your comments!
Thank you for quick turnaround with the answers!
Quoted 30 lines
Quoted 3 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in
xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0then we allocate new pirq with
pirq =xen_allocate_pirq_msi()
pirq -> 54and we have the following mapping:
xen: msi --> pirq=54 --> irq=72in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),After kexec we see the following:
__read_msi_msg()
pirq -> 54but as
xen_irq_from_pirq()fails we follow the same path allocating new pirq:
pirq =xen_allocate_pirq_msi()
pirq -> 55Quoted 7 lines
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),
Quoted 8 lines
I am wondering if that is related to in QEMU traditional:
qemu-xen-trad: free all the pirqs for msi/msix when driver unloads
(which in the upstream QEMU is
1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)If you have that patch in, is the PIRQ value correctly updated?
Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?
Quoted 29 lines
Quoted 23 lines
Quoted 5 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.I meant to say that in case we have pirq > 0 from
__read_msi_msg()butxen_irq_from_pirq(pirq)fails (kexec-only case?) we always doxen_allocate_pirq_msi()which brings us new pirq.Quoted 5 lines
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.But if we're called twice we'll see the same pirq, right? Or there are
Quoted 7 lines
Good point.
Quoted 1 line
some cases when we see 'crap' instead of pirq here?
For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)
AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.
Quoted 39 lines
Quoted 4 lines
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
I concur.
Stefano, do you recall why you used
xen_irq_from_pirqinstead of just trusting the 'pirq' value? Was it to workaround broken QEMU?Quoted 29 lines
Thanks for your comments again!
Quoted 23 lines
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3Signature · 1 line
Vitaly
Signature · 1 line
Vitaly
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
Quoted 60 lines · Vitaly Kuznetsov
Quoted 58 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.Quoted 4 lines
Quoted 2 lines
receive
Thanks for your comments!
Thank you for quick turnaround with the answers!
Quoted 30 lines
Quoted 3 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in
xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0then we allocate new pirq with
pirq =xen_allocate_pirq_msi()
pirq -> 54and we have the following mapping:
xen: msi --> pirq=54 --> irq=72in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),After kexec we see the following:
__read_msi_msg()
pirq -> 54but as
xen_irq_from_pirq()fails we follow the same path allocating new pirq:
pirq =xen_allocate_pirq_msi()
pirq -> 55Quoted 7 lines
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),I am wondering if that is related to in QEMU traditional:
qemu-xen-trad: free all the pirqs for msi/msix when driver unloads
Quoted 7 lines
Quoted 4 lines
(which in the upstream QEMU is
1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)If you have that patch in, is the PIRQ value correctly updated?
Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?
Without your patch on older QEMU's with PCI passthrough we won't get any more interrupts after we kexec in the guest right?
As in, this issue happens _only_ with PCI passthrough devices that use MSI or MSI-X?
Still need to get Stefano's view on this.
Quoted 83 lines
Quoted 30 lines
Quoted 24 lines
Quoted 6 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.I meant to say that in case we have pirq > 0 from
__read_msi_msg()butxen_irq_from_pirq(pirq)fails (kexec-only case?) we always doxen_allocate_pirq_msi()which brings us new pirq.Quoted 5 lines
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.But if we're called twice we'll see the same pirq, right? Or there are
Quoted 7 lines
Good point.
Quoted 1 line
some cases when we see 'crap' instead of pirq here?
For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)
AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.
Quoted 39 lines
Quoted 4 lines
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
I concur.
Stefano, do you recall why you used
xen_irq_from_pirqinstead of just trusting the 'pirq' value? Was it to workaround broken QEMU?Quoted 29 lines
Thanks for your comments again!
Quoted 23 lines
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3Signature · 1 line
VitalySignature · 1 line
Vitaly
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
In reply to Vitaly KuznetsovQuoted 60 lines · Vitaly Kuznetsov
Quoted 58 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.Quoted 4 lines
Quoted 2 lines
receive
Thanks for your comments!
Thank you for quick turnaround with the answers!
Quoted 30 lines
Quoted 3 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in
xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0then we allocate new pirq with
pirq =xen_allocate_pirq_msi()
pirq -> 54and we have the following mapping:
xen: msi --> pirq=54 --> irq=72in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),After kexec we see the following:
__read_msi_msg()
pirq -> 54but as
xen_irq_from_pirq()fails we follow the same path allocating new pirq:
pirq =xen_allocate_pirq_msi()
pirq -> 55Quoted 7 lines
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),I am wondering if that is related to in QEMU traditional:
qemu-xen-trad: free all the pirqs for msi/msix when driver unloads
Quoted 7 lines
Quoted 4 lines
(which in the upstream QEMU is
1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)If you have that patch in, is the PIRQ value correctly updated?
Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?
Without your patch on older QEMU's with PCI passthrough we won't get any more interrupts after we kexec in the guest right?
As in, this issue happens _only_ with PCI passthrough devices that use MSI or MSI-X?
Still need to get Stefano's view on this.
Quoted 83 lines
Quoted 30 lines
Quoted 24 lines
Quoted 6 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.I meant to say that in case we have pirq > 0 from
__read_msi_msg()butxen_irq_from_pirq(pirq)fails (kexec-only case?) we always doxen_allocate_pirq_msi()which brings us new pirq.Quoted 5 lines
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.But if we're called twice we'll see the same pirq, right? Or there are
Quoted 7 lines
Good point.
Quoted 1 line
some cases when we see 'crap' instead of pirq here?
For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)
AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.
Quoted 39 lines
Quoted 4 lines
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
I concur.
Stefano, do you recall why you used
xen_irq_from_pirqinstead of just trusting the 'pirq' value? Was it to workaround broken QEMU?Quoted 29 lines
Thanks for your comments again!
Quoted 23 lines
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3Signature · 1 line
VitalySignature · 1 line
Vitaly
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
Quoted 63 lines · Konrad Rzeszutek Wilk
Quoted 62 lines · Vitaly Kuznetsov
Quoted 60 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.Quoted 4 lines
Quoted 2 lines
receive
Thanks for your comments!
Thank you for quick turnaround with the answers!
Quoted 30 lines
Quoted 3 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in
xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0then we allocate new pirq with
pirq =xen_allocate_pirq_msi()
pirq -> 54and we have the following mapping:
xen: msi --> pirq=54 --> irq=72in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),After kexec we see the following:
__read_msi_msg()
pirq -> 54but as
xen_irq_from_pirq()fails we follow the same path allocating new pirq:
pirq =xen_allocate_pirq_msi()
pirq -> 55Quoted 7 lines
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),I am wondering if that is related to in QEMU traditional:
qemu-xen-trad: free all the pirqs for msi/msix when driver unloads
(which in the upstream QEMU is
1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
If you have that patch in, is the PIRQ value correctly updated?
Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?
Without your patch on older QEMU's with PCI passthrough we won't get any more interrupts after we kexec in the guest right?
Correct.
Quoted 2 lines
As in, this issue happens _only_ with PCI passthrough devices that use MSI or MSI-X?
I haven't tested MSI-X but in theory yes, only MSI and MSI-X passthrough-ed devices are affected.
Quoted 3 lines
Still need to get Stefano's view on this.
Sure, thanks!
Quoted 83 lines
Quoted 83 lines
Quoted 30 lines
Quoted 24 lines
Quoted 6 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.I meant to say that in case we have pirq > 0 from
__read_msi_msg()butxen_irq_from_pirq(pirq)fails (kexec-only case?) we always doxen_allocate_pirq_msi()which brings us new pirq.Quoted 5 lines
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.But if we're called twice we'll see the same pirq, right? Or there are
Quoted 7 lines
Good point.
Quoted 1 line
some cases when we see 'crap' instead of pirq here?
For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)
AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.
Quoted 39 lines
Quoted 4 lines
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
I concur.
Stefano, do you recall why you used
xen_irq_from_pirqinstead of just trusting the 'pirq' value? Was it to workaround broken QEMU?Quoted 29 lines
Thanks for your comments again!
Quoted 23 lines
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3Signature · 1 line
VitalySignature · 1 line
Vitaly
Signature · 1 line
Vitaly
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
In reply to Konrad Rzeszutek WilkQuoted 63 lines · Konrad Rzeszutek Wilk
Quoted 62 lines · Vitaly Kuznetsov
Quoted 60 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.Quoted 4 lines
Quoted 2 lines
receive
Thanks for your comments!
Thank you for quick turnaround with the answers!
Quoted 30 lines
Quoted 3 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in
xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0then we allocate new pirq with
pirq =xen_allocate_pirq_msi()
pirq -> 54and we have the following mapping:
xen: msi --> pirq=54 --> irq=72in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),After kexec we see the following:
__read_msi_msg()
pirq -> 54but as
xen_irq_from_pirq()fails we follow the same path allocating new pirq:
pirq =xen_allocate_pirq_msi()
pirq -> 55Quoted 7 lines
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),I am wondering if that is related to in QEMU traditional:
qemu-xen-trad: free all the pirqs for msi/msix when driver unloads
(which in the upstream QEMU is
1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
If you have that patch in, is the PIRQ value correctly updated?
Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?
Without your patch on older QEMU's with PCI passthrough we won't get any more interrupts after we kexec in the guest right?
Correct.
Quoted 2 lines
As in, this issue happens _only_ with PCI passthrough devices that use MSI or MSI-X?
I haven't tested MSI-X but in theory yes, only MSI and MSI-X passthrough-ed devices are affected.
Quoted 3 lines
Still need to get Stefano's view on this.
Sure, thanks!
Quoted 83 lines
Quoted 83 lines
Quoted 30 lines
Quoted 24 lines
Quoted 6 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.I meant to say that in case we have pirq > 0 from
__read_msi_msg()butxen_irq_from_pirq(pirq)fails (kexec-only case?) we always doxen_allocate_pirq_msi()which brings us new pirq.Quoted 5 lines
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.But if we're called twice we'll see the same pirq, right? Or there are
Quoted 7 lines
Good point.
Quoted 1 line
some cases when we see 'crap' instead of pirq here?
For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)
AFAICT emulated e1000 doesn't use MSI (at least with qemu-tradidtional) and with my patch series it works after kexec.
Quoted 39 lines
Quoted 4 lines
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
I concur.
Stefano, do you recall why you used
xen_irq_from_pirqinstead of just trusting the 'pirq' value? Was it to workaround broken QEMU?Quoted 29 lines
Thanks for your comments again!
Quoted 23 lines
Quoted 23 lines
Signed-off-by Vitaly Kuznetsov <[email protected]>--- arch/x86/pci/xen.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)arch/x86/pci/xen.c +1
-2@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA || pirq <=0 ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;Signature · 1 line
1.9.3Signature · 1 line
VitalySignature · 1 line
Vitaly
Signature · 1 line
Vitaly
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
In reply to Vitaly KuznetsovQuoted 3 lines · Vitaly Kuznetsov
Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?
I don't think so.
David
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
In reply to Vitaly KuznetsovQuoted 3 lines · Vitaly Kuznetsov
Thanks, that really works! I tested both kexec -e / kdump cases. I'm wondering if we although need my commit to workaround non-fixed qemus?
I don't think so.
David
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
In reply to Konrad Rzeszutek WilkQuoted 100 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.Quoted 4 lines
Quoted 2 lines
receive
Thanks for your comments!
Thank you for quick turnaround with the answers!
Quoted 30 lines
Quoted 3 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in
xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0then we allocate new pirq with
pirq =xen_allocate_pirq_msi()
pirq -> 54and we have the following mapping:
xen: msi --> pirq=54 --> irq=72in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),After kexec we see the following:
__read_msi_msg()
pirq -> 54but as
xen_irq_from_pirq()fails we follow the same path allocating new pirq:
pirq =xen_allocate_pirq_msi()
pirq -> 55Quoted 7 lines
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),I am wondering if that is related to in QEMU traditional:
qemu-xen-trad: free all the pirqs for msi/msix when driver unloads
(which in the upstream QEMU is
1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)If you have that patch in, is the PIRQ value correctly updated?
Quoted 23 lines
Quoted 5 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.I meant to say that in case we have pirq > 0 from
__read_msi_msg()butxen_irq_from_pirq(pirq)fails (kexec-only case?) we always doxen_allocate_pirq_msi()which brings us new pirq.Quoted 5 lines
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.But if we're called twice we'll see the same pirq, right? Or there are
Good point.
Quoted 1 line
some cases when we see 'crap' instead of pirq here?
For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
I concur.
Stefano, do you recall why you used
xen_irq_from_pirqinstead of just trusting the 'pirq' value? Was it to workaround broken QEMU?
If I recall correctly the problem is that pirq == 0 is a valid pirq number. So the check pirq <= 0 is wrong.
Can we rely on the fact that msg.data is always 0 on first read? If so, then we could simply:
arch/x86/pci/xen.c +1 -2
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;
Re: [PATCH RFC 4/4] xen/pvhvm: Make MSI IRQs work after kexec
In reply to Konrad Rzeszutek WilkQuoted 100 lines · Konrad Rzeszutek Wilk
Quoted 9 lines · Vitaly Kuznetsov
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Vitaly Kuznetsov
When kexec was peformed MSI IRQs for passthrough-ed devices were already mapped and we see non-zero pirq extracted from MSI msg.
xen_irq_from_pirq()fails as we have no IRQ mapping information for that. Requesting for new mapping with__write_msi_msg()does not result in MSI IRQ being remapped so we don't recieve these IRQs.Quoted 4 lines
Quoted 2 lines
receive
Thanks for your comments!
Thank you for quick turnaround with the answers!
Quoted 30 lines
Quoted 3 lines
How come '
__write_msi_msg' does not result in new MSI IRQs?Actually that was the hidden question in my RFC :-)
Let me describe what I see. When normal boot is performed we have the following in
xen_hvm_setup_msi_irqs():
__read_msi_msg()
pirq -> 0then we allocate new pirq with
pirq =xen_allocate_pirq_msi()
pirq -> 54and we have the following mapping:
xen: msi --> pirq=54 --> irq=72in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:04 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(----),After kexec we see the following:
__read_msi_msg()
pirq -> 54but as
xen_irq_from_pirq()fails we follow the same path allocating new pirq:
pirq =xen_allocate_pirq_msi()
pirq -> 55Quoted 7 lines
and we have the following mapping:
xen: msi --> pirq=55 --> irq=75However (afaict) mapping in xen wasn't updated:
in 'xl debug-keys i':
(XEN) IRQ: 29 affinity:02 vec:b9 type=PCI-MSI status=00000030 in-flight=0 domain-list=7: 54(--M-),I am wondering if that is related to in QEMU traditional:
qemu-xen-trad: free all the pirqs for msi/msix when driver unloads
(which in the upstream QEMU is
1d4fd4f0e2fc5dcae0c60e00cc9af95f52988050)If you have that patch in, is the PIRQ value correctly updated?
Quoted 23 lines
Quoted 5 lines
Is it fair to state that your code ends up reading the MSI IRQ (PIRQ) from the device and updating the internal PIRQ<->IRQ code to match with the reality?
Yea, 'always trust the device'.
Quoted 10 lines
Quoted 6 lines
RFC: I wasn't able to understand why commit
af42b8d1which introducedxen_irq_from_pirq()check inxen_hvm_setup_msi_irqs()is checking that instead of checking pirq > 0 as if the mapping was already done (and we have pirq>0 here) we don't need to request for a new pirq. We're loosing existing PIRQ and I'm also not sure when__write_msi_msg()with new PIRQ will result in new mapping.We don't request a new pirq. We end up returning before we call
xen_allocate_pirq_msi. At least that is how the commit you mentioned worked.I meant to say that in case we have pirq > 0 from
__read_msi_msg()butxen_irq_from_pirq(pirq)fails (kexec-only case?) we always doxen_allocate_pirq_msi()which brings us new pirq.Quoted 5 lines
Quoted 3 lines
In regards to why using '
xen_irq_from_pirq' instead of just checking the PIRQ - is that we might be called twice by a buggy driver. As such we want to check our PIRQ<->IRQ to figure this out.But if we're called twice we'll see the same pirq, right? Or there are
Good point.
Quoted 1 line
some cases when we see 'crap' instead of pirq here?
For PCI passthrough devices they will be zero until they are enabled. But I am not sure about the emulated devices, such as e1000 or such, which would also go through this path (I think - do we have MSI devices that we emulate in QEMU?)
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
I think it would be nice to use the same pirq after kexec instead of allocating a new one even in case we can make remapping work.
I concur.
Stefano, do you recall why you used
xen_irq_from_pirqinstead of just trusting the 'pirq' value? Was it to workaround broken QEMU?
If I recall correctly the problem is that pirq == 0 is a valid pirq number. So the check pirq <= 0 is wrong.
Can we rely on the fact that msg.data is always 0 on first read? If so, then we could simply:
arch/x86/pci/xen.c +1 -2
@@ -231,8 +231,7 @@ static int xen_hvm_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)__read_msi_msg (msidesc, &msg); pirq =MSI_ADDR_EXT_DEST_ID (msg.address_hi) | ((msg.address_lo >>MSI_ADDR_DEST_ID_SHIFT ) &0xff ); -if (msg.data !=XEN_PIRQ_MSI_DATA || -xen_irq_from_pirq (pirq) <0 ) { +if (msg.data !=XEN_PIRQ_MSI_DATA ) { pirq =xen_allocate_pirq_msi (dev, msidesc);if (pirq <0 ) { irq = -ENODEV ;
Quoted 8 lines · Vitaly Kuznetsov
With this patch series I'm trying to address several issues with kexec on pvhvm:
-shared_infoissue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.
In addition to the fixes for multi-VCPU guests, what else remains?
What's the plan for handling granted pages?
I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.
David
Quoted 8 lines · Vitaly Kuznetsov
With this patch series I'm trying to address several issues with kexec on pvhvm:
-shared_infoissue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.
In addition to the fixes for multi-VCPU guests, what else remains?
What's the plan for handling granted pages?
I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.
David
Quoted 7 lines · David Vrabel
Quoted 6 lines · Vitaly Kuznetsov
With this patch series I'm trying to address several issues with kexec on pvhvm:
-shared_infoissue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)
Quoted 5 lines
Quoted 2 lines
This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.
In addition to the fixes for multi-VCPU guests, what else remains?
I'm aware of grants and ballooned out pages.
Quoted 2 lines
What's the plan for handling granted pages?
(if I got the design right) we have two issues:
1) Pages we grant access to other domains. We have the list so we can try doing gnttab_end_foreign_access for all unmapped grants but there is nothing we can do with mapped ones from guest. We can either assume that all such usages are short-term and try waiting for them to finish or we need to do something like force-unmap from hypervisor side.
2) Pages we mapped from other domains. There is no easy way to collect all grant handles from different places in kernel atm so I can see two possible solutions:
- we keep track of all handles with new kernel structure in guest and unmap them all on kexec/kdump.
- we introduce new GNTTABOP_reset which does something similar to gnttab_release_mappings().
There is nothing we need to do with transferred grants (and I don't see transfer usages in kernel).
Please correct me if I'm wrong.
Quoted 2 lines
I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.
*I think* we can leave ballooned out pages out of scope for now.
Thanks,
Quoted 2 lines
David
Signature · 1 line
Vitaly
Quoted 7 lines · David Vrabel
Quoted 6 lines · Vitaly Kuznetsov
With this patch series I'm trying to address several issues with kexec on pvhvm:
-shared_infoissue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)
Quoted 5 lines
Quoted 2 lines
This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.
In addition to the fixes for multi-VCPU guests, what else remains?
I'm aware of grants and ballooned out pages.
Quoted 2 lines
What's the plan for handling granted pages?
(if I got the design right) we have two issues:
1) Pages we grant access to other domains. We have the list so we can try doing gnttab_end_foreign_access for all unmapped grants but there is nothing we can do with mapped ones from guest. We can either assume that all such usages are short-term and try waiting for them to finish or we need to do something like force-unmap from hypervisor side.
2) Pages we mapped from other domains. There is no easy way to collect all grant handles from different places in kernel atm so I can see two possible solutions:
- we keep track of all handles with new kernel structure in guest and unmap them all on kexec/kdump.
- we introduce new GNTTABOP_reset which does something similar to gnttab_release_mappings().
There is nothing we need to do with transferred grants (and I don't see transfer usages in kernel).
Please correct me if I'm wrong.
Quoted 2 lines
I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.
*I think* we can leave ballooned out pages out of scope for now.
Thanks,
Quoted 2 lines
David
Signature · 1 line
Vitaly
Quoted 20 lines · Vitaly Kuznetsov
Quoted 7 lines · David Vrabel
Quoted 6 lines · Vitaly Kuznetsov
With this patch series I'm trying to address several issues with kexec on pvhvm:
-shared_infoissue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)Quoted 5 lines
Quoted 2 lines
This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.
In addition to the fixes for multi-VCPU guests, what else remains?
I'm aware of grants and ballooned out pages.
Quoted 2 lines
What's the plan for handling granted pages?
Quoted 7 lines
(if I got the design right) we have two issues:
1) Pages we grant access to other domains. We have the list so we can try doing
gnttab_end_foreign_accessfor all unmapped grants but there is nothing we can do with mapped ones from guest. We can either assume that all such usages are short-term and try waiting for them to finish or we need to do something like force-unmap from hypervisor side.
Shared rings and persistent grants (used in blkfront) remain mapped for long periods so just waiting won't work.
Force unmap by the hypervisor might be a possibility but the hypervisor needs to atomically replace the grant mapping with a different valid mapping, or the force unmap would cause the backend that was accesses the pages to fault.
Every writable mapping would have to be replaced with a mapping to a unique page (to prevent information leaking between different granted pages). Read-only mappings could be replaces with a read-only mapping to shared zero page safely.
The only way I can see how to do this requires co-operation from the backend kernel -- it would need to provide replacement frames for every grant map. Xen would then use this frame when force-unmapping (revoking) the mapping.
Quoted 7 lines
2) Pages we mapped from other domains. There is no easy way to collect all grant handles from different places in kernel atm so I can see two possible solutions:
- we keep track of all handles with new kernel structure in guest and unmap them all on kexec/kdump.
- we introduce newGNTTABOP_resetwhich does something similar tognttab_release_mappings().
I think you can ignore this for now -- frontend drivers do not grant map, but see suggestion about kexec_is_safe below.
Quoted 2 lines
There is nothing we need to do with transferred grants (and I don't see transfer usages in kernel).
Agreed.
Quoted 4 lines
Quoted 2 lines
I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.
*I think* we can leave ballooned out pages out of scope for now.
I'm thinking we want a global flag (kexec_is_safe) that is cleared even any unsafe operation (e.g., ballooning, grant mapping) is done by the guest. kexec load and exec can then be made to fail if this flag is clear.
This would allow you to fix only the use cases you're interested in without introducing subtle failures if someone tries an unsupported configurations.
David
Quoted 42 lines · David Vrabel
Quoted 20 lines · Vitaly Kuznetsov
Quoted 7 lines · David Vrabel
Quoted 6 lines · Vitaly Kuznetsov
With this patch series I'm trying to address several issues with kexec on pvhvm:
-shared_infoissue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)Quoted 5 lines
Quoted 2 lines
This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.
In addition to the fixes for multi-VCPU guests, what else remains?
I'm aware of grants and ballooned out pages.
Quoted 2 lines
What's the plan for handling granted pages?
Quoted 7 lines
(if I got the design right) we have two issues:
1) Pages we grant access to other domains. We have the list so we can try doing
gnttab_end_foreign_accessfor all unmapped grants but there is nothing we can do with mapped ones from guest. We can either assume that all such usages are short-term and try waiting for them to finish or we need to do something like force-unmap from hypervisor side.Shared rings and persistent grants (used in blkfront) remain mapped for long periods so just waiting won't work.
Force unmap by the hypervisor might be a possibility but the hypervisor needs to atomically replace the grant mapping with a different valid mapping, or the force unmap would cause the backend that was accesses the pages to fault.
Every writable mapping would have to be replaced with a mapping to a unique page (to prevent information leaking between different granted pages). Read-only mappings could be replaces with a read-only mapping to shared zero page safely.
Quoted 4 lines
The only way I can see how to do this requires co-operation from the backend kernel -- it would need to provide replacement frames for every grant map. Xen would then use this frame when force-unmapping (revoking) the mapping.
We can introduce something like GNTTABOP_safe_map_grant_ref op with such replacement frames and use then in 'force unmap' but to be honest I'd like to avoid that.. What if xen could allocate and provide replacement pages on its own? I understand we need to take some precautions against malicious domains hogging all memory with such approach..
Actually if we narrow down our case to backend/frontend usage only we can think that grants do not introduce any issue for kexec/kdump because:
- frontends are being closed on kexec so backends unmap corresponding grants. That's not true for persistent grant case as persistent grants are being unmaped only in xen_blkif_free() but not in xen_blkif_disconnect() atm. The following patch helps:
(quoted diff) +0 -0
b/drivers/block/xen-blkback/xenbus.c
index 3a8b810..54f4089 100644
drivers/block/xen-blkback/xenbus.c +0 -0
@@ -270,6 +270,9 @@ static int xen_blkif_disconnect(struct xen_blkif
*blkif)
blkif->blk_rings.common.sring = NULL;
}+ /* Remove all persistent grants and the cache of ballooned
pages. */
+ xen_blkbk_free_caches(blkif);
+
return 0;
}
@@ -281,9 +284,6 @@ static void xen_blkif_free(struct xen_blkif *blkif)
xen_blkif_disconnect(blkif);
xen_vbd_free(&blkif->vbd);- /* Remove all persistent grants and the cache of ballooned
pages. */
- xen_blkbk_free_caches(blkif);
-
/* Make sure everything is drained before shutting down */
BUG_ON(blkif->persistent_gnt_c != 0);
BUG_ON(atomic_read(&blkif->persistent_gnt_in_use) != 0);- in kdump case our new kernel lives in its own space so we won't hit granted pages (we can read some crap when collecting memory dump but it's not any better in case these pages were unmapped)
- frontends are being reinitialized on new kernel startup, that also causes the unmap (with the patch from above applied).
So I think we need to do something with grants in two cases: they were used outside frontends/backends and we have no idea if they are going to be unmapped (can be solved with kexec_is_safe flag) and when backend got stuck and refused to close/unmap (should be rare).
Quoted 22 lines
Quoted 8 lines
2) Pages we mapped from other domains. There is no easy way to collect all grant handles from different places in kernel atm so I can see two possible solutions:
- we keep track of all handles with new kernel structure in guest and unmap them all on kexec/kdump.
- we introduce newGNTTABOP_resetwhich does something similar tognttab_release_mappings().I think you can ignore this for now -- frontend drivers do not grant map, but see suggestion about
kexec_is_safebelow.Quoted 2 lines
There is nothing we need to do with transferred grants (and I don't see transfer usages in kernel).
Agreed.
Quoted 4 lines
Quoted 2 lines
I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.
*I think* we can leave ballooned out pages out of scope for now.
Quoted 7 lines
I'm thinking we want a global flag (
kexec_is_safe) that is cleared even any unsafe operation (e.g., ballooning, grant mapping) is done by the guest. kexec load and exec can then be made to fail if this flag is clear.This would allow you to fix only the use cases you're interested in without introducing subtle failures if someone tries an unsupported configurations.
Makes sense but it would also be nice to have this flag supported in kexec utility itself (so it will report something like "kexec is not safe, use '-f' to override...).
Thanks,
Quoted 2 lines
David
Signature · 1 line
Vitaly
Quoted 20 lines · Vitaly Kuznetsov
Quoted 7 lines · David Vrabel
Quoted 6 lines · Vitaly Kuznetsov
With this patch series I'm trying to address several issues with kexec on pvhvm:
-shared_infoissue (1st patch, just sending Olaf's work with Konrad's fix)
- create specific pvhvm shutdown handler for kexec (2nd patch)
- GSI PIRQ issue (3rd patch, I'm pretty confident that it does the right thing)
- MSI PIRQ issue (4th patch, and I'm not sure it doesn't break anything -> RFC)Quoted 5 lines
Quoted 2 lines
This patch series can be tested on single vCPU guest. We still have SMP issues with pvhvm guests and kexec which require additional fixes.
In addition to the fixes for multi-VCPU guests, what else remains?
I'm aware of grants and ballooned out pages.
Quoted 2 lines
What's the plan for handling granted pages?
Quoted 7 lines
(if I got the design right) we have two issues:
1) Pages we grant access to other domains. We have the list so we can try doing
gnttab_end_foreign_accessfor all unmapped grants but there is nothing we can do with mapped ones from guest. We can either assume that all such usages are short-term and try waiting for them to finish or we need to do something like force-unmap from hypervisor side.
Shared rings and persistent grants (used in blkfront) remain mapped for long periods so just waiting won't work.
Force unmap by the hypervisor might be a possibility but the hypervisor needs to atomically replace the grant mapping with a different valid mapping, or the force unmap would cause the backend that was accesses the pages to fault.
Every writable mapping would have to be replaced with a mapping to a unique page (to prevent information leaking between different granted pages). Read-only mappings could be replaces with a read-only mapping to shared zero page safely.
The only way I can see how to do this requires co-operation from the backend kernel -- it would need to provide replacement frames for every grant map. Xen would then use this frame when force-unmapping (revoking) the mapping.
Quoted 7 lines
2) Pages we mapped from other domains. There is no easy way to collect all grant handles from different places in kernel atm so I can see two possible solutions:
- we keep track of all handles with new kernel structure in guest and unmap them all on kexec/kdump.
- we introduce newGNTTABOP_resetwhich does something similar tognttab_release_mappings().
I think you can ignore this for now -- frontend drivers do not grant map, but see suggestion about kexec_is_safe below.
Quoted 2 lines
There is nothing we need to do with transferred grants (and I don't see transfer usages in kernel).
Agreed.
Quoted 4 lines
Quoted 2 lines
I don't think we want to accept a partial solution unless the known non-working configurations fail fast on kexec load.
*I think* we can leave ballooned out pages out of scope for now.
I'm thinking we want a global flag (kexec_is_safe) that is cleared even any unsafe operation (e.g., ballooning, grant mapping) is done by the guest. kexec load and exec can then be made to fail if this flag is clear.
This would allow you to fix only the use cases you're interested in without introducing subtle failures if someone tries an unsupported configurations.
David