x86/xen: Make idle tasks reliable
Subsystems (3)
Miroslav Benes
maintainerv2 cover x86/xen: Make idle tasks reliable
The unwinder reports idle tasks' stack on XEN PV as unreliable which complicates things for at least live patching. The two patches in the series try to amend that by using similar approach as non-XEN x86 does.
v1->v2:
- call instruction used instead of push+jmp
- initial_stack used directly
There is a thing which makes me slightly uncomfortable. s/jmp/call/ means that, theoretically, the called function could return. GCC then generates not so nice code and there is asm_cpu_bringup_and_idle+0x5/0x1000 symbol last on the stack due to alignment in asm/x86/xen/xen-head.S which could be confusing. Practically it is all fine, because neither xen_start_kernel(), nor cpu_bringup_and_idle() return (there is unbounded loop in cpu_startup_entry() around do_idle()). __noreturn annotation of these functions did not help.
So I don't think it is really a problem, but one may wonder.
Miroslav Benes (2):
x86/xen: Make the boot CPU idle task reliable x86/xen: Make the secondary CPU idle tasks reliable
arch/x86/xen/smp_pv.c | 3 ++- arch/x86/xen/xen-head.S | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 3 deletions(-)
Signature · 1 line
2.25.1
Miroslav Benes
maintainerPATCH v2 1/2 x86/xen: Make the boot CPU idle task reliable
The unwinder reports the boot CPU idle task's stack on XEN PV as unreliable, which affects at least live patching. There are two reasons for this. First, the task does not follow the x86 convention that its stack starts at the offset right below saved pt_regs. It allows the unwinder to easily detect the end of the stack and verify it. Second, startup_xen() function does not store the return address before jumping to xen_start_kernel() which confuses the unwinder.
Amend both issues by moving the starting point of initial stack in startup_xen() and storing the return address before the jump, which is exactly what call instruction does.
--- arch/x86/xen/xen-head.S | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
arch/x86/xen/xen-head.S +6 -2
@@ -35,7 +35,11 @@ SYM_CODE_START(startup_xen) rep __ASM_SIZE(stos) mov %_ASM_SI, xen_start_info - mov $init_thread_union+THREAD_SIZE, %_ASM_SP +#ifdef CONFIG_X86_64 + mov initial_stack(%rip), %_ASM_SP +#else + mov pa(initial_stack), %_ASM_SP +#endif #ifdef CONFIG_X86_64 /* Set up %gs.
@@ -51,7 +55,7 @@ SYM_CODE_START(startup_xen) wrmsr #endif - jmp xen_start_kernel + call xen_start_kernel SYM_CODE_END(startup_xen) __FINIT #endif
Signature · 1 line
2.25.1
Re: [PATCH v2 1/2] x86/xen: Make the boot CPU idle task reliable
Quoted 22 lines · Miroslav Benes
The unwinder reports the boot CPU idle task's stack on XEN PV as unreliable, which affects at least live patching. There are two reasons for this. First, the task does not follow the x86 convention that its stack starts at the offset right below saved
pt_regs. It allows the unwinder to easily detect the end of the stack and verify it. Second,startup_xen()function does not store the return address before jumping toxen_start_kernel()which confuses the unwinder.Amend both issues by moving the starting point of initial stack in
startup_xen()and storing the return address before the jump, which is exactly what call instruction does.Signed-off-by Miroslav Benes <[email protected]>--- arch/x86/xen/xen-head.S | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)arch/x86/xen/xen-head.S +0
-0
arch/x86/xen/xen-head.S SYM_CODE_START(startup_xen) · line 42
Quoted 9 lines
arch/x86/xen/xen-head.S +5
-1@@ -35,7 +35,11 @@ SYM_CODE_START(startup_xen) rep __ASM_SIZE(stos) mov %_ASM_SI, xen_start_info - mov $init_thread_union+THREAD_SIZE, %_ASM_SP +#ifdef CONFIG_X86_64 + mov initial_stack(%rip), %_ASM_SP +#else + mov pa(initial_stack), %_ASM_SP +#endif
If you need to distinguish the two anyway, why not use %rsp and %esp respectively?
Jan
Miroslav Benes
maintainerRe: [PATCH v2 1/2] x86/xen: Make the boot CPU idle task reliable
Quoted 25 lines · Jan Beulich
Quoted 24 lines · Miroslav Benes
The unwinder reports the boot CPU idle task's stack on XEN PV as unreliable, which affects at least live patching. There are two reasons for this. First, the task does not follow the x86 convention that its stack starts at the offset right below saved
pt_regs. It allows the unwinder to easily detect the end of the stack and verify it. Second,startup_xen()function does not store the return address before jumping toxen_start_kernel()which confuses the unwinder.Amend both issues by moving the starting point of initial stack in
startup_xen()and storing the return address before the jump, which is exactly what call instruction does.Signed-off-by Miroslav Benes <[email protected]>--- arch/x86/xen/xen-head.S | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)arch/x86/xen/xen-head.S +0
-0@@ -35,7 +35,11 @@ SYM_CODE_START(startup_xen) rep __ASM_SIZE(stos)
Quoted 10 lines
arch/x86/xen/xen-head.S SYM_CODE_START(startup_xen) · line 42
Quoted 7 lines
arch/x86/xen/xen-head.S +5
-1mov %_ASM_SI, xen_start_info - mov $init_thread_union+THREAD_SIZE, %_ASM_SP +#ifdef CONFIG_X86_64 + mov initial_stack(%rip), %_ASM_SP +#else + mov pa(initial_stack), %_ASM_SP +#endifIf you need to distinguish the two anyway, why not use %rsp and %esp respectively?
I could, I just preferred the unification instead. Will change it if you think it would be better.
Miroslav
Miroslav Benes
maintainerPATCH v2 2/2 x86/xen: Make the secondary CPU idle tasks reliable
In reply to Miroslav BenesThe unwinder reports the secondary CPU idle tasks' stack on XEN PV as unreliable, which affects at least live patching. cpu_initialize_context() sets up the context of the CPU through VCPUOP_initialise hypercall. After it is woken up, the idle task starts in cpu_bringup_and_idle() function and its stack starts at the offset right below pt_regs. The unwinder correctly detects the end of stack there but it is confused by NULL return address in the last frame.
Introduce a wrapper in assembly, which just calls cpu_bringup_and_idle(). The return address is thus pushed on the stack and the wrapper contains the annotation hint for the unwinder regarding the stack state.
--- arch/x86/xen/smp_pv.c | 3 ++- arch/x86/xen/xen-head.S | 8 ++++++++ 2 files changed, 10 insertions(+), 1 deletion(-)
arch/x86/xen/smp_pv.c +2 -1
@@ -53,6 +53,7 @@ static DEFINE_PER_CPU(struct xen_common_irq, xen_irq_work) = { .irq = -1 }; static DEFINE_PER_CPU(struct xen_common_irq, xen_pmu_irq) = { .irq = -1 };static irqreturn_t xen_irq_work_interrupt (int irq,void *dev_id); +extern unsigned char asm_cpu_bringup_and_idle[];static void cpu_bringup (void ) {
@@ -309,7 +310,7 @@ cpu_initialize_context(unsigned int cpu, struct task_struct *idle)* pointing just below where pt_regs would be if it were a normal * kernel entry. */ - ctxt->user_regs.eip = (unsigned long )cpu_bringup_and_idle; + ctxt->user_regs.eip = (unsigned long )asm_cpu_bringup_and_idle; ctxt->flags =VGCF_IN_KERNEL ; ctxt->user_regs.eflags =0x1000 ;/* IOPL_RING1 */ ctxt->user_regs.ds = __USER_DS;
arch/x86/xen/xen-head.S +8 -0
@@ -58,6 +58,14 @@ SYM_CODE_START(startup_xen) call xen_start_kernel SYM_CODE_END(startup_xen) __FINIT + +.pushsection .text +SYM_CODE_START(asm_cpu_bringup_and_idle) + UNWIND_HINT_EMPTY + + call cpu_bringup_and_idle +SYM_CODE_END(asm_cpu_bringup_and_idle) +.popsection #endif .pushsection .text
Signature · 1 line
2.25.1
Re: [PATCH v2 2/2] x86/xen: Make the secondary CPU idle tasks reliable
arch/x86/xen/smp_pv.c static DEFINE_PER_CPU(struct xen_common_irq, xen_irq_work) = { .irq = -1 }; · line 56
Quoted 7 lines · Miroslav Benes
arch/x86/xen/smp_pv.c +1
-0@@ -53,6 +53,7 @@ static DEFINE_PER_CPU(struct xen_common_irq, xen_irq_work) = { .irq = -1 }; static DEFINE_PER_CPU(struct xen_common_irq, xen_pmu_irq) = { .irq = -1 };static irqreturn_t xen_irq_work_interrupt (int irq,void *dev_id); +extern unsigned char asm_cpu_bringup_and_idle[];
Imo this would better reflect the actual type, i.e. be a function decl. If left as an array one, I guess you may want to add const.
Jan
Miroslav Benes
maintainerRe: [PATCH v2 2/2] x86/xen: Make the secondary CPU idle tasks reliable
Quoted 11 lines · Jan Beulich
arch/x86/xen/smp_pv.c static DEFINE_PER_CPU(struct xen_common_irq, xen_irq_work) = { .irq = -1 }; · line 56
Quoted 7 lines · Miroslav Benes
arch/x86/xen/smp_pv.c +1
-0@@ -53,6 +53,7 @@ static DEFINE_PER_CPU(struct xen_common_irq, xen_irq_work) = { .irq = -1 }; static DEFINE_PER_CPU(struct xen_common_irq, xen_pmu_irq) = { .irq = -1 };static irqreturn_t xen_irq_work_interrupt (int irq,void *dev_id); +extern unsigned char asm_cpu_bringup_and_idle[];Imo this would better reflect the actual type, i.e. be a function decl. If left as an array one, I guess you may want to add const.
I sticked to what x86 has for secondary_startup_64. I can make it
void asm_cpu_bringup_and_idle(void);
Miroslav
kbuild test robot
build botRe: [PATCH v2 2/2] x86/xen: Make the secondary CPU idle tasks reliable
In reply to Miroslav BenesI love your patch! Yet something to improve:
Hi Miroslav,
I love your patch! Yet something to improve:
[auto build test ERROR on xen-tip/linux-next]
[also build test ERROR on v5.6-rc7 next-20200320]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Miroslav-Benes/x86-xen-Make-idle-tasks-reliable/20200320-000757 base: https://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git linux-next config: x86_64-randconfig-a003-20200322 (attached as .config) compiler: gcc-5 (Ubuntu 5.5.0-12ubuntu1) 5.5.0 20171010 reproduce: # save the attached .config to linux build tree make ARCH=x86_64
If you fix the issue, kindly add following tag
All errors (new ones prefixed by >>):
ld: arch/x86/kernel/head_64.o: in function `asm_cpu_bringup_and_idle':
>> arch/x86/kernel/../../x86/xen/xen-head.S:66: undefined reference to `cpu_bringup_and_idle'
vim +66 arch/x86/kernel/../../x86/xen/xen-head.S
57
58 call xen_start_kernel
59 SYM_CODE_END(startup_xen)
60 __FINIT
61
62 .pushsection .text
63 SYM_CODE_START(asm_cpu_bringup_and_idle)
64 UNWIND_HINT_EMPTY
65
> 66 call cpu_bringup_and_idle
67 SYM_CODE_END(asm_cpu_bringup_and_idle)
68 .popsection
69 #endif
70 ---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org
Miroslav Benes
maintainerRe: [PATCH v2 2/2] x86/xen: Make the secondary CPU idle tasks reliable
Quoted 37 lines · kbuild test robot
Hi Miroslav,
I love your patch! Yet something to improve:
[auto build test ERROR on xen-tip/linux-next]
[also build test ERROR on v5.6-rc7 next-20200320]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]url: https://github.com/0day-ci/linux/commits/Miroslav-Benes/x86-xen-Make-idle-tasks-reliable/20200320-000757 base: https://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git linux-next config: x86_64-randconfig-a003-20200322 (attached as .config) compiler: gcc-5 (Ubuntu 5.5.0-12ubuntu1) 5.5.0 20171010 reproduce: # save the attached .config to linux build tree make ARCH=x86_64If you fix the issue, kindly add following tag
Reported-by kbuild test robot <[email protected]>All errors (new ones prefixed by >>):
ld: arch/x86/kernel/head_64.o: in function `asm_cpu_bringup_and_idle':Quoted 1 line
Quoted 1 line
arch/x86/kernel/../../x86/xen/xen-head.S:66: undefined reference to `cpu_bringup_and_idle'vim +66
arch/x86/kernel/../../x86/xen/xen-head.S57 58 call xen_start_kernel 59 SYM_CODE_END(startup_xen) 60 __FINIT 61 62 .pushsection .text 63 SYM_CODE_START(asm_cpu_bringup_and_idle) 64 UNWIND_HINT_EMPTY 65
Quoted 5 lines
Quoted 1 line
66 call
cpu_bringup_and_idle67 SYM_CODE_END(asm_cpu_bringup_and_idle) 68 .popsection 69 #endif 70
Right, cpu_bringup_and_idle() is there only for CONFIG_XEN_PV_SMP.
---
arch/x86/xen/xen-head.S +2 -0
@@ -59,6 +59,7 @@ SYM_CODE_START(startup_xen) SYM_CODE_END(startup_xen) __FINIT +#ifdef CONFIG_XEN_PV_SMP .pushsection .text SYM_CODE_START(asm_cpu_bringup_and_idle) UNWIND_HINT_EMPTY
@@ -67,6 +68,7 @@ SYM_CODE_START(asm_cpu_bringup_and_idle) SYM_CODE_END(asm_cpu_bringup_and_idle) .popsection #endif +#endif .pushsection .text .balign PAGE_SIZE
---
on fixes that.
Thanks
Miroslav