lib/s390x: fix SMP setup bug
PATCH 1/1 lib/s390x: fix SMP setup bug
The lowcore pointer pointing to the current CPU (THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.
The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.
Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.
--- lib/s390x/io.c | 9 +++++++++ lib/s390x/smp.c | 1 + 2 files changed, 10 insertions(+)
lib/s390x/io.c +9 -0
@@ -33,6 +33,15 @@ void puts(const char *s)void setup (void ) { +struct cpu this_cpu_tmp = {0 }; + +/* +* Set a temporary empty struct cpu for the boot CPU, needed for +* correct interrupt handling in the setup process. +* smp_setup will allocate and set the permanent one. +*/ +THIS_CPU = &this_cpu_tmp; +setup_args_progname (ipl_args);setup_facilities ();sclp_read_info ();
lib/s390x/smp.c +1 -0
@@ -353,6 +353,7 @@ void smp_setup(void) cpus[0 ].stack = stackptr; cpus[0 ].lowcore = (void *)0 ; cpus[0 ].active = true; +THIS_CPU = &cpus[0 ]; } }spin_unlock (&lock);
Signature · 1 line
2.37.2
Quoting Claudio Imbrenda (2022-08-18 17:21:14)
Quoted 11 lines
The lowcore pointer pointing to the current CPU (
THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.
Quoted 6 lines
Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.
Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")Signed-off-by Claudio Imbrenda <[email protected]>
Quoted 11 lines · Claudio Imbrenda
The lowcore pointer pointing to the current CPU (
THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.
Quoted 6 lines
Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.
Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")Signed-off-by Claudio Imbrenda <[email protected]>
I've considered letting the IPL cpu have a static struct cpu and setting it up in cstart64.S. But that would mean that we would need extra handling when using smp functions and that'll look even worse.
Quoted 10 lines
--- lib/s390x/io.c | 9 +++++++++ lib/s390x/smp.c | 1 + 2 files changed, 10 insertions(+)lib/s390x/io.c +0
-0
lib/s390x/io.c void puts(const char *s) · line 36
Quoted 4 lines
lib/s390x/io.c +1
-0@@ -33,6 +33,15 @@ void puts(const char *s)void setup (void ) { +struct cpu this_cpu_tmp = {0 };
We can setup these struct members here and memcpy in smp_setup() .addr = stap();
.stack = stackptr;
.lowcore = (void *)0;
.active = true;
Quoted 16 lines
lib/s390x/io.c +8
-0+ +/* +* Set a temporary empty struct cpu for the boot CPU, needed for +* correct interrupt handling in the setup process. +* smp_setup will allocate and set the permanent one. +*/ +THIS_CPU = &this_cpu_tmp; +setup_args_progname (ipl_args);setup_facilities ();sclp_read_info ();lib/s390x/smp.c +0
-0
lib/s390x/smp.c void smp_setup(void) · line 356
Quoted 4 lines
lib/s390x/smp.c +1
-0@@ -353,6 +353,7 @@ void smp_setup(void) cpus[0 ].stack = stackptr; cpus[0 ].lowcore = (void *)0 ; cpus[0 ].active = true; +THIS_CPU = &cpus[0 ];
/* Override temporary struct cpu address with permanent one */
Quoted 3 lines
lib/s390x/smp.c +0
-0} }spin_unlock (&lock);
Quoted 36 lines · On Fri, 19 Aug 2022 10:52:40 +0200 Janosch Frank <[email protected]> wrote:
Quoted 11 lines · Claudio Imbrenda
The lowcore pointer pointing to the current CPU (
THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.
Quoted 6 lines
Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.
Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")Signed-off-by Claudio Imbrenda <[email protected]>I've considered letting the IPL cpu have a static struct cpu and setting it up in
cstart64.S. But that would mean that we would need extra handling when using smp functions and that'll look even worse.Reported-by Janosch Frank <[email protected]>Quoted 11 lines
--- lib/s390x/io.c | 9 +++++++++ lib/s390x/smp.c | 1 + 2 files changed, 10 insertions(+)lib/s390x/io.c +0
-0@@ -33,6 +33,15 @@ void puts(const char *s)
Quoted 7 lines
lib/s390x/io.c void puts(const char *s) · line 36
Quoted 3 lines
lib/s390x/io.c +1
-0void setup (void ) { +struct cpu this_cpu_tmp = {0 };We can setup these struct members here and memcpy in
smp_setup().addr =stap();
.stack = stackptr;
stackptr is accessible in io.c (I would need to add an extern declaration)
Quoted 2 lines
.lowcore = (void *)0;
.active = true;
this temporary struct is then not accessible from smp_setup, so it can't be memcpy-ed.
if you really want something meaningful in the temporary struct, it has to be initialized in smp.c and called in io.c (something like smp_boot_cpu_tmp_setup(&this_cpu_tmp) ), but then still no memcpy.
in the end the struct cpu is needed only to allow interrupts to happen without crashes, I don't think we strictly need initialization
Quoted 16 lines
Quoted 16 lines
lib/s390x/io.c +8
-0+ +/* +* Set a temporary empty struct cpu for the boot CPU, needed for +* correct interrupt handling in the setup process. +* smp_setup will allocate and set the permanent one. +*/ +THIS_CPU = &this_cpu_tmp; +setup_args_progname (ipl_args);setup_facilities ();sclp_read_info ();lib/s390x/smp.c +0
-1--- a/lib/s390x/smp.c
Quoted 8 lines
lib/s390x/smp.c void smp_setup(void) · line 356
Quoted 6 lines
+++
b/lib/s390x/smp.clib/s390x/smp.c +1
-0@@ -353,6 +353,7 @@ void smp_setup(void) cpus[0 ].stack = stackptr; cpus[0 ].lowcore = (void *)0 ; cpus[0 ].active = true; +THIS_CPU = &cpus[0 ];/* Override temporary struct cpu address with permanent one */
will be done
Quoted 5 lines
Quoted 4 lines
lib/s390x/smp.c +0
-0} }spin_unlock (&lock);
Quoted 56 lines · Claudio Imbrenda
Quoted 36 lines · On Fri, 19 Aug 2022 10:52:40 +0200 Janosch Frank <[email protected]> wrote:
Quoted 11 lines · Claudio Imbrenda
The lowcore pointer pointing to the current CPU (
THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.
Quoted 6 lines
Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.
Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")Signed-off-by Claudio Imbrenda <[email protected]>I've considered letting the IPL cpu have a static struct cpu and setting it up in
cstart64.S. But that would mean that we would need extra handling when using smp functions and that'll look even worse.Reported-by Janosch Frank <[email protected]>Quoted 11 lines
--- lib/s390x/io.c | 9 +++++++++ lib/s390x/smp.c | 1 + 2 files changed, 10 insertions(+)lib/s390x/io.c +0
-0@@ -33,6 +33,15 @@ void puts(const char *s)Quoted 7 lines
lib/s390x/io.c void puts(const char *s) · line 36
Quoted 3 lines
lib/s390x/io.c +1
-0void setup (void ) { +struct cpu this_cpu_tmp = {0 };We can setup these struct members here and memcpy in
smp_setup().addr =stap();
.stack = stackptr;stackptr is accessible in
io.c(I would need to add an extern declaration)Quoted 2 lines
.lowcore = (void *)0;
.active = true;this temporary struct is then not accessible from
smp_setup, so it can't be memcpy-ed.
Quoted 6 lines
if you really want something meaningful in the temporary struct, it has to be initialized in
smp.cand called inio.c(something likesmp_boot_cpu_tmp_setup(&this_cpu_tmp)), but then still no memcpy.in the end the struct cpu is needed only to allow interrupts to happen without crashes, I don't think we strictly need initialization
Ugh, this feels like a quick fix.
But alright, I've just tried setting it up from cstart64.S and it's way more ugly code so let's stick with this for now.
Quoted 34 lines
Quoted 17 lines
Quoted 17 lines
lib/s390x/io.c +8
-0+ +/* +* Set a temporary empty struct cpu for the boot CPU, needed for +* correct interrupt handling in the setup process. +* smp_setup will allocate and set the permanent one. +*/ +THIS_CPU = &this_cpu_tmp; +setup_args_progname (ipl_args);setup_facilities ();sclp_read_info ();lib/s390x/smp.c +0
-1--- a/lib/s390x/smp.cQuoted 8 lines
lib/s390x/smp.c void smp_setup(void) · line 356
Quoted 6 lines
+++
b/lib/s390x/smp.clib/s390x/smp.c +1
-0@@ -353,6 +353,7 @@ void smp_setup(void) cpus[0 ].stack = stackptr; cpus[0 ].lowcore = (void *)0 ; cpus[0 ].active = true; +THIS_CPU = &cpus[0 ];/* Override temporary struct cpu address with permanent one */
will be done
Quoted 5 lines
Quoted 4 lines
lib/s390x/smp.c +0
-0} }spin_unlock (&lock);
Janosch Frank
Reviewed-byQuoted 29 lines · Janosch Frank
Quoted 28 lines · Claudio Imbrenda
Quoted 25 lines · On Fri, 19 Aug 2022 10:52:40 +0200 Janosch Frank <[email protected]> wrote:
Quoted 11 lines · Claudio Imbrenda
The lowcore pointer pointing to the current CPU (
THIS_CPU) was not initialized for the boot CPU. The pointer is needed for correct interrupt handling, which is needed in the setup process before the struct cpu array is allocated.The bug went unnoticed because some environments (like qemu/KVM) clear all memory and don't write anything in the lowcore area before starting the payload. The pointer thus pointed to 0, an area of memory also not used. Other environments will write to memory before starting the payload, causing the unit tests to crash at the first interrupt.
Quoted 6 lines
Fix by assigning a temporary struct cpu before the rest of the setup process, and assigning the pointer to the correct allocated struct during smp initialization.
Fixes 4e5dd758 ("lib: s390x: better smp interrupt checks")Signed-off-by Claudio Imbrenda <[email protected]>I've considered letting the IPL cpu have a static struct cpu and setting it up in
cstart64.S. But that would mean that we would need extra handling when using smp functions and that'll look even worse.Reported-by Janosch Frank <[email protected]>
[...]
Quoted 8 lines
Quoted 8 lines
this temporary struct is then not accessible from
smp_setup, so it can't be memcpy-ed.if you really want something meaningful in the temporary struct, it has to be initialized in
smp.cand called inio.c(something likesmp_boot_cpu_tmp_setup(&this_cpu_tmp)), but then still no memcpy.
Quoted 7 lines
Quoted 2 lines
in the end the struct cpu is needed only to allow interrupts to happen without crashes, I don't think we strictly need initialization
Ugh, this feels like a quick fix.
But alright, I've just tried setting it up fromcstart64.Sand it's way more ugly code so let's stick with this for now.
Anyway:
Thanks, picked