Skip to contentSkip to conversation

pc-bios/s390 fixes for reboot-to-vfio-ccw

PATCH In review Eric Farman 3 participants 7 messages Nov 20, 2020 – Nov 23, 2020

lore
Share & tools
lore
#1 lore raw

PATCH v2 1/2 pc-bios: s390x: Ensure Read IPL memory is clean

In reply to Eric Farman

If, for example, we boot off a virtio device and chreipl to a vfio-ccw device, the space at lowcore will be non-zero. We build a Read IPL CCW at address zero, but it will have leftover PSW data that will conflict with the Format-0 CCW being generated:

0x0: 00080000 80010000
------ Ccw0.cda
-- Ccw0.chainData
-- Reserved bits

The data address will be overwritten with the correct value (0x0), but the apparent data chain bit will cause subsequent memory to be used as the target of the data store, which may not be where we expect (0x0).

Clear out this space when we boot from DASD, so that we know it exists exactly as we expect.

---
 pc-bios/s390-ccw/dasd-ipl.c | 3 +++
 1 file changed, 3 insertions(+)
pc-bios/s390-ccw/dasd-ipl.c +3 -0
index 0fc879bb8e..71cbae2f16 100644
@@ -100,6 +100,9 @@ static void make_readipl(void)
 {
     Ccw0 *ccwIplRead = (Ccw0 *)0x00;
 
+    /* Clear out any existing data */
+    memset(ccwIplRead, 0, sizeof(Ccw0));
+
     /* Create Read IPL ccw at address 0 */
     ccwIplRead->cmd_code = CCW_CMD_READ_IPL;
     ccwIplRead->cda = 0x00; /* Read into address 0x00 in main memory */
Signature · 1 line
2.17.1
#2 lore raw

PATCH v2 2/2 pc-bios: s390x: Clear out leftover S390EP string

In reply to Eric Farman

A Linux binary will have the string "S390EP" at address 0x10008, which is important in getting the guest up off the ground. In the case of a reboot (specifically chreipl going to a new device), we should defer to the PSW at address zero for the new config, which will re-write "S390EP" from the new image.

Let's clear it out at this point so that a reipl to, say, a DASD passthrough device drives the IPL path from scratch without disrupting disrupting the order of operations for other boots.

Rather than hardcoding the address of this magic (again), let's define it somewhere so that the two users are visibly related.

---
 pc-bios/s390-ccw/jump2ipl.c  | 2 +-
 pc-bios/s390-ccw/main.c      | 6 ++++++
 pc-bios/s390-ccw/s390-arch.h | 3 +++
 3 files changed, 10 insertions(+), 1 deletion(-)
pc-bios/s390-ccw/jump2ipl.c +1 -1
index fbae45b03c..b9c70d64a5 100644
@@ -78,7 +78,7 @@ void jump_to_low_kernel(void)
      * kernel start address (when jumping to the PSW-at-zero address instead,
      * the kernel startup code fails when we booted from a network device).
      */
-    if (!memcmp((char *)0x10008, "S390EP", 6)) {
+    if (!memcmp((char *)S390EP, "S390EP", 6)) {
         jump_to_IPL_code(KERN_IMAGE_START);
     }
 
pc-bios/s390-ccw/main.c +6 -0
index fc4bfaa455..5d2b7ba94d 100644
@@ -178,6 +178,12 @@ static void boot_setup(void)
     memcpy(lpmsg + 10, loadparm_str, 8);
     sclp_print(lpmsg);
 
+    /*
+     * Clear out any potential S390EP magic (see jump_to_low_kernel()),
+     * so we don't taint our decision-making process during a reboot.
+     */
+    memset((char *)S390EP, 0, 6);
+
     have_iplb = store_iplb(&iplb);
 }
 
pc-bios/s390-ccw/s390-arch.h +3 -0
index 6da44d4436..a741488aaa 100644
@@ -95,6 +95,9 @@ typedef struct LowCore {
 
 extern LowCore *lowcore;
 
+/* Location of "S390EP" in a Linux binary (see arch/s390/boot/head.S) */
+#define S390EP 0x10008
+
 static inline void set_prefix(uint32_t address)
 {
     asm volatile("spx %0" : : "m" (address) : "memory");
Signature · 1 line
2.17.1
#0 lore raw

v2 cover pc-bios/s390 fixes for reboot-to-vfio-ccw

Thomas, Conny,

Here is the updated series for the re-IPL fixes mentioned yesterday [1]

I've updated the reviews/acks to Patch 1 (Thank you Conny, Janosch!), and implemented Thomas' suggestion to clear out the memory where S390EP exists early in our processing. This both preserves the delicate balance we have today, and fixes the scenario I've been wrestling with.

For this second patch, I opted to #define the address x10008 and use it in both the new memset and existing memcmp, rather than having the address used directly in two places. I put this definition in s390-arch.h as it's shared by both files, even though it's not strictly an "s390 architecture" thing. Considering its importance, it seems like a reasonable fit.

Tested with my permutations of virtio/vfio/chreipl/install things, but surely missing some others that I don't have readily configured.

[1] https://lore.kernel.org/qemu-devel/[email protected]/

Eric Farman (2):
pc-bios: s390x: Ensure Read IPL memory is clean
pc-bios: s390x: Clear out leftover S390EP string

 pc-bios/s390-ccw/dasd-ipl.c  | 3 +++
 pc-bios/s390-ccw/jump2ipl.c  | 2 +-
 pc-bios/s390-ccw/main.c      | 6 ++++++
 pc-bios/s390-ccw/s390-arch.h | 3 +++
 4 files changed, 13 insertions(+), 1 deletion(-)
Signature · 1 line
2.17.1
#3 lore raw

Re: [PATCH v2 2/2] pc-bios: s390x: Clear out leftover S390EP string

In reply to Eric Farman
Quoted 6 lines · Eric Farman

A Linux binary will have the string "S390EP" at address 0x10008, which is important in getting the guest up off the ground. In the case of a reboot (specifically chreipl going to a new device), we should defer to the PSW at address zero for the new config, which will re-write "S390EP" from the new image.

Quoted 6 lines

Let's clear it out at this point so that a reipl to, say, a DASD passthrough device drives the IPL path from scratch without disrupting disrupting the order of operations for other boots.

Rather than hardcoding the address of this magic (again), let's define it somewhere so that the two users are visibly related.

Hmmm, this might have side effects, e.g. if you do something like a kdump or kexec to a non-Linux binary that happens to have code at 0x10008, no?

As far as I can tell, the problem should only happen for a ccw type IPL so why not

[...]

pc-bios/s390-ccw/main.c static void boot_setup(void) · line 185

Quoted 11 lines
pc-bios/s390-ccw/main.c +5 -0
@@ -178,6 +178,12 @@ static void boot_setup(void)
     memcpy(lpmsg + 10, loadparm_str, 8);
     sclp_print(lpmsg);
 
+    /*
+     * Clear out any potential S390EP magic (see jump_to_low_kernel()),
+     * so we don't taint our decision-making process during a reboot.
+     */
+    memset((char *)S390EP, 0, 6);

move this into find_subch
in here:
------------- snip ---------------

            case CU_TYPE_DASD_3990:
            case CU_TYPE_DASD_2107:
                return true;

------------- snip ---------------

#4 lore raw

Re: [PATCH v2 2/2] pc-bios: s390x: Clear out leftover S390EP string

Quoted 11 lines · Christian Borntraeger
Quoted 10 lines · Eric Farman

A Linux binary will have the string "S390EP" at address 0x10008, which is important in getting the guest up off the ground. In the case of a reboot (specifically chreipl going to a new device), we should defer to the PSW at address zero for the new config, which will re-write "S390EP" from the new image.

Let's clear it out at this point so that a reipl to, say, a DASD passthrough device drives the IPL path from scratch without disrupting disrupting the order of operations for other boots.

Quoted 6 lines
Quoted 2 lines

Rather than hardcoding the address of this magic (again), let's define it somewhere so that the two users are visibly related.

Hmmm, this might have side effects, e.g. if you do something like a kdump or kexec to a non-Linux binary that happens to have code at 0x10008, no?

Do these scenarios really go through the s390-ccw bios again, or do they rather bypass the bios and jump directly into the new kernel?

Quoted 2 lines

As far as I can tell, the problem should only happen for a ccw type IPL so why not

Not sure whether it really can only happen in these cases... for example, would it also be possible to reboot from a Linux kernel into a kvm-unit-test? ... these also do not have the S390EP magic, IIRC.

Quoted 17 lines

[...]

pc-bios/s390-ccw/main.c static void boot_setup(void) · line 185

Quoted 11 lines
pc-bios/s390-ccw/main.c +5 -0
@@ -178,6 +178,12 @@ static void boot_setup(void)
     memcpy(lpmsg + 10, loadparm_str, 8);
     sclp_print(lpmsg);
 
+    /*
+     * Clear out any potential S390EP magic (see jump_to_low_kernel()),
+     * so we don't taint our decision-making process during a reboot.
+     */
+    memset((char *)S390EP, 0, 6);

move this into find_subch
in here:
------------- snip ---------------

Quoted 5 lines
            case CU_TYPE_DASD_3990:
            case CU_TYPE_DASD_2107:
                return true;

------------- snip ---------------

That would be is_dev_possibly_bootable() now? ... not sure whether this is the best location... maybe put it better at the beginning of dasd_ipl() instead?

Thomas

#5 lore raw

Re: [PATCH v2 2/2] pc-bios: s390x: Clear out leftover S390EP string

Quoted 16 lines · Thomas Huth
Quoted 15 lines · Christian Borntraeger
Quoted 12 lines · Eric Farman

A Linux binary will have the string "S390EP" at address 0x10008, which is important in getting the guest up off the ground. In the case of a reboot (specifically chreipl going to a new device), we should defer to the PSW at address zero for the new config, which will re-write "S390EP" from the new image.

Let's clear it out at this point so that a reipl to, say, a DASD passthrough device drives the IPL path from scratch without disrupting disrupting the order of operations for other boots.

Rather than hardcoding the address of this magic (again), let's define it somewhere so that the two users are visibly related.

Quoted 5 lines
Quoted 2 lines

Hmmm, this might have side effects, e.g. if you do something like a kdump or kexec to a non-Linux binary that happens to have code at 0x10008, no?

Do these scenarios really go through the s390-ccw bios again, or do they rather bypass the bios and jump directly into the new kernel?

Right they jump directly into the new kernel. So this patch could actually be "good enough" for 5.2 as is?

#6 lore raw

Re: [PATCH v2 2/2] pc-bios: s390x: Clear out leftover S390EP string

Quoted 19 lines · Christian Borntraeger
Quoted 16 lines · Thomas Huth
Quoted 15 lines · Christian Borntraeger
Quoted 12 lines · Eric Farman

A Linux binary will have the string "S390EP" at address 0x10008, which is important in getting the guest up off the ground. In the case of a reboot (specifically chreipl going to a new device), we should defer to the PSW at address zero for the new config, which will re-write "S390EP" from the new image.

Let's clear it out at this point so that a reipl to, say, a DASD passthrough device drives the IPL path from scratch without disrupting disrupting the order of operations for other boots.

Rather than hardcoding the address of this magic (again), let's define it somewhere so that the two users are visibly related.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Hmmm, this might have side effects, e.g. if you do something like a kdump or kexec to a non-Linux binary that happens to have code at 0x10008, no?

Do these scenarios really go through the s390-ccw bios again, or do they rather bypass the bios and jump directly into the new kernel?

Right they jump directly into the new kernel. So this patch could actually be "good enough" for 5.2 as is?

I think it should be fine, yes. I'll give it a try with my usual s390-ccw bios tests, and if there are no regressions (and no other objections on the mailing list here), I'll prepare a pull request.

Thomas