Skip to contentSkip to conversation

xen/arm: set CPSR Z bit when creating aarch32 guests

PATCH In review Stefano Stabellini 5 participants 7 messages Mar 22, 2022 – Mar 24, 2022

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 xen/arm: set CPSR Z bit when creating aarch32 guests

From: Stefano Stabellini <[email protected]>

The first 32 bytes of zImage are NOPs. When CONFIG_EFI is enabled in the kernel, certain versions of Linux will use an UNPREDICATABLE NOP encoding, sometimes resulting in an unbootable kernel. Whether the resulting kernel is bootable or not depends on the processor. See commit a92882a4d270 in the Linux kernel for all the details.

All kernel releases starting from Linux 4.9 without commit a92882a4d270 are affected.

Fortunately there is a simple workaround: setting the "Z" bit in CPSR make it so those invalid NOP instructions are never executed. That is because the instruction is conditional (not equal). So, on QEMU at least, the instruction will end up to be ignored and not generate an exception. Setting the "Z" bit makes those kernel versions bootable again and it is harmless in the other cases.

---
Changes in v2:
- improve commit message
- add in-code comment
- move PSR_Z to the beginning

---
 xen/include/public/arch-arm.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
xen/include/public/arch-arm.h +7 -1
index 94b31511dd..81cee95f14 100644
@@ -361,6 +361,7 @@ typedef uint64_t xen_callback_t;
 #define PSR_DBG_MASK    (1<<9)        /* arm64: Debug Exception mask */
 #define PSR_IT_MASK     (0x0600fc00)  /* Thumb If-Then Mask */
 #define PSR_JAZELLE     (1<<24)       /* Jazelle Mode */
+#define PSR_Z           (1<<30)       /* Zero condition flag */
 
 /* 32 bit modes */
 #define PSR_MODE_USR 0x10
@@ -383,7 +384,12 @@ typedef uint64_t xen_callback_t;
 #define PSR_MODE_EL1t 0x04
 #define PSR_MODE_EL0t 0x00
 
-#define PSR_GUEST32_INIT  (PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)
+/*
+ * We set PSR_Z to be able to boot Linux kernel versions with an invalid
+ * encoding of the first 8 NOP instructions. See commit a92882a4d270 in
+ * Linux.
+ */
+#define PSR_GUEST32_INIT  (PSR_Z|PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)
 #define PSR_GUEST64_INIT (PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_EL1h)
 
 #define SCTLR_GUEST_INIT    xen_mk_ullong(0x00c50078)
Signature · 1 line
2.25.1
#1 lore raw
Quoted 7 lines · Stefano Stabellini

From: Stefano Stabellini <[email protected]>

The first 32 bytes of zImage are NOPs. When CONFIG_EFI is enabled in the kernel, certain versions of Linux will use an UNPREDICATABLE NOP encoding, sometimes resulting in an unbootable kernel. Whether the resulting kernel is bootable or not depends on the processor. See commit a92882a4d270 in the Linux kernel for all the details.

Is this a problem only under Xen or also on bare hardware? In the latter case I'd be even more inclined to require this issue to be dealt with in the kernels, rather than working around it by an ABI change in Xen.

Quoted 9 lines

All kernel releases starting from Linux 4.9 without commit a92882a4d270 are affected.

Fortunately there is a simple workaround: setting the "Z" bit in CPSR make it so those invalid NOP instructions are never executed. That is because the instruction is conditional (not equal). So, on QEMU at least, the instruction will end up to be ignored and not generate an exception. Setting the "Z" bit makes those kernel versions bootable again and it is harmless in the other cases.

I'm afraid such an ABI change being harmless needs to be not just claimed, but proven. There could certainly be reasons this is safe, e.g. the same path being taken on bare hardware, and the state of the bit not being specified there. Yet even in presence of such a specification it cannot be excluded that non-standard (something XTF-like, for example) uses might have grown a dependency on the Xen ABI specification.

Jan

#2 lore raw
In reply to Stefano Stabellini

Hi Stefano,

Quoted 12 lines · Stefano Stabellini

From: Stefano Stabellini <[email protected]>

The first 32 bytes of zImage are NOPs. When CONFIG_EFI is enabled in the kernel, certain versions of Linux will use an UNPREDICATABLE NOP encoding, sometimes resulting in an unbootable kernel. Whether the resulting kernel is bootable or not depends on the processor. See commit a92882a4d270 in the Linux kernel for all the details.

All kernel releases starting from Linux 4.9 without commit a92882a4d270 are affected.

Can you confirm if those kernels are also affected when started natively ?

Quoted 7 lines

Fortunately there is a simple workaround: setting the "Z" bit in CPSR make it so those invalid NOP instructions are never executed. That is because the instruction is conditional (not equal). So, on QEMU at least, the instruction will end up to be ignored and not generate an exception. Setting the "Z" bit makes those kernel versions bootable again and it is harmless in the other cases.

I agree with Jan here. This will never be set or should not be expected to be set by anyone when started.
It feels to me that we are introducing an ack for a temporary issue in Linux which will makes us derive from the behaviour that could be expected on native hardware.

Could you give more details on how blocking this is ?
Is the kernel update with the fix available on any of the affected distributions ?

Depending on the answers I think we could for example have a config around this to flag it as workaround for a specific guest issue so that this is only activated when needed.

Cheers
Bertrand

Quoted 40 lines

---
Changes in v2:
- improve commit message
- add in-code comment
- move PSR_Z to the beginning
---
xen/include/public/arch-arm.h | 8 +++++++-

1 file changed, 7 insertions(+), 1 deletion(-)
xen/include/public/arch-arm.h +7 -1
index 94b31511dd..81cee95f14 100644
@@ -361,6 +361,7 @@ typedef uint64_t xen_callback_t;
 #define PSR_DBG_MASK    (1<<9)        /* arm64: Debug Exception mask */
 #define PSR_IT_MASK     (0x0600fc00)  /* Thumb If-Then Mask */
 #define PSR_JAZELLE     (1<<24)       /* Jazelle Mode */
+#define PSR_Z           (1<<30)       /* Zero condition flag */
 
 /* 32 bit modes */
 #define PSR_MODE_USR 0x10
@@ -383,7 +384,12 @@ typedef uint64_t xen_callback_t;
 #define PSR_MODE_EL1t 0x04
 #define PSR_MODE_EL0t 0x00
 
-#define PSR_GUEST32_INIT  (PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)
+/*
+ * We set PSR_Z to be able to boot Linux kernel versions with an invalid
+ * encoding of the first 8 NOP instructions. See commit a92882a4d270 in
+ * Linux.
+ */
+#define PSR_GUEST32_INIT  (PSR_Z|PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)
 #define PSR_GUEST64_INIT (PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_EL1h)
 
 #define SCTLR_GUEST_INIT    xen_mk_ullong(0x00c50078)
Signature · 1 line
2.25.1
#3 lore raw
Quoted 10 lines · Bertrand Marquis
Quoted 10 lines · Stefano Stabellini

From: Stefano Stabellini <[email protected]>

The first 32 bytes of zImage are NOPs. When CONFIG_EFI is enabled in the kernel, certain versions of Linux will use an UNPREDICATABLE NOP encoding, sometimes resulting in an unbootable kernel. Whether the resulting kernel is bootable or not depends on the processor. See commit a92882a4d270 in the Linux kernel for all the details.

Quoted 4 lines
Quoted 2 lines

All kernel releases starting from Linux 4.9 without commit a92882a4d270 are affected.

Can you confirm if those kernels are also affected when started natively ?

Theoretically yes, but in practice only booting on Xen is affected because:

- the issue cannot happen when booting from u-boot because u-boot sets the "Z" bit
- the issue cannot happen when booting with QEMU -kernel because it also sets "Z"
- older bootloaders on native skip the first 32 bytes of the start address, which also masks this problem

Thus, in practice, I have no idea how one could reproduce the problem on native.

This info is in the commit message a92882a4d270 on Linux and in-code comments in the kernel.

Quoted 7 lines
Quoted 6 lines

Fortunately there is a simple workaround: setting the "Z" bit in CPSR make it so those invalid NOP instructions are never executed. That is because the instruction is conditional (not equal). So, on QEMU at least, the instruction will end up to be ignored and not generate an exception. Setting the "Z" bit makes those kernel versions bootable again and it is harmless in the other cases.

Quoted 7 lines

I agree with Jan here. This will never be set or should not be expected to be set by anyone when started.
It feels to me that we are introducing an ack for a temporary issue in Linux which will makes us derive from the behaviour that could be expected on native hardware.

Could you give more details on how blocking this is ?

Without this change, none of the Debian arm32 kernels boot on Xen after Jessie (on QEMU).

Quoted 1 line

Is the kernel update with the fix available on any of the affected distributions ?

None that I could find. I tried Debian Buster, Debian Bullseye, Debian testing and the latest Alpine Linux. Happy to try more if you give me a download link or two.

Quoted 3 lines

Depending on the answers I think we could for example have a config around this to flag it as workaround for a specific guest issue so that this is only activated when needed.

Also note that this alternative workaround also solves the problem, however it has other drawbacks as Julien described:
[1] https://marc.info/?l=xen-devel&m=164774063802402

My take on this is the following. PSR_GUEST32_INIT is not part of the ABI so this cannot be considered an ABI change.

But in any case, given that without this change (or another change [1]) most of the kernels out there don't work, is there a point in discussing ABI breakages? Basically nothing works right now :-D

I think it makes sense to think whether this change could cause a kernel that used to boot, not to boot anymore. However, I don't think is possible because:

- we only support zImage on arm32 and "Z" works well with it
- both u-boot and qemu -kernel set "Z" so we would already now if something broke

Quoted 40 lines
Quoted 39 lines

---
Changes in v2:
- improve commit message
- add in-code comment
- move PSR_Z to the beginning
---
xen/include/public/arch-arm.h | 8 +++++++-

1 file changed, 7 insertions(+), 1 deletion(-)
xen/include/public/arch-arm.h +7 -1
index 94b31511dd..81cee95f14 100644
@@ -361,6 +361,7 @@ typedef uint64_t xen_callback_t;
 #define PSR_DBG_MASK    (1<<9)        /* arm64: Debug Exception mask */
 #define PSR_IT_MASK     (0x0600fc00)  /* Thumb If-Then Mask */
 #define PSR_JAZELLE     (1<<24)       /* Jazelle Mode */
+#define PSR_Z           (1<<30)       /* Zero condition flag */
 
 /* 32 bit modes */
 #define PSR_MODE_USR 0x10
@@ -383,7 +384,12 @@ typedef uint64_t xen_callback_t;
 #define PSR_MODE_EL1t 0x04
 #define PSR_MODE_EL0t 0x00
 
-#define PSR_GUEST32_INIT  (PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)
+/*
+ * We set PSR_Z to be able to boot Linux kernel versions with an invalid
+ * encoding of the first 8 NOP instructions. See commit a92882a4d270 in
+ * Linux.
+ */
+#define PSR_GUEST32_INIT  (PSR_Z|PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)
 #define PSR_GUEST64_INIT (PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_EL1h)
 
 #define SCTLR_GUEST_INIT    xen_mk_ullong(0x00c50078)
Signature · 1 line
2.25.1
#4 lore raw

Hi Stefano,

Thanks a lot for the detailed answers.

Quoted 28 lines · Stefano Stabellini
Quoted 10 lines · Bertrand Marquis
Quoted 10 lines · Stefano Stabellini

From: Stefano Stabellini <[email protected]>

The first 32 bytes of zImage are NOPs. When CONFIG_EFI is enabled in the kernel, certain versions of Linux will use an UNPREDICATABLE NOP encoding, sometimes resulting in an unbootable kernel. Whether the resulting kernel is bootable or not depends on the processor. See commit a92882a4d270 in the Linux kernel for all the details.

Quoted 4 lines
Quoted 2 lines

All kernel releases starting from Linux 4.9 without commit a92882a4d270 are affected.

Can you confirm if those kernels are also affected when started natively ?

Theoretically yes, but in practice only booting on Xen is affected because:

- the issue cannot happen when booting from u-boot because u-boot sets the "Z" bit
- the issue cannot happen when booting with QEMU -kernel because it also sets "Z"
- older bootloaders on native skip the first 32 bytes of the start address, which also masks this problem

Quoted 5 lines

Thus, in practice, I have no idea how one could reproduce the problem on native.

This info is in the commit message a92882a4d270 on Linux and in-code comments in the kernel.

If uboot is setting we can consider that we have a behaviour equivalent to a native boot.
Could you add something in the comment in your patch to state that the Z flag is also set by uboot ?
This will help in the future to remember why it is ok to have that as the current comment could make one think that this is something only done by Xen.

Quoted 15 lines
Quoted 15 lines
Quoted 8 lines

Fortunately there is a simple workaround: setting the "Z" bit in CPSR make it so those invalid NOP instructions are never executed. That is because the instruction is conditional (not equal). So, on QEMU at least, the instruction will end up to be ignored and not generate an exception. Setting the "Z" bit makes those kernel versions bootable again and it is harmless in the other cases.

I agree with Jan here. This will never be set or should not be expected to be set by anyone when started.
It feels to me that we are introducing an ack for a temporary issue in Linux which will makes us derive from the behaviour that could be expected on native hardware.

Quoted 4 lines
Quoted 1 line

Could you give more details on how blocking this is ?

Without this change, none of the Debian arm32 kernels boot on Xen after Jessie (on QEMU).

Ok

Quoted 7 lines
Quoted 3 lines

Is the kernel update with the fix available on any of the affected distributions ?

None that I could find. I tried Debian Buster, Debian Bullseye, Debian testing and the latest Alpine Linux. Happy to try more if you give me a download link or two.

I think the list is long enough to justify the change.

Quoted 9 lines
Quoted 5 lines

Depending on the answers I think we could for example have a config around this to flag it as workaround for a specific guest issue so that this is only activated when needed.

Also note that this alternative workaround also solves the problem, however it has other drawbacks as Julien described:
[1] https://marc.info/?l=xen-devel&m=164774063802402

Definitely setting the Z bit is better I think.

Quoted 9 lines

My take on this is the following. PSR_GUEST32_INIT is not part of the ABI so this cannot be considered an ABI change.

But in any case, given that without this change (or another change [1]) most of the kernels out there don't work, is there a point in discussing ABI breakages? Basically nothing works right now :-D

Quoted 8 lines

I think it makes sense to think whether this change could cause a kernel that used to boot, not to boot anymore. However, I don't think is possible because:

- we only support zImage on arm32 and "Z" works well with it
- both u-boot and qemu -kernel set "Z" so we would already now if something broke

Agree so please add that both in the comment and in the commit message.

Cheers
Bertrand

Quoted 42 lines
Quoted 42 lines
Quoted 41 lines

---
Changes in v2:
- improve commit message
- add in-code comment
- move PSR_Z to the beginning
---
xen/include/public/arch-arm.h | 8 +++++++-

1 file changed, 7 insertions(+), 1 deletion(-)
xen/include/public/arch-arm.h +7 -1
index 94b31511dd..81cee95f14 100644
@@ -361,6 +361,7 @@ typedef uint64_t xen_callback_t;
 #define PSR_DBG_MASK    (1<<9)        /* arm64: Debug Exception mask */
 #define PSR_IT_MASK     (0x0600fc00)  /* Thumb If-Then Mask */
 #define PSR_JAZELLE     (1<<24)       /* Jazelle Mode */
+#define PSR_Z           (1<<30)       /* Zero condition flag */
 
 /* 32 bit modes */
 #define PSR_MODE_USR 0x10
@@ -383,7 +384,12 @@ typedef uint64_t xen_callback_t;
 #define PSR_MODE_EL1t 0x04
 #define PSR_MODE_EL0t 0x00
 
-#define PSR_GUEST32_INIT  (PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)
+/*
+ * We set PSR_Z to be able to boot Linux kernel versions with an invalid
+ * encoding of the first 8 NOP instructions. See commit a92882a4d270 in
+ * Linux.
+ */
+#define PSR_GUEST32_INIT  (PSR_Z|PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)
 #define PSR_GUEST64_INIT (PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_EL1h)
 
 #define SCTLR_GUEST_INIT    xen_mk_ullong(0x00c50078)
Signature · 1 line
2.25.1
#5 lore raw
In reply to Stefano Stabellini
Quoted 11 lines · Stefano Stabellini

From: Stefano Stabellini <[email protected]>

The first 32 bytes of zImage are NOPs. When CONFIG_EFI is enabled in the kernel, certain versions of Linux will use an UNPREDICATABLE NOP encoding, sometimes resulting in an unbootable kernel. Whether the resulting kernel is bootable or not depends on the processor. See commit a92882a4d270 in the Linux kernel for all the details.

All kernel releases starting from Linux 4.9 without commit a92882a4d270 are affected.

Quoted 8 lines

Fortunately there is a simple workaround: setting the "Z" bit in CPSR make it so those invalid NOP instructions are never executed. That is because the instruction is conditional (not equal). So, on QEMU at least, the instruction will end up to be ignored and not generate an exception. Setting the "Z" bit makes those kernel versions bootable again and it is harmless in the other cases.

A discussion relevant to this came up with XTF/ARM.

There is not currently a written ABI for the start state of vCPUs, and there needs to be.  I know x86 is in a poor shape too, but we do at least have some scraps of docs littered around and a plan to write some proper Sphinx docs.

(A separate conversation was about booting from plain ELF files.  Linux ARM Zimage is entirely undocumented for 32bit, and discussions with RMK suggest that we've got bugs, while 64bit has insufficient documentation to demonstrate that our logic is correct.)

In particular...

Quoted 22 lines

---
Changes in v2:
- improve commit message
- add in-code comment
- move PSR_Z to the beginning

---
 xen/include/public/arch-arm.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
xen/include/public/arch-arm.h +1 -0
index 94b31511dd..81cee95f14 100644
@@ -361,6 +361,7 @@ typedef uint64_t xen_callback_t;
 #define PSR_DBG_MASK    (1<<9)        /* arm64: Debug Exception mask */
 #define PSR_IT_MASK     (0x0600fc00)  /* Thumb If-Then Mask */
 #define PSR_JAZELLE     (1<<24)       /* Jazelle Mode */
+#define PSR_Z           (1<<30)       /* Zero condition flag */
 
 /* 32 bit modes */
 #define PSR_MODE_USR 0x10

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 392

Quoted 10 lines
xen/include/public/arch-arm.h +6 -1
@@ -383,7 +384,12 @@ typedef uint64_t xen_callback_t;
 #define PSR_MODE_EL1t 0x04
 #define PSR_MODE_EL0t 0x00
 
-#define PSR_GUEST32_INIT  (PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)
+/*
+ * We set PSR_Z to be able to boot Linux kernel versions with an invalid
+ * encoding of the first 8 NOP instructions. See commit a92882a4d270 in
+ * Linux.
+ */
+#define PSR_GUEST32_INIT  (PSR_Z|PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)

... this change is either breaking the ABI, or demonstrates that these values must not be in a public header file to begin with.

~Andrew

#6 lore raw

Hi Andrew,

Quoted 23 lines · Andrew Cooper
Quoted 11 lines · Stefano Stabellini

From: Stefano Stabellini <[email protected]>

The first 32 bytes of zImage are NOPs. When CONFIG_EFI is enabled in the kernel, certain versions of Linux will use an UNPREDICATABLE NOP encoding, sometimes resulting in an unbootable kernel. Whether the resulting kernel is bootable or not depends on the processor. See commit a92882a4d270 in the Linux kernel for all the details.

All kernel releases starting from Linux 4.9 without commit a92882a4d270 are affected.

Quoted 8 lines

Fortunately there is a simple workaround: setting the "Z" bit in CPSR make it so those invalid NOP instructions are never executed. That is because the instruction is conditional (not equal). So, on QEMU at least, the instruction will end up to be ignored and not generate an exception. Setting the "Z" bit makes those kernel versions bootable again and it is harmless in the other cases.

A discussion relevant to this came up with XTF/ARM.

Quoted 8 lines

There is not currently a written ABI for the start state of vCPUs, and there needs to be.  I know x86 is in a poor shape too, but we do at least have some scraps of docs littered around and a plan to write some proper Sphinx docs.

(A separate conversation was about booting from plain ELF files.  Linux ARM Zimage is entirely undocumented for 32bit, and discussions with RMK suggest that we've got bugs

Do you mind providing more details on what would be the bugs here?

Quoted 2 lines

, while 64bit has insufficient documentation
to demonstrate that our logic is correct.)

Did you actually read https://github.com/torvalds/linux/blob/master/Documentation/arm64/booting.rst?

Quoted 28 lines

In particular...

Quoted 25 lines

---
Changes in v2:
- improve commit message
- add in-code comment
- move PSR_Z to the beginning

---
  xen/include/public/arch-arm.h | 8 +++++++-
  1 file changed, 7 insertions(+), 1 deletion(-)
xen/include/public/arch-arm.h +1 -0
index 94b31511dd..81cee95f14 100644
@@ -361,6 +361,7 @@ typedef uint64_t xen_callback_t;
  #define PSR_DBG_MASK    (1<<9)        /* arm64: Debug Exception mask */
  #define PSR_IT_MASK     (0x0600fc00)  /* Thumb If-Then Mask */
  #define PSR_JAZELLE     (1<<24)       /* Jazelle Mode */
+#define PSR_Z           (1<<30)       /* Zero condition flag */
 
  /* 32 bit modes */
  #define PSR_MODE_USR 0x10
@@ -383,7 +384,12 @@ typedef uint64_t xen_callback_t;
  #define PSR_MODE_EL1t 0x04
  #define PSR_MODE_EL0t 0x00
 
Quoted 10 lines

xen/include/public/arch-arm.h typedef uint64_t xen_callback_t; · line 392

Quoted 7 lines
xen/include/public/arch-arm.h +6 -1
-#define PSR_GUEST32_INIT  (PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)
+/*
+ * We set PSR_Z to be able to boot Linux kernel versions with an invalid
+ * encoding of the first 8 NOP instructions. See commit a92882a4d270 in
+ * Linux.
+ */
+#define PSR_GUEST32_INIT  (PSR_Z|PSR_ABT_MASK|PSR_FIQ_MASK|PSR_IRQ_MASK|PSR_MODE_SVC)

... this change is either breaking the ABI, or demonstrates that these values must not be in a public header file to begin with.

PSR_GUEST32_INIT is only exposed to the toolstack (see the ifdef above). It is defined in the arch-arm.h because it makes easier to keep the value in sync.

This is not part of the ABI and therefore we are free to change the value in any way we want. Setting Z is a convenient way to handle the Linux issue without making it too invasive.

Cheers,

Signature · 1 line
Julien Grall