Kernel boot regression with PAuth and aarch64-softmmu -cpu max and el2 enabled
Alex Bennée
Reviewed-by Peter MaydellHi,
Following up on yesterday's discussion on IRC I thought I'd better report on my findings in the permanent record so things don't get lost.
As I tend to periodically rebuild my test kernels from the current state of linux.git I occasionally run into these things. My test invocation is:
qemu-system-aarch64 -machine type=virt,virtualization=on \
-display none -m 4096 -serial mon:stdio \
-kernel ../../kernel-v8-plain.build/arch/arm64/boot/Image \
-append 'console=ttyAMA0 panic=-1' -no-reboot -cpu maxThe kernel is essentially a defconfig kernel with a bunch of the VIRTIO device drivers built-in for when I actually boot a more complex setup with disks and drives. However this is a boot test so doesn't really matter.
The -machine type=virt,virtualization=on enables our virt machine model with EL2 turned on. As there is no BIOS involved the kernel is invoked directly at EL2.
The -cpu max enabled a cortex-a57 + whatever extra features we've enabled in QEMU so far. It won't match any "real" CPU but it should be architecturally correct in so far we implement prerequisite features for any given feature. The cpuid feature bits should also be correct as we test them internally in QEMU to enable features.
The breakage is the kernel never boots (no output on serial port) and on attaching with gdb I found it stuck in:
(gdb) bt
#0 0xffffff8010a9e480 in overflow_stack ()
Backtrace stopped: not enough registers or memory available to unwind further
If I turn on exception tracing it looks like we go into an exception loop.
On the QEMU side this breakage comes in at:
commit 1ce32e47db52e3511132c7104770eae65d412144 (HEAD, refs/bisect/bad)
Author: Richard Henderson <[email protected]>
Date: Mon Jan 21 10:23:13 2019 +0000
target/arm: Enable PAuth for -cpu max
Message-id: [email protected]
and as you would expect the system boots fine with -cpu cortex-a57
On the kernel side it breaks at:
commit 04ca3204fa09f5f55c8f113b0072004a7b364ff4
Author: Mark Rutland <[email protected]>
Date: Fri Dec 7 18:39:30 2018 +0000
arm64: enable pointer authentication
Now that all the necessary bits are in place for userspace, add the necessary Kconfig logic to allow this to be enabled.
So predictably we failed at enabling PAuth somewhere between the kernel and QEMU.
I'm guessing the kernel so far has been tested on the fast model with a full chain of TF, UEFI and kernel?
I think Richard's tests were without EL2 enabled.
So in the case that the kernel boots in EL2 is it expecting anyone else to deal with Pauth exceptions or should it be able to cope with an enabled Pauth but no firmware underneath it?
Either we've got something wrong or we'll need to rethink what features the user can have enabled by -cpu max on a direct kernel boot.
--
Alex Bennée
Hi,
[adding Kristina, who is in charge of Linux pointer authentication]
Quoted 23 lines · Alex Bennée
Hi,
Following up on yesterday's discussion on IRC I thought I'd better report on my findings in the permanent record so things don't get lost.
As I tend to periodically rebuild my test kernels from the current state of linux.git I occasionally run into these things. My test invocation is:
qemu-system-aarch64 -machine type=virt,virtualization=on \ -display none -m 4096 -serial mon:stdio \ -kernel ../../kernel-v8-plain.build/arch/arm64/boot/Image \ -append 'console=ttyAMA0 panic=-1' -no-reboot -cpu maxThe kernel is essentially a defconfig kernel with a bunch of the VIRTIO device drivers built-in for when I actually boot a more complex setup with disks and drives. However this is a boot test so doesn't really matter.
The -machine type=virt,virtualization=on enables our virt machine model with EL2 turned on. As there is no BIOS involved the kernel is invoked directly at EL2.
Quoted 5 lines
The -cpu max enabled a cortex-a57 + whatever extra features we've enabled in QEMU so far. It won't match any "real" CPU but it should be architecturally correct in so far we implement prerequisite features for any given feature. The cpuid feature bits should also be correct as we test them internally in QEMU to enable features.
Just to check, does this enable VHE?
Quoted 9 lines
The breakage is the kernel never boots (no output on serial port) and on attaching with gdb I found it stuck in:
(gdb) bt
#00xffffff8010a9e480inoverflow_stack()
Backtrace stopped: not enough registers or memory available to unwind furtherIf I turn on exception tracing it looks like we go into an exception loop.
As mentioned on IRC, this looks very odd, since overflow_stack is a data pointer, not code. I can't presently see how we could branch here.
If you pass the kernel 'earlycon keep_bootcon', do you get any output?
Quoted 32 lines
On the QEMU side this breakage comes in at:
commit
1ce32e47db52e3511132c7104770eae65d412144(HEAD, refs/bisect/bad)
Author: Richard Henderson <[email protected]>
Date: Mon Jan 21 10:23:13 2019 +0000target/arm: Enable PAuth for -cpu max
Reviewed-by Peter Maydell <[email protected]>Signed-off-by Richard Henderson <[email protected]>Message-id: [email protected]
Signed-off-by Peter Maydell <[email protected]>and as you would expect the system boots fine with -cpu cortex-a57
On the kernel side it breaks at:
commit
04ca3204fa09f5f55c8f113b0072004a7b364ff4
Author: Mark Rutland <[email protected]>
Date: Fri Dec 7 18:39:30 2018 +0000arm64: enable pointer authentication
Now that all the necessary bits are in place for userspace, add the necessary
Kconfiglogic to allow this to be enabled.Signed-off-by Mark Rutland <[email protected]>Signed-off-by Kristina Martsenko <[email protected]>Cc Catalin Marinas <[email protected]>Cc Will Deacon <[email protected]>Signed-off-by Will Deacon <[email protected]>
Quoted 5 lines
So predictably we failed at enabling PAuth somewhere between the kernel and QEMU.
I'm guessing the kernel so far has been tested on the fast model with a full chain of TF, UEFI and kernel?
The kernel has been tested on a fast model with the Linux bootwrapper:
https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/
Kristina, could you confirm whether or not it's been tested with ATF+UEFI?
Quoted 5 lines
I think Richard's tests were without EL2 enabled.
So in the case that the kernel boots in EL2 is it expecting anyone else to deal with Pauth exceptions or should it be able to cope with an enabled Pauth but no firmware underneath it?
So long as the highest implemented exception level is EL2, the kernel should handle that itself. During boot we'll configure HCR_EL2.{API,APK} in el2_setup().
From that point onwards, there should be no traps for pointer authentication functionality from EL1, AFAICT.
Quoted 2 lines
Either we've got something wrong or we'll need to rethink what features the user can have enabled by -cpu max on a direct kernel boot.
It's not immediately clear to me when precisely things are going wrong, so I think we need to narrow that down first. For example, it's not clear whether a trap is being taken, or something is unexpectedly behaving is UNDEF.
Is it possible to watch the exception vectors to see if/when an exception is taken, and from where?
Thanks,
Mark.
Quoted 8 lines · Mark Rutland
Quoted 5 lines · Alex Bennée
The -cpu max enabled a cortex-a57 + whatever extra features we've enabled in QEMU so far. It won't match any "real" CPU but it should be architecturally correct in so far we implement prerequisite features for any given feature. The cpuid feature bits should also be correct as we test them internally in QEMU to enable features.
Just to check, does this enable VHE?
It does not. (We have no implementation of VHE yet -- as and when we do implement that "-cpu max" will turn it on, but today it doesn't.)
thanks
-- PMM
Quoted 8 lines · Peter Maydell
Quoted 7 lines · Mark Rutland
Quoted 5 lines · Alex Bennée
The -cpu max enabled a cortex-a57 + whatever extra features we've enabled in QEMU so far. It won't match any "real" CPU but it should be architecturally correct in so far we implement prerequisite features for any given feature. The cpuid feature bits should also be correct as we test them internally in QEMU to enable features.
Quoted 4 lines
Quoted 1 line
Just to check, does this enable VHE?
It does not. (We have no implementation of VHE yet -- as and when we do implement that "-cpu max" will turn it on, but today it doesn't.)
Ok. In that case, the kernel should drop to EL1N and stay there after it configures EL2.
There are no pointer authentication traps taken to EL1N, so if something is going wrong, I suspect either:
* a trap is taken to EL2, and we hyp_panic()
* an UNDEF is being taken to EL1N
Thanks,
Mark.