ARM: initialize jump labels before setup_machine_fdt()
Subsystems (4)
This patch is fatal for me in the downstream Raspberry Pi kernel - it locks up on boot even before the earlycon output is available. Hacking jump_label_init to skip the jump_entry for "crng_is_ready" allows it to boot, but is likely to have consequences further down the line.
The stable branch may not be living up to its name, but I don't think this is a quick fix.
Phil
Jason A. Donenfeld
maintainerHi Phil,
Thanks for testing this. Can you let me know if v1 of this works?
https://lore.kernel.org/lkml/[email protected]/
(I'll also fashion a revert for this part of stable.)
Jason
Hi Jason,
Quoted 9 lines · Jason A. Donenfeld
Hi Phil,
Thanks for testing this. Can you let me know if v1 of this works?
https://lore.kernel.org/lkml/[email protected]/
(I'll also fashion a revert for this part of stable.)
Jason
Thanks for the quick response, but that doesn't work for me either. Let me say again that I'm on a downstream kernel (rpi-5.15.y) so this may not be a universal problem, but merging either of these fixing patches would be fatal for us.
Phil
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Jason A. Donenfeld
maintainerHi Phil,
Quoted 3 lines · Phil Elwell
Thanks for the quick response, but that doesn't work for me either. Let me say again that I'm on a downstream kernel (rpi-5.15.y) so this may not be a universal problem, but merging either of these fixing patches would be fatal for us.
Alright, thanks. And I'm guessing you don't currently have a problem *without* either of the fixing patches, because your device tree doesn't use rng-seed. Is that right?
In anycase, I sent in a revert to get all the static branch stuff out of stable -- https://lore.kernel.org/stable/[email protected]/
-- so the "urgency" of this should decrease and we can fix this as normal during the 5.19 cycle. But with that said, I do want to get this fixed as soon as possible still. I'll be back at my desk tonight and will finally have access to real hardware again. Are you hitting this by loading a 32bit kernel on a raspi4? Or running a 32bit kernel on the raspi1? Or some other combo?
Jason
Hi Jason,
Quoted 5 lines · Jason A. Donenfeld
Quoted 3 lines · Phil Elwell
Thanks for the quick response, but that doesn't work for me either. Let me say again that I'm on a downstream kernel (rpi-5.15.y) so this may not be a universal problem, but merging either of these fixing patches would be fatal for us.
Quoted 8 lines
Alright, thanks. And I'm guessing you don't currently have a problem *without* either of the fixing patches, because your device tree doesn't use rng-seed. Is that right?
In anycase, I sent in a revert to get all the static branch stuff out of stable -- https://lore.kernel.org/stable/[email protected]/
-- so the "urgency" of this should decrease and we can fix this as normal during the 5.19 cycle.
Since the above revert got queued in -stable, I assume you don't need commit 73e2d827a501 ("arm64: Initialize jump labels before setup_machine_fdt()") in stable either.
Do you plan to fix the crng_ready() static branch differently? If you do, I'd like to revert the corresponding arm64 commit as well. It seems to be harmless but I'd rather not keep it if no longer needed. So please keep me updated whatever you decide.
Thanks.
Signature · 1 line
Catalin
Jason A. Donenfeld
maintainerHi Catalin,
Quoted 12 lines · Catalin Marinas
Hi Jason,
Quoted 9 lines · Jason A. Donenfeld
Quoted 3 lines · Phil Elwell
Thanks for the quick response, but that doesn't work for me either. Let me say again that I'm on a downstream kernel (rpi-5.15.y) so this may not be a universal problem, but merging either of these fixing patches would be fatal for us.
Alright, thanks. And I'm guessing you don't currently have a problem *without* either of the fixing patches, because your device tree doesn't use rng-seed. Is that right?
Quoted 8 lines
Quoted 4 lines
In anycase, I sent in a revert to get all the static branch stuff out of stable -- https://lore.kernel.org/stable/[email protected]/
-- so the "urgency" of this should decrease and we can fix this as normal during the 5.19 cycle.Since the above revert got queued in -stable, I assume you don't need commit
73e2d827a501("arm64: Initialize jump labels beforesetup_machine_fdt()") in stable either.
I made the point here:
https://lore.kernel.org/stable/[email protected]/T/#m8f33bc14b677980abe690e5c7a4909b5902010cc
Quoted 4 lines
Do you plan to fix the
crng_ready()static branch differently? If you do, I'd like to revert the corresponding arm64 commit as well. It seems to be harmless but I'd rather not keep it if no longer needed. So please keep me updated whatever you decide.
I sent a "backup commit" for that here: https://lore.kernel.org/all/[email protected]/
But I would like a few days to see if there's some trivial way of not
needing that on arm32. If it turns out to be easy, then I'd prefer the
direct fix akin to the arm64 one. If it turns out to be not easy, then
I'll merge the backup commit. I'll keep you posted (and I assume anyway
you'll see the arm32 attempts progress or fail here, also).
Jason
Jason A. Donenfeld
maintainerHi Catalin,
Quoted 4 lines · Catalin Marinas
Do you plan to fix the
crng_ready()static branch differently? If you do, I'd like to revert the corresponding arm64 commit as well. It seems to be harmless but I'd rather not keep it if no longer needed. So please keep me updated whatever you decide.
Feel free to revert. I'm aborting this line of inquiry and will just fix it downstream in random.c.
Jason
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Jason,
Quoted 8 lines · Jason A. Donenfeld
Quoted 4 lines · Catalin Marinas
Do you plan to fix the
crng_ready()static branch differently? If you do, I'd like to revert the corresponding arm64 commit as well. It seems to be harmless but I'd rather not keep it if no longer needed. So please keep me updated whatever you decide.Feel free to revert. I'm aborting this line of inquiry and will just fix it downstream in
random.c.
Thanks for the update.
Signature · 6 lines
Catalin _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Greg KH
maintainerQuoted 9 lines · Phil Elwell
Hi Jason,
Quoted 6 lines · Jason A. Donenfeld
Hi Phil,
Thanks for testing this. Can you let me know if v1 of this works?
Quoted 8 lines
Quoted 3 lines
(I'll also fashion a revert for this part of stable.)
Jason
Thanks for the quick response, but that doesn't work for me either. Let me say again that I'm on a downstream kernel (rpi-5.15.y) so this may not be a universal problem, but merging either of these fixing patches would be fatal for us.
I have reports of a "clean" 5.15.45 working just fine on a rpi. Anything special in your tree that isn't upstream yet that might be conflicting with this? Any chance you can try a kernel.org release instead?
thanks,
greg k-h
Hi Greg,
Quoted 19 lines · Greg KH
Quoted 9 lines · Phil Elwell
Hi Jason,
Quoted 6 lines · Jason A. Donenfeld
Hi Phil,
Thanks for testing this. Can you let me know if v1 of this works?
Quoted 8 lines
Quoted 3 lines
(I'll also fashion a revert for this part of stable.)
Jason
Thanks for the quick response, but that doesn't work for me either. Let me say again that I'm on a downstream kernel (rpi-5.15.y) so this may not be a universal problem, but merging either of these fixing patches would be fatal for us.
Quoted 4 lines
I have reports of a "clean" 5.15.45 working just fine on a rpi. Anything special in your tree that isn't upstream yet that might be conflicting with this? Any chance you can try a kernel.org release instead?
A clean 5.15.45 boots cleanly, whereas a downstream kernel shows the static key warning (but it does go on to boot). The significant difference is that our defconfigs set CONFIG_RANDOM_TRUST_BOOTLOADER=y - defining that on top of multi_v7_defconfig demonstrates the issue on a clean 5.15.45. Conversely, not setting that option in a downstream kernel build avoids the warning, presumably because it takes much longer to accumulate the required entropy.
Phil
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Jason A. Donenfeld
maintainerHi Phil,
Quoted 5 lines · Phil Elwell
A clean 5.15.45 boots cleanly, whereas a downstream kernel shows the static key warning (but it does go on to boot). The significant difference is that our defconfigs set
CONFIG_RANDOM_TRUST_BOOTLOADER=y- defining that on top ofmulti_v7_defconfigdemonstrates the issue on a clean 5.15.45. Conversely, not setting that option in a downstream kernel build avoids the warning
Ah, that makes sense. Note that I've got a patch out for changing that defconfig as well to be Y, which means the CI will catch this sort of thing in the future.
Jason
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Russell King (Oracle)
maintainerQuoted 7 lines · Jason A. Donenfeld
Hi Phil,
Thanks for testing this. Can you let me know if v1 of this works?
https://lore.kernel.org/lkml/[email protected]/
(I'll also fashion a revert for this part of stable.)
As the arm32 version hasn't been merged yet, how is it in stable already? If it is in stable, isn't that a yet another violation of the stable kernel rules?
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
Jason A. Donenfeld
maintainerQuoted 5 lines · Russell King (Oracle)
Quoted 4 lines · Jason A. Donenfeld
Hi Phil,
Thanks for testing this. Can you let me know if v1 of this works?
Quoted 7 lines
Quoted 3 lines
https://lore.kernel.org/lkml/[email protected]/
(I'll also fashion a revert for this part of stable.)
As the arm32 version hasn't been merged yet, how is it in stable already? If it is in stable, isn't that a yet another violation of the stable kernel rules?
You misunderstood my ambiguous sentence, sorry. I meant a revert of the thing in stable that makes this discussion here a stable discussion rather than a 5.19 discussion. No arm32 stuff has been committed anywhere.
Jason
Jason A. Donenfeld
maintainerHi Phil,
Quoted 5 lines · Phil Elwell
This patch is fatal for me in the downstream Raspberry Pi kernel - it locks up on boot even before the earlycon output is available. Hacking
jump_label_initto skip thejump_entryfor "crng_is_ready" allows it to boot, but is likely to have consequences further down the line.
Also, reading this a few times, I'm not 100% sure I understand what you did to hack around this and why that works. Think you could paste your hackpatch just out of interest to the discussion (but obviously not to be applied)?
Jason
Quoted 9 lines · Jason A. Donenfeld
Hi Phil,
Quoted 5 lines · Phil Elwell
This patch is fatal for me in the downstream Raspberry Pi kernel - it locks up on boot even before the earlycon output is available. Hacking
jump_label_initto skip thejump_entryfor "crng_is_ready" allows it to boot, but is likely to have consequences further down the line.
Quoted 4 lines
Also, reading this a few times, I'm not 100% sure I understand what you did to hack around this and why that works. Think you could paste your hackpatch just out of interest to the discussion (but obviously not to be applied)?
This is the minimal version of my workaround patch that at least allows the board to boot. Bear in mind that it was written with no previous knowledge of jump labels and was arrived at by iteratively bisecting the list of jump_labels until the first dangerous one was found, then later working out that there was only one.
kernel/jump_label.c +4 -1
@@ -466,6 +466,7 @@ void __init jump_label_init(void)struct jump_entry *iter_stop = __stop___jump_table;struct static_key *key =NULL ;struct jump_entry *iter; +int iter_count =0 ;/* * Since we are initializing the static_key.enabled field with
@@ -499,7 +500,9 @@ void __init jump_label_init(void)continue ; key = iterk; -static_key_set_entries (key, iter); + iter_count++; +if (iter_count !=1083 ) +static_key_set_entries (key, iter); } static_key_initialized = true;jump_label_unlock ();
I'm sure this could be rewritten in a less fragile manner making reference to crng_is_ready directly, but this is where I got to yesterday.
Ha! I just proved the patch's fragility by switching from a Pi 2 to a Pi 4, so the saner version is:
drivers/char/random.c +1 -1
@@ -79,7 +79,7 @@ static enum {CRNG_EARLY =1 ,/* At least POOL_EARLY_BITS collected */ CRNG_READY = 2 /* Fully initialized with POOL_READY_BITS collected */ } crng_init__read_mostly =CRNG_EMPTY ; -static DEFINE_STATIC_KEY_FALSE (crng_is_ready); +DEFINE_STATIC_KEY_FALSE (crng_is_ready);#define crng_ready () (static_branch_likely(&crng_is_ready) || crng_init >=
CRNG_READY)
/* Various types of waiters for crng_init->CRNG_READY transition. */ static DECLARE_WAIT_QUEUE_HEAD(crng_init_wait);
kernel/jump_label.c +3 -1
@@ -484,6 +484,7 @@ void __init jump_label_init(void)jump_label_sort_entries (iter_start, iter_stop);for (iter = iter_start; iter < iter_stop; iter++) { +extern struct static_key_false crng_is_ready;struct static_key *iterk;bool in_init;
@@ -499,7 +500,8 @@ void __init jump_label_init(void)continue ; key = iterk; -static_key_set_entries (key, iter); +if (key != &crng_is_ready.key) +static_key_set_entries (key, iter); } static_key_initialized = true;jump_label_unlock ();
And to answer your questions from the other thread:
* Without any fixing patches we see the warning messages because we are using rng-seed in DT.
* I've seen the problem on a Pi 2 and a Pi 4 running 32-bit kernels.
Phil
Jason A. Donenfeld
maintainerHey again,
Quoted 15 lines · Phil Elwell
Quoted 9 lines · Jason A. Donenfeld
Hi Phil,
Quoted 5 lines · Phil Elwell
This patch is fatal for me in the downstream Raspberry Pi kernel - it locks up on boot even before the earlycon output is available. Hacking
jump_label_initto skip thejump_entryfor "crng_is_ready" allows it to boot, but is likely to have consequences further down the line.Quoted 4 lines
Also, reading this a few times, I'm not 100% sure I understand what you did to hack around this and why that works. Think you could paste your hackpatch just out of interest to the discussion (but obviously not to be applied)?
Quoted 5 lines
This is the minimal version of my workaround patch that at least allows the board to boot. Bear in mind that it was written with no previous knowledge of jump labels and was arrived at by iteratively bisecting the list of
jump_labelsuntil the first dangerous one was found, then later working out that there was only one.
Looks like this patch fails due to CONFIG_STRICT_KERNEL_RWX. Investigating deeper now, but that for starters seems to be the differentiating factor between my prior test rig and one that reproduces the error. I assume your raspi also sets CONFIG_STRICT_KERNEL_RWX.
Jason
Jason,
Quoted 24 lines · Jason A. Donenfeld
Hey again,
Quoted 15 lines · Phil Elwell
Quoted 9 lines · Jason A. Donenfeld
Hi Phil,
Quoted 5 lines · Phil Elwell
This patch is fatal for me in the downstream Raspberry Pi kernel - it locks up on boot even before the earlycon output is available. Hacking
jump_label_initto skip thejump_entryfor "crng_is_ready" allows it to boot, but is likely to have consequences further down the line.Quoted 4 lines
Also, reading this a few times, I'm not 100% sure I understand what you did to hack around this and why that works. Think you could paste your hackpatch just out of interest to the discussion (but obviously not to be applied)?
Quoted 5 lines
This is the minimal version of my workaround patch that at least allows the board to boot. Bear in mind that it was written with no previous knowledge of jump labels and was arrived at by iteratively bisecting the list of
jump_labelsuntil the first dangerous one was found, then later working out that there was only one.
Quoted 4 lines
Looks like this patch fails due to
CONFIG_STRICT_KERNEL_RWX. Investigating deeper now, but that for starters seems to be the differentiating factor between my prior test rig and one that reproduces the error. I assume your raspi also setsCONFIG_STRICT_KERNEL_RWX.
Yes, it does, as does multi_v7_defconfig.
Phil
Jason A. Donenfeld
maintainerHi Phil,
Quoted 27 lines · Phil Elwell
Jason,
Quoted 24 lines · Jason A. Donenfeld
Hey again,
Quoted 20 lines · Phil Elwell
Quoted 9 lines · Jason A. Donenfeld
Hi Phil,
Quoted 5 lines · Phil Elwell
This patch is fatal for me in the downstream Raspberry Pi kernel - it locks up on boot even before the earlycon output is available. Hacking
jump_label_initto skip thejump_entryfor "crng_is_ready" allows it to boot, but is likely to have consequences further down the line.Quoted 4 lines
Also, reading this a few times, I'm not 100% sure I understand what you did to hack around this and why that works. Think you could paste your hackpatch just out of interest to the discussion (but obviously not to be applied)?
This is the minimal version of my workaround patch that at least allows the board to boot. Bear in mind that it was written with no previous knowledge of jump labels and was arrived at by iteratively bisecting the list of
jump_labelsuntil the first dangerous one was found, then later working out that there was only one.
Quoted 6 lines
Quoted 4 lines
Looks like this patch fails due to
CONFIG_STRICT_KERNEL_RWX. Investigating deeper now, but that for starters seems to be the differentiating factor between my prior test rig and one that reproduces the error. I assume your raspi also setsCONFIG_STRICT_KERNEL_RWX.Yes, it does, as does
multi_v7_defconfig.
Oh good. Adjusting my CI now to have that.
Having tickled arch/arm/ a little bit now, this is looking sort of complicated. So I think I might be leaning toward giving up and just rolling with <https://git.zx2c4.com/linux-rng/commit/?id=78f79dda>.
Unless of course somebody has some ARM chops and can think of a quick easy fix.
Jason
Quoted 37 lines · Jason A. Donenfeld
Hi Phil,
Quoted 27 lines · Phil Elwell
Jason,
Quoted 24 lines · Jason A. Donenfeld
Hey again,
Quoted 20 lines · Phil Elwell
Quoted 9 lines · Jason A. Donenfeld
Hi Phil,
Quoted 5 lines · Phil Elwell
This patch is fatal for me in the downstream Raspberry Pi kernel - it locks up on boot even before the earlycon output is available. Hacking
jump_label_initto skip thejump_entryfor "crng_is_ready" allows it to boot, but is likely to have consequences further down the line.Quoted 4 lines
Also, reading this a few times, I'm not 100% sure I understand what you did to hack around this and why that works. Think you could paste your hackpatch just out of interest to the discussion (but obviously not to be applied)?
This is the minimal version of my workaround patch that at least allows the board to boot. Bear in mind that it was written with no previous knowledge of jump labels and was arrived at by iteratively bisecting the list of
jump_labelsuntil the first dangerous one was found, then later working out that there was only one.Quoted 6 lines
Quoted 4 lines
Looks like this patch fails due to
CONFIG_STRICT_KERNEL_RWX. Investigating deeper now, but that for starters seems to be the differentiating factor between my prior test rig and one that reproduces the error. I assume your raspi also setsCONFIG_STRICT_KERNEL_RWX.Yes, it does, as does
multi_v7_defconfig.
Quoted 8 lines
Oh good. Adjusting my CI now to have that.
Having tickled
arch/arm/a little bit now, this is looking sort of complicated. So I think I might be leaning toward giving up and just rolling with <https://git.zx2c4.com/linux-rng/commit/?id=78f79dda>.Unless of course somebody has some ARM chops and can think of a quick easy fix.
I've not looked at the performance implications (if any), but in terms of when the RNG initilialization completes and the lack of a WARN I'm happy with that patch, a.k.a. [1].
Phil
Jason A. Donenfeld
maintainerPATCH v3 1/1 ARM: initialize jump labels before setup_machine_fdt()
In reply to Jason A. DonenfeldStephen reported that a static key warning splat appears during early boot on arm64 systems that credit randomness from device trees that contain an "rng-seed" property, because setup_machine_fdt() is called before jump_label_init() during setup_arch(), which was fixed by 73e2d827a501 ("arm64: Initialize jump labels before setup_machine_fdt()").
The same basic issue applies to arm32 as well. So this commit adds a call to jump_label_init() just before setup_machine_fdt(). Since the page maps haven't been set yet, this also requires us to use the early patching code in the jump label code.
--- arch/arm/kernel/jump_label.c | 3 ++- arch/arm/kernel/setup.c | 1 + arch/arm/mm/mmu.c | 3 +++ 3 files changed, 6 insertions(+), 1 deletion(-)
arch/arm/kernel/jump_label.c +2 -1
@@ -8,6 +8,7 @@ static void __arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type, bool is_static) { +extern bool early_mm_initialized;void *addr = (void *)entry->code;unsigned int insn;
@@ -16,7 +17,7 @@ static void __arch_jump_label_transform(struct jump_entry *entry,else insn =arm_gen_nop (); -if (is_static) +if (is_static || !early_mm_initialized)__patch_text_early (addr, insn);else patch_text (addr, insn);
arch/arm/kernel/setup.c +1 -0
@@ -1101,6 +1101,7 @@ void __init setup_arch(char **cmdline_p) atags_vaddr =FDT_VIRT_BASE (__atags_pointer);setup_processor (); +jump_label_init ();if (atags_vaddr) { mdesc =setup_machine_fdt (atags_vaddr);if (mdesc)
arch/arm/mm/mmu.c +3 -0
@@ -1754,10 +1754,13 @@ void __init paging_init(const struct machine_desc *mdesc)__flush_dcache_page (NULL , empty_zero_page); } +bool early_mm_initialized; +void __init early_mm_init (const struct machine_desc *mdesc) {build_mem_type_table ();early_paging_init (mdesc); + early_mm_initialized = true; } void set_pte_at(struct mm_struct *mm, unsigned long addr,
Signature · 7 lines
2.35.1 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Jason A. Donenfeld
maintainerThis patch isn't needed in the end. An equivalent patch is needed on xtensa, powerpc, arc, mips, arm32, arm64, riscv. That's a bit much and points to a larger issue. So I'll fix this the ugly way in the random.c code :(.
Jason
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel