Skip to contentSkip to conversation

x86-32: Early microcode loading stumbles over disabled DYNAMIC_FTRACE

PATCH Jan Kiszka 4 participants 14 messages Aug 23, 2013 – Aug 31, 2013 x86

lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

Hi all,

32-bit kernels currently crash/reboot in early microcode loading when non-dynamic function tracing is enabled. Several functions in that path get instrumented with mcount, but its non-dynamic implementation does not work before paging is enabled (it accesses global variables at wrong addresses).

Below some hunks to get it working again - at least in the absence of any microcode in the initrd. Marking all involved functions as __init is another option (as __init implies notrace). But I bet there is more hidden. I see e.g. a pr_warn() in find_cpio_init that should trigger the issue as well if we hit the error it reports (btw. printing at this point of the boot should not work anyway, should it?).

Better ideas?

Jan

arch/x86/kernel/Makefile +3 -0
index 88d99ea..c093a9c 100644
@@ -14,6 +14,9 @@ CFLAGS_REMOVE_pvclock.o = -pg
 CFLAGS_REMOVE_kvmclock.o = -pg
 CFLAGS_REMOVE_ftrace.o = -pg
 CFLAGS_REMOVE_early_printk.o = -pg
+CFLAGS_REMOVE_microcode_core_early.o = -pg
+CFLAGS_REMOVE_microcode_intel_early.o = -pg
+CFLAGS_REMOVE_microcode_amd_early.o = -pg
 endif
 
 CFLAGS_irq.o := -I$(src)/../include/asm/trace
arch/x86/lib/Makefile +4 -0
index 96b2c66..0917142 100644
@@ -2,6 +2,10 @@
 # Makefile for x86 specific library files.
 #
 
+ifdef CONFIG_FUNCTION_TRACER
+CFLAGS_REMOVE_string_32.o = -pg
+endif
+
 inat_tables_script = $(srctree)/arch/x86/tools/gen-insn-attr-x86.awk
 inat_tables_maps = $(srctree)/arch/x86/lib/x86-opcode-map.txt
 quiet_cmd_inat_tables = GEN     $@
Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
maintainer
#1 lore raw
Quoted 6 lines · Jan Kiszka

Below some hunks to get it working again - at least in the absence of any microcode in the initrd. Marking all involved functions as __init is another option (as __init implies notrace). But I bet there is more hidden. I see e.g. a pr_warn() in find_cpio_init that should trigger the issue as well if we hit the error it reports (btw. printing at this point of the boot should not work anyway, should it?).

I guess we can do early_printk there instead as x86_64_start_kernel() does it but I'm not sure for the 32-bit case where we call load_ucode_bsp/ap before we've even enabled paging.

Signature · 5 lines
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--
author
#2 lore raw
Quoted 11 lines · Borislav Petkov
Quoted 6 lines · Jan Kiszka

Below some hunks to get it working again - at least in the absence of any microcode in the initrd. Marking all involved functions as __init is another option (as __init implies notrace). But I bet there is more hidden. I see e.g. a pr_warn() in find_cpio_init that should trigger the issue as well if we hit the error it reports (btw. printing at this point of the boot should not work anyway, should it?).

I guess we can do early_printk there instead as x86_64_start_kernel() does it but I'm not sure for the 32-bit case where we call load_ucode_bsp/ap before we've even enabled paging.

early_printk doesn't seem to work as well on 32-bit until paging was enabled.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
#3 lore raw
In reply to Jan Kiszka
Quoted 15 lines · Jan Kiszka

Hi all,

32-bit kernels currently crash/reboot in early microcode loading when non-dynamic function tracing is enabled. Several functions in that path get instrumented with mcount, but its non-dynamic implementation does not work before paging is enabled (it accesses global variables at wrong addresses).

Below some hunks to get it working again - at least in the absence of any microcode in the initrd. Marking all involved functions as __init is another option (as __init implies notrace). But I bet there is more hidden. I see e.g. a pr_warn() in find_cpio_init that should trigger the issue as well if we hit the error it reports (btw. printing at this point of the boot should not work anyway, should it?).

Quoted 4 lines

Better ideas?

Jan

A better idea would be for the mcount/__fentry__ function to simply return until the function tracing stuff is actually ready.

-hpa

#4 lore raw
Quoted 16 lines · On Fri, 30 Aug 2013 08:36:58 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 15 lines · Jan Kiszka

Hi all,

32-bit kernels currently crash/reboot in early microcode loading when non-dynamic function tracing is enabled. Several functions in that path get instrumented with mcount, but its non-dynamic implementation does not work before paging is enabled (it accesses global variables at wrong addresses).

Below some hunks to get it working again - at least in the absence of any microcode in the initrd. Marking all involved functions as __init is another option (as __init implies notrace). But I bet there is more hidden. I see e.g. a pr_warn() in find_cpio_init that should trigger the issue as well if we hit the error it reports (btw. printing at this point of the boot should not work anyway, should it?).

Quoted 8 lines
Quoted 4 lines

Better ideas?

Jan

A better idea would be for the mcount/__fentry__ function to simply return until the function tracing stuff is actually ready.

It does when dynamic tracing is enabled. But this issue is with static tracing, there's no code modification, thus all functions call mcount/__fentry__ and we need to look at a variable to determine if we should trace or not.

-- Steve

#5 lore raw
Quoted 6 lines · Steven Rostedt

It does when dynamic tracing is enabled. But this issue is with static tracing, there's no code modification, thus all functions call mcount/__fentry__ and we need to look at a variable to determine if we should trace or not.

And of course global variables are a no-no without special handling in pre-paging mode. Sigh.

You *could* bail out of the calling address is < PAGE_OFFSET.

-hpa

#6 lore raw
Quoted 8 lines · On Fri, 30 Aug 2013 09:35:18 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 6 lines · Steven Rostedt

It does when dynamic tracing is enabled. But this issue is with static tracing, there's no code modification, thus all functions call mcount/__fentry__ and we need to look at a variable to determine if we should trace or not.

Quoted 5 lines

And of course global variables are a no-no without special handling in pre-paging mode. Sigh.

You *could* bail out of the calling address is < PAGE_OFFSET.

We could do that too, as static tracing already gives us high overhead, I'm not sure how much more overhead another check like this even matters.

Who uses static tracing? I only use it to test that it still works ;-)

-- Steve

#7 lore raw
Quoted 14 lines · Steven Rostedt
Quoted 11 lines · On Fri, 30 Aug 2013 09:35:18 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 6 lines · Steven Rostedt

It does when dynamic tracing is enabled. But this issue is with static tracing, there's no code modification, thus all functions call mcount/__fentry__ and we need to look at a variable to determine if we should trace or not.

And of course global variables are a no-no without special handling in pre-paging mode. Sigh.

Quoted 6 lines
Quoted 2 lines

You *could* bail out of the calling address is < PAGE_OFFSET.

We could do that too, as static tracing already gives us high overhead, I'm not sure how much more overhead another check like this even matters.

It's not too expensive: a compare and a jump.

Quoted 1 line

Who uses static tracing? I only use it to test that it still works ;-)

Can we get rid of it?

#8 lore raw
Quoted 4 lines · On Fri, 30 Aug 2013 10:16:00 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 1 line

Who uses static tracing? I only use it to test that it still works ;-)

Can we get rid of it?

I've though about it. But as some archs only have static tracing available (I need to revisit to make sure that's still the case), I kept it because I only test static tracing on x86.

I'm not sure if people still use it for x86 or not.

Jan, you use static tracing on x86?

-- Steve

author
#9 lore raw
Quoted 8 lines · Steven Rostedt
Quoted 4 lines · On Fri, 30 Aug 2013 10:16:00 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 1 line

Who uses static tracing? I only use it to test that it still works ;-)

Can we get rid of it?

Quoted 7 lines

I've though about it. But as some archs only have static tracing available (I need to revisit to make sure that's still the case), I kept it because I only test static tracing on x86.

I'm not sure if people still use it for x86 or not.

Jan, you use static tracing on x86?

Yes, due to out-of-tree code that is not compatible with dynamic tracing. It might be feasible to change this, I didn't try again recently. If static tracing goes away upstream, I'll have to sooner or later.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux
#10 lore raw
In reply to Steven Rostedt
Quoted 14 lines · Steven Rostedt
Quoted 11 lines · On Fri, 30 Aug 2013 09:35:18 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 6 lines · Steven Rostedt

It does when dynamic tracing is enabled. But this issue is with static tracing, there's no code modification, thus all functions call mcount/__fentry__ and we need to look at a variable to determine if we should trace or not.

And of course global variables are a no-no without special handling in pre-paging mode. Sigh.

Quoted 9 lines
Quoted 2 lines

You *could* bail out of the calling address is < PAGE_OFFSET.

We could do that too, as static tracing already gives us high overhead, I'm not sure how much more overhead another check like this even matters.

Who uses static tracing? I only use it to test that it still works ;-)

Anyway... can we add this to the static tracing code? It is two instructions and is only needed on x86-32. For performance, use dynamic.

-hpa

#11 lore raw
Quoted 4 lines · On Fri, 30 Aug 2013 13:48:55 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 1 line
Quoted 1 line

You *could* bail out of the calling address is < PAGE_OFFSET.

Anyway... can we add this to the static tracing code? It is two instructions and is only needed on x86-32. For performance, use dynamic.

Sure, do you want to add it ?

-- Steve

maintainer
#12 lore raw
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Fri, 30 Aug 2013 13:48:55 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 1 line
Quoted 1 line

You *could* bail out of the calling address is < PAGE_OFFSET.

Anyway... can we add this to the static tracing code? It is two instructions and is only needed on x86-32. For performance, use dynamic.

Sure, do you want to add it ?

Another, simpler option could be to make static tracing depend on !MICROCODE until the first has been removed.

Signature · 2 lines
Regards/Gruss,
Boris.
#13 lore raw
Quoted 6 lines · On Sat, 31 Aug 2013 07:07:37 +0200 Borislav Petkov <[email protected]> wrote:
Quoted 5 lines · Steven Rostedt
Quoted 2 lines · On Fri, 30 Aug 2013 13:48:55 -0700 "H. Peter Anvin" <[email protected]> wrote:
Quoted 1 line
Quoted 1 line

You *could* bail out of the calling address is < PAGE_OFFSET.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Anyway... can we add this to the static tracing code? It is two instructions and is only needed on x86-32. For performance, use dynamic.

Sure, do you want to add it ?

Another, simpler option could be to make static tracing depend on !MICROCODE until the first has been removed.

Might be trickier than you think. Static tracing is the default. Dynamic tracing is the extension. By making static tracing depend on !MICROCODE, you basically made function tracing depend on !MICROCODE.

I guess you could make MICROCODE select DYNAMIC_FTRACE, but that could get messy.

-- Steve