x86/alternatives: Nest them
cover x86/alternatives: Nest them
From: "Borislav Petkov (AMD)" <[email protected]>
Hi,
here's the next revision
[ v1 - the last one was v0 even if not denoted as such:
https://lore.kernel.org/r/[email protected] ]
of the alternatives nesting patchset.
All feedback has been incorporated and testing with it still looks good.
Thx.
Changelog:
v0:
---
this is basically peterz's idea to nest the alternative macros with a fix to an issue I encountered while testing.
For ease of bisection, the old macros are converted to the new, nested variants in a step-by-step manner so that in case an issue is encountered during testing, one can pinpoint the place where it fails easier. Because debugging alternatives is a serious pain.
Testing here on my farm looks good but who knows what happens out there, in the wild.
Thx.
Borislav Petkov (AMD) (13):
x86/alternative: Zap alternative_ternary() x86/alternative: Convert alternative() x86/alternative: Convert alternative_2() x86/alternative: Convert alternative_input() x86/alternative: Convert alternative_io() x86/alternative: Convert alternative_call() x86/alternative: Convert alternative_call_2() x86/alternative: Convert ALTERNATIVE_TERNARY() x86/alternative: Convert ALTERNATIVE_3() x86/alternative: Convert the asm ALTERNATIVE() macro x86/alternative: Convert the asm ALTERNATIVE_2() macro x86/alternative: Convert the asm ALTERNATIVE_3() macro x86/alternative: Replace the old macros
Peter Zijlstra (1):
x86/alternatives: Add nested alternatives macros
arch/x86/include/asm/alternative.h | 225 +++++++++-------------------- arch/x86/kernel/alternative.c | 20 ++- arch/x86/kernel/fpu/xstate.h | 14 +- tools/objtool/arch/x86/special.c | 23 +++ tools/objtool/special.c | 16 +- 5 files changed, 125 insertions(+), 173 deletions(-)
Signature · 1 line
2.43.0
PATCH 1/14 x86/alternative: Zap alternative_ternary()
From: "Borislav Petkov (AMD)" <[email protected]>
Unused.
--- arch/x86/include/asm/alternative.h | 3 --- 1 file changed, 3 deletions(-)
arch/x86/include/asm/alternative.h +0 -3
@@ -271,9 +271,6 @@ static inline int alternatives_text_reserved(void *start, void *end)#define alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ asm_inline volatile(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory") -#define alternative_ternary (oldinstr, ft_flags, newinstr_yes, newinstr_no) \ - asm_inline volatile(ALTERNATIVE_TERNARY(oldinstr, ft_flags, newinstr_yes, newinstr_no) ::: "memory") -/* * Alternative inline assembly with input. *
Signature · 1 line
2.43.0
PATCH 2/14 x86/alternatives: Add nested alternatives macros
In reply to Borislav PetkovFrom: Peter Zijlstra <[email protected]>
Instead of making increasingly complicated ALTERNATIVE_n() implementations, use a nested alternative expression.
The only difference between:
ALTERNATIVE_2(oldinst, newinst1, flag1, newinst2, flag2)
and
ALTERNATIVE(ALTERNATIVE(oldinst, newinst1, flag1),
newinst2, flag2)
is that the outer alternative can add additional padding when the inner alternative is the shorter one, which then results in alt_instr::instrlen being inconsistent.
However, this is easily remedied since the alt_instr entries will be consecutive and it is trivial to compute the max(alt_instr::instrlen) at runtime while patching.
Specifically, after this the ALTERNATIVE_2 macro, after CPP expansion (and manual layout), looks like this:
.macro ALTERNATIVE_2 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2 740: 740: \oldinstr ; 741: .skip -(((744f-743f)-(741b-740b)) > 0) * ((744f-743f)-(741b-740b)),0x90 ; 742: .pushsection .altinstructions,"a" ; altinstr_entry 740b,743f,\ft_flags1,742b-740b,744f-743f ; .popsection ; .pushsection .altinstr_replacement,"ax" ; 743: \newinstr1 ; 744: .popsection ; ; 741: .skip -(((744f-743f)-(741b-740b)) > 0) * ((744f-743f)-(741b-740b)),0x90 ; 742: .pushsection .altinstructions,"a" ; altinstr_entry 740b,743f,\ft_flags2,742b-740b,744f-743f ; .popsection ; .pushsection .altinstr_replacement,"ax" ; 743: \newinstr2 ; 744: .popsection ; .endm
The only label that is ambiguous is 740, however they all reference the same spot, so that doesn't matter.
NOTE: obviously only @oldinstr may be an alternative; making @newinstr an alternative would mean patching .altinstr_replacement which very likely isn't what is intended, also the labels will be confused in that case.
[ bp: Debug an issue where it would match the wrong two insns and
and consider them nested due to the same signed offsets in the .alternative section and use instr_va() to compare the full virtual addresses instead.
- Use new labels to denote that the new, nested alternatives are being used when staring at preprocessed output.
- Use the %c constraint everywhere instead of %P and document the difference for future reference. ]
--- arch/x86/include/asm/alternative.h | 120 ++++++++++++++++++++++++++++- arch/x86/kernel/alternative.c | 20 ++++- tools/objtool/arch/x86/special.c | 23 ++++++ tools/objtool/special.c | 16 ++-- 4 files changed, 167 insertions(+), 12 deletions(-)
arch/x86/include/asm/alternative.h +118 -2
@@ -161,8 +161,13 @@ static inline int alternatives_text_reserved(void *start, void *end)#define alt_end_marker "663"#define alt_slen "662b-661b" +#define n_alt_slen "772b-771b" +#define alt_total_slen alt_end_marker"b-661b" +#define n_alt_total_slen "773b-771b" +#define alt_rlen (num) e_replacement(num)"f-"b_replacement(num)"f" +#define n_alt_rlen "775f-774f"#define OLDINSTR (oldinstr, num) \ "# ALT: oldnstr\n" \
@@ -172,6 +177,14 @@ static inline int alternatives_text_reserved(void *start, void *end) "((" alt_rlen(num) ")-(" alt_slen ")),0x90\n" \ alt_end_marker ":\n" +#define N_OLDINSTR (oldinstr) \ + "# N_ALT: oldinstr\n" \ + "771:\n\t" oldinstr "\n772:\n" \ + "# N_ALT: padding\n" \ + ".skip -(((" n_alt_rlen ")-(" n_alt_slen ")) > 0) * " \ + "((" n_alt_rlen ")-(" n_alt_slen ")),0x90\n" \ + "773:\n" +/* * gas compatible max based on the idea from: * http://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax
@@ -209,10 +222,25 @@ static inline int alternatives_text_reserved(void *start, void *end)" .byte " alt_total_slen"\n" /* source len */ \ " .byte " alt_rlen(num) "\n" /* replacement len */ +#define N_ALTINSTR_ENTRY (ft_flags) \ + ".pushsection .altinstructions,\"a\"\n" \ + " .long 771b - .\n"/* label */ \ +" .long 774f - .\n" /* new instruction */ \ + " .4byte " __stringify(ft_flags) "\n" /* feature + flags */ \ +" .byte " n_alt_total_slen"\n" /* source len */ \ +" .byte " n_alt_rlen"\n" /* replacement len */ \ + ".popsection\n" +#define ALTINSTR_REPLACEMENT (newinstr, num)/* replacement */ \ "# ALT: replacement " #num "\n" \ b_replacement(num)":\n\t" newinstr "\n" e_replacement(num) ":\n" +#define N_ALTINSTR_REPLACEMENT (newinstr)/* replacement */ \ + ".pushsection .altinstr_replacement, \"ax\"\n" \ + "# N_ALT: replacement\n" \ + "774:\n\t" newinstr "\n775:\n" \ + ".popsection\n" +/* alternative assembly primitive: */ #define ALTERNATIVE (oldinstr, newinstr, ft_flags) \ OLDINSTR(oldinstr, 1) \
@@ -223,6 +251,12 @@ static inline int alternatives_text_reserved(void *start, void *end) ALTINSTR_REPLACEMENT(newinstr, 1) \".popsection\n" +/* Nested alternatives macro variant */ +#define N_ALTERNATIVE (oldinstr, newinstr, ft_flags) \ + N_OLDINSTR(oldinstr) \ + N_ALTINSTR_ENTRY(ft_flags) \ + N_ALTINSTR_REPLACEMENT(newinstr) +#define ALTERNATIVE_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ OLDINSTR_2(oldinstr, 1, 2) \ ".pushsection .altinstructions,\"a\"\n" \
@@ -234,11 +268,21 @@ static inline int alternatives_text_reserved(void *start, void *end) ALTINSTR_REPLACEMENT(newinstr2, 2) \ ".popsection\n" +#define N_ALTERNATIVE_2 (oldinst, newinst1, flag1, newinst2, flag2) \ + N_ALTERNATIVE(N_ALTERNATIVE(oldinst, newinst1, flag1), \ + newinst2, flag2) +/* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ #define ALTERNATIVE_TERNARY (oldinstr, ft_flags, newinstr_yes, newinstr_no) \ ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ newinstr_yes, ft_flags) +/* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ +#define N_ALTERNATIVE_TERNARY (oldinstr, ft_flags, newinstr_yes, newinstr_no) \ + N_ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ + newinstr_yes, ft_flags) + +#define ALTERNATIVE_3 (oldinsn, newinsn1, ft_flags1, newinsn2, ft_flags2, \ newinsn3, ft_flags3) \ OLDINSTR_3(oldinsn, 1, 2, 3) \
@@ -253,6 +297,12 @@ static inline int alternatives_text_reserved(void *start, void *end) ALTINSTR_REPLACEMENT(newinsn3, 3) \ ".popsection\n" + +#define N_ALTERNATIVE_3 (oldinst, newinst1, flag1, newinst2, flag2, \ + newinst3, flag3) \ + N_ALTERNATIVE(N_ALTERNATIVE_2(oldinst, newinst1, flag1, newinst2, flag2), \ + newinst3, flag3) +/* * Alternative instructions for different CPU types or capabilities. *
@@ -271,6 +321,12 @@ static inline int alternatives_text_reserved(void *start, void *end)#define alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ asm_inline volatile(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory") +#define n_alternative (oldinstr, newinstr, ft_flags) \ + asm_inline volatile (N_ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory") + +#define n_alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ + asm_inline volatile(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory") +/* * Alternative inline assembly with input. *
@@ -288,11 +344,32 @@ static inline int alternatives_text_reserved(void *start, void *end) asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, ft_flags) \ : output : "i" (0), ## input) -/* Like alternative_io, but for replacing a direct call with another one. */ +#define n_alternative_io(oldinstr, newinstr, ft_flags, output, input...) \ + asm_inline volatile (N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + : output : "i" (0), ## input) + +/* +* Like alternative_io, but for replacing a direct call with another one. +* +* Use the %c operand modifier which is the generic way to print a bare +* constant expression with all syntax-specific punctuation omitted. %P +* is the x86-specific variant which can handle constants too, for +* historical reasons, but it should be used primarily for PIC +* references: i.e., if used for a function, it would add the PLT +* suffix. +*/ #define alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ asm_inline volatile (ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input) +#define n_alternative_input(oldinstr, newinstr, ft_flags, input...) \ + asm_inline volatile (N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + : : "i" (0), ## input) + +#define n_alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ + asm_inline volatile (N_ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ + : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input) +/* * Like alternative_call, but there are two features and respective functions. * If CPU has feature2, function2 is used.
@@ -307,6 +384,15 @@ static inline int alternatives_text_reserved(void *start, void *end) : [old] "i" (oldfunc), [new1] "i" (newfunc1), \ [new2] "i" (newfunc2), ## input) +#define n_alternative_call_2 (oldfunc, newfunc1, ft_flags1, newfunc2, ft_flags2, \ + output, input...) \ + asm_inline volatile (N_ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1,\ + "call %c[new2]", ft_flags2) \ + : output, ASM_CALL_CONSTRAINT \ + : [old] "i" (oldfunc), [new1] "i" (newfunc1), \ + [new2] "i" (newfunc2), ## input) + +/* * use this macro(s) if you need more than one output parameter * in alternative_io
@@ -403,6 +489,27 @@ void nop_func(void); .popsection .endm +#define __N_ALTERNATIVE (oldinst, newinst, flag) \ +740: \ + oldinst ; \ +741: \ + .skip -(((744f-743f)-(741b-740b)) > 0) * ((744f-743f)-(741b-740b)),0x90 ;\ +742: \ + .pushsection .altinstructions,"a" ; \ + altinstr_entry 740b,743f,flag,742b-740b,744f-743f ; \ + .popsection ; \ + .pushsection .altinstr_replacement,"ax" ; \ +743: \ + newinst ; \ +744: \ + .popsection ; + + +.macro N_ALTERNATIVE oldinstr, newinstr, ft_flags + __N_ALTERNATIVE(\oldinstr, \newinstr, \ft_flags) +.endm + +#define old_len 141b-140b#define new_len1 144f-143f#define new_len2 145f-144f
@@ -417,7 +524,6 @@ void nop_func(void);#define alt_max_2 (a, b) ((a) ^ (((a) ^ (b)) & -(-((a) < (b)))))#define alt_max_3 (a, b, c) (alt_max_2(alt_max_2(a, b), c)) -/* * Same as ALTERNATIVE macro above but for two alternatives. If CPU * has @feature1, it replaces @oldinstr with @newinstr1. If CPU has
@@ -445,6 +551,11 @@ void nop_func(void); .popsection .endm +.macro N_ALTERNATIVE_2 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2 + __N_ALTERNATIVE(__N_ALTERNATIVE(\oldinstr, \newinstr1, \ft_flags1), + \newinstr2, \ft_flags2) +.endm + .macro ALTERNATIVE_3 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, newinstr3, ft_flags3 140: \oldinstr
@@ -470,6 +581,11 @@ void nop_func(void); .popsection .endm +.macro N_ALTERNATIVE_3 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, newinstr3, ft_flags3 + __N_ALTERNATIVE(N_ALTERNATIVE_2(\oldinstr, \newinstr1, \ft_flags1, \newinstr2, \ft_flags2), + \newinstr3, \ft_flags3) +.endm + /* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ #define ALTERNATIVE_TERNARY(oldinstr, ft_flags, newinstr_yes, newinstr_no) \ ALTERNATIVE_2 oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \
arch/x86/kernel/alternative.c +18 -2
@@ -432,6 +432,11 @@ static int alt_replace_call(u8 *instr, u8 *insn_buff, struct alt_instr *a)return 5 ; } +static inline u8 *instr_va (struct alt_instr *i) +{ +return (u8 *)&i->instr_offset + i->instr_offset; +} +/* * Replace instructions with better alternatives for this CPU type. This runs * before SMP is initialized to avoid SMP problems with self modifying code.
@@ -447,7 +452,7 @@ void __init_or_module noinline apply_alternatives(struct alt_instr *start, {u8 insn_buff[MAX_PATCH_LEN ];u8 *instr, *replacement; -struct alt_instr *a; +struct alt_instr *a, *b;DPRINTK (ALT ,"alt table %px, -> %px" , start, end);
@@ -473,7 +478,18 @@ void __init_or_module noinline apply_alternatives(struct alt_instr *start,for (a = start; a < end; a++) {int insn_buff_sz =0 ; - instr = (u8 *)&a->instr_offset + a->instr_offset; +/* +* In case of nested ALTERNATIVE()s the outer alternative might +* add more padding. To ensure consistent patching find the max +* padding for all alt_instr entries for this site (nested +* alternatives result in consecutive entries). +*/ +for (b = a+1 ; b < end &&instr_va (b) ==instr_va (a); b++) { +u8 len =max (a->instrlen, b->instrlen); + a->instrlen = b->instrlen = len; + } + + instr =instr_va (a); replacement = (u8 *)&a->repl_offset + a->repl_offset;BUG_ON (a->instrlen >sizeof (insn_buff));BUG_ON (a->cpuid >= (NCAPINTS +NBUGINTS ) *32 );
tools/objtool/arch/x86/special.c +23 -0
@@ -9,6 +9,29 @@void arch_handle_alternative (unsigned short feature,struct special_alt *alt) { +static struct special_alt *group, *prev; + +/* +* Recompute orig_len for nested ALTERNATIVE()s. +*/ +if (group && group->orig_sec == alt->orig_sec && + group->orig_off == alt->orig_off) { + +struct special_alt *iter = group; +for (;;) { +unsigned int len =max (iter->orig_len, alt->orig_len); + iter->orig_len = alt->orig_len = len; + +if (iter == prev) +break ; + + iter =list_next_entry (iter, list); + } + + }else group = alt; + + prev = alt; +switch (feature) {case X86_FEATURE_SMAP :/*
tools/objtool/special.c +8 -8
@@ -84,6 +84,14 @@ static int get_alt_entry(struct elf *elf, const struct special_entry *entry, entry->new_len); } + orig_reloc =find_reloc_by_dest (elf, sec, offset + entry->orig); +if (!orig_reloc) { +WARN_FUNC ("can't find orig reloc" , sec, offset + entry->orig); +return -1 ; + } + +reloc_to_sec_off (orig_reloc, &alt->orig_sec, &alt->orig_off); +if (entry->feature) {unsigned short feature;
@@ -94,14 +102,6 @@ static int get_alt_entry(struct elf *elf, const struct special_entry *entry,arch_handle_alternative (feature, alt); } - orig_reloc =find_reloc_by_dest (elf, sec, offset + entry->orig); -if (!orig_reloc) { -WARN_FUNC ("can't find orig reloc" , sec, offset + entry->orig); -return -1 ; - } - -reloc_to_sec_off (orig_reloc, &alt->orig_sec, &alt->orig_off); -if (!entry->group || alt->new_len) { new_reloc =find_reloc_by_dest (elf, sec, offset + entry->new);if (!new_reloc) {
Signature · 1 line
2.43.0
PATCH 3/14 x86/alternative: Convert alternative()
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
Split conversion deliberately into minimal pieces to ease bisection because debugging alternatives is a nightmare.
--- arch/x86/include/asm/alternative.h | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
arch/x86/include/asm/alternative.h +1 -4
@@ -316,14 +316,11 @@ static inline int alternatives_text_reserved(void *start, void *end)* without volatile and memory clobber. */ #define alternative (oldinstr, newinstr, ft_flags) \ - asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory") + asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory")#define alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ asm_inline volatile(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory") -#define n_alternative (oldinstr, newinstr, ft_flags) \ - asm_inline volatile (N_ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory") -#define n_alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ asm_inline volatile(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory")
Signature · 1 line
2.43.0
PATCH 4/14 x86/alternative: Convert alternative_2()
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
--- arch/x86/include/asm/alternative.h | 3 --- 1 file changed, 3 deletions(-)
arch/x86/include/asm/alternative.h +0 -3
@@ -319,9 +319,6 @@ static inline int alternatives_text_reserved(void *start, void *end) asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory")#define alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ - asm_inline volatile(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory") - -#define n_alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ asm_inline volatile(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory")/*
Signature · 1 line
2.43.0
PATCH 5/14 x86/alternative: Convert alternative_input()
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
--- arch/x86/include/asm/alternative.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
arch/x86/include/asm/alternative.h +1 -5
@@ -330,7 +330,7 @@ static inline int alternatives_text_reserved(void *start, void *end)* Leaving an unused argument 0 to keep API compatibility. */ #define alternative_input(oldinstr, newinstr, ft_flags, input...) \ - asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ : : "i" (0), ## input)/* Like alternative_input, but with a single output argument */
@@ -356,10 +356,6 @@ static inline int alternatives_text_reserved(void *start, void *end) asm_inline volatile (ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input) -#define n_alternative_input(oldinstr, newinstr, ft_flags, input...) \ - asm_inline volatile (N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ - : : "i" (0), ## input) - #define n_alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ asm_inline volatile (N_ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input)
Signature · 1 line
2.43.0
PATCH 6/14 x86/alternative: Convert alternative_io()
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
--- arch/x86/include/asm/alternative.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
arch/x86/include/asm/alternative.h +1 -5
@@ -335,11 +335,7 @@ static inline int alternatives_text_reserved(void *start, void *end)/* Like alternative_input, but with a single output argument */ #define alternative_io(oldinstr, newinstr, ft_flags, output, input...) \ - asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, ft_flags) \ - : output : "i" (0), ## input) - -#define n_alternative_io(oldinstr, newinstr, ft_flags, output, input...) \ - asm_inline volatile (N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ : output : "i" (0), ## input)/*
Signature · 1 line
2.43.0
PATCH 7/14 x86/alternative: Convert alternative_call()
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
--- arch/x86/include/asm/alternative.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
arch/x86/include/asm/alternative.h +1 -5
@@ -349,11 +349,7 @@ static inline int alternatives_text_reserved(void *start, void *end)* suffix. */ #define alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ - asm_inline volatile (ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ - : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input) - -#define n_alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ - asm_inline volatile (N_ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ + asm_inline volatile(N_ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input)/*
Signature · 1 line
2.43.0
PATCH 8/14 x86/alternative: Convert alternative_call_2()
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
--- arch/x86/include/asm/alternative.h | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-)
arch/x86/include/asm/alternative.h +6 -15
@@ -358,23 +358,14 @@ static inline int alternatives_text_reserved(void *start, void *end)* Otherwise, if CPU has feature1, function1 is used. * Otherwise, old function is used. */ -#define alternative_call_2 (oldfunc, newfunc1, ft_flags1, newfunc2, ft_flags2, \ - output, input...) \ - asm_inline volatile (ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1, \ - "call %c[new2]", ft_flags2) \ - : output, ASM_CALL_CONSTRAINT \ - : [old] "i" (oldfunc), [new1] "i" (newfunc1), \ +#define alternative_call_2 (oldfunc, newfunc1, ft_flags1, newfunc2, ft_flags2, \ + output, input...) \ + asm_inline volatile(N_ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1, \ + "call %c[new2]", ft_flags2) \ + : output, ASM_CALL_CONSTRAINT \ + : [old] "i" (oldfunc), [new1] "i" (newfunc1), \ [new2] "i" (newfunc2), ## input) -#define n_alternative_call_2 (oldfunc, newfunc1, ft_flags1, newfunc2, ft_flags2, \ - output, input...) \ - asm_inline volatile (N_ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1,\ - "call %c[new2]", ft_flags2) \ - : output, ASM_CALL_CONSTRAINT \ - : [old] "i" (oldfunc), [new1] "i" (newfunc1), \ - [new2] "i" (newfunc2), ## input) - -/* * use this macro(s) if you need more than one output parameter * in alternative_io
Signature · 1 line
2.43.0
PATCH 9/14 x86/alternative: Convert ALTERNATIVE_TERNARY()
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
The C macro.
--- arch/x86/include/asm/alternative.h | 6 ------ 1 file changed, 6 deletions(-)
arch/x86/include/asm/alternative.h +0 -6
@@ -274,15 +274,9 @@ static inline int alternatives_text_reserved(void *start, void *end)/* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ #define ALTERNATIVE_TERNARY (oldinstr, ft_flags, newinstr_yes, newinstr_no) \ - ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ - newinstr_yes, ft_flags) - -/* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ -#define N_ALTERNATIVE_TERNARY (oldinstr, ft_flags, newinstr_yes, newinstr_no) \ N_ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ newinstr_yes, ft_flags) -#define ALTERNATIVE_3 (oldinsn, newinsn1, ft_flags1, newinsn2, ft_flags2, \ newinsn3, ft_flags3) \ OLDINSTR_3(oldinsn, 1, 2, 3) \
Signature · 1 line
2.43.0
PATCH 10/14 x86/alternative: Convert ALTERNATIVE_3()
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
Zap the hack of using an ALTERNATIVE_3() internal label, as suggested by bgerst:
https://lore.kernel.org/r/CAMzpN2i4oJ-Dv0qO46Fd-DxNv5z9=x%2BvO%[email protected]
in favor of a label local to this macro only, as it should be done.
--- arch/x86/include/asm/alternative.h | 24 ++++-------------------- arch/x86/kernel/fpu/xstate.h | 14 +++++--------- 2 files changed, 9 insertions(+), 29 deletions(-)
arch/x86/include/asm/alternative.h +4 -20
@@ -277,26 +277,10 @@ static inline int alternatives_text_reserved(void *start, void *end)N_ALTERNATIVE_2 (oldinstr, newinstr_no,X86_FEATURE_ALWAYS , \ newinstr_yes, ft_flags) -#define ALTERNATIVE_3 (oldinsn, newinsn1, ft_flags1, newinsn2, ft_flags2, \ - newinsn3, ft_flags3) \ - OLDINSTR_3(oldinsn, 1, 2, 3) \ - ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY(ft_flags1, 1) \ - ALTINSTR_ENTRY(ft_flags2, 2) \ - ALTINSTR_ENTRY(ft_flags3, 3) \ - ".popsection\n" \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - ALTINSTR_REPLACEMENT(newinsn1, 1) \ - ALTINSTR_REPLACEMENT(newinsn2, 2) \ - ALTINSTR_REPLACEMENT(newinsn3, 3) \ - ".popsection\n" - - -#define N_ALTERNATIVE_3 (oldinst, newinst1, flag1, newinst2, flag2, \ - newinst3, flag3) \ - N_ALTERNATIVE(N_ALTERNATIVE_2(oldinst, newinst1, flag1, newinst2, flag2), \ - newinst3, flag3) - +#define ALTERNATIVE_3 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, \ + newinstr3, ft_flags3) \ + N_ALTERNATIVE(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2), \ + newinstr3, ft_flags3)/* * Alternative instructions for different CPU types or capabilities. *
arch/x86/kernel/fpu/xstate.h +5 -9
@@ -106,21 +106,17 @@ static inline u64 xfeatures_mask_independent(void)* Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT * supports modified optimization which is not supported by XSAVE. * -* We use XSAVE as a fallback. -* -* The 661 label is defined in the ALTERNATIVE* macros as the address of the -* original instruction which gets replaced. We need to use it here as the -* address of the instruction where we might get an exception at. +* Use XSAVE as a fallback. */ #define XSTATE_XSAVE (st, lmask, hmask, err) \ - asm volatile(ALTERNATIVE_3(XSAVE, \ + asm volatile("1: " ALTERNATIVE_3(XSAVE, \ XSAVEOPT, X86_FEATURE_XSAVEOPT, \ XSAVEC, X86_FEATURE_XSAVEC, \ XSAVES, X86_FEATURE_XSAVES) \ "\n" \ "xor %[err], %[err]\n" \ "3:\n" \ - _ASM_EXTABLE_TYPE_REG(661b, 3b, EX_TYPE_EFAULT_REG, %[err]) \ + _ASM_EXTABLE_TYPE_REG(1b, 3b, EX_TYPE_EFAULT_REG, %[err]) \ : [err] "=r" (err) \ : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \ : "memory")
@@ -130,11 +126,11 @@ static inline u64 xfeatures_mask_independent(void)* XSAVE area format. */ #define XSTATE_XRESTORE (st, lmask, hmask) \ - asm volatile(ALTERNATIVE(XRSTOR, \ + asm volatile("1: " ALTERNATIVE(XRSTOR, \ XRSTORS, X86_FEATURE_XSAVES) \ "\n" \ "3:\n" \ - _ASM_EXTABLE_TYPE(661b, 3b, EX_TYPE_FPU_RESTORE) \ + _ASM_EXTABLE_TYPE(1b, 3b, EX_TYPE_FPU_RESTORE) \ : \ : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \ : "memory")
Signature · 1 line
2.43.0
PATCH 11/14 x86/alternative: Convert the asm ALTERNATIVE() macro
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
--- arch/x86/include/asm/alternative.h | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-)
arch/x86/include/asm/alternative.h +1 -21
@@ -422,24 +422,6 @@ void nop_func(void);* @newinstr. ".skip" directive takes care of proper instruction padding * in case @newinstr is longer than @oldinstr. */ -.macro ALTERNATIVE oldinstr, newinstr, ft_flags -140: - \oldinstr -141: - .skip -(((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstr_entry 140b,143f,\ft_flags,142b-140b,144f-143f - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr -144: - .popsection -.endm -#define __N_ALTERNATIVE (oldinst, newinst, flag) \ 740: \ oldinst ; \
@@ -455,12 +437,10 @@ void nop_func(void); 744: \ .popsection ; - -.macro N_ALTERNATIVE oldinstr, newinstr, ft_flags +.macro ALTERNATIVE oldinstr, newinstr, ft_flags __N_ALTERNATIVE(\oldinstr, \newinstr, \ft_flags) .endm -#define old_len 141b-140b#define new_len1 144f-143f#define new_len2 145f-144f
Signature · 1 line
2.43.0
PATCH 12/14 x86/alternative: Convert the asm ALTERNATIVE_2() macro
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
--- arch/x86/include/asm/alternative.h | 22 ---------------------- 1 file changed, 22 deletions(-)
arch/x86/include/asm/alternative.h +0 -22
@@ -461,28 +461,6 @@ void nop_func(void);* @feature2, it replaces @oldinstr with @feature2. */ .macro ALTERNATIVE_2 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2 -140: - \oldinstr -141: - .skip -((alt_max_2(new_len1, new_len2) - (old_len)) > 0) * \ - (alt_max_2(new_len1, new_len2) - (old_len)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstr_entry 140b,143f,\ft_flags1,142b-140b,144f-143f - altinstr_entry 140b,144f,\ft_flags2,142b-140b,145f-144f - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr1 -144: - \newinstr2 -145: - .popsection -.endm - -.macro N_ALTERNATIVE_2 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2 __N_ALTERNATIVE(__N_ALTERNATIVE(\oldinstr, \newinstr1, \ft_flags1), \newinstr2, \ft_flags2) .endm
Signature · 1 line
2.43.0
PATCH 13/14 x86/alternative: Convert the asm ALTERNATIVE_3() macro
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
--- arch/x86/include/asm/alternative.h | 25 ------------------------- 1 file changed, 25 deletions(-)
arch/x86/include/asm/alternative.h +0 -25
@@ -466,31 +466,6 @@ void nop_func(void); .endm .macro ALTERNATIVE_3 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, newinstr3, ft_flags3 -140: - \oldinstr -141: - .skip -((alt_max_3(new_len1, new_len2, new_len3) - (old_len)) > 0) * \ - (alt_max_3(new_len1, new_len2, new_len3) - (old_len)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstr_entry 140b,143f,\ft_flags1,142b-140b,144f-143f - altinstr_entry 140b,144f,\ft_flags2,142b-140b,145f-144f - altinstr_entry 140b,145f,\ft_flags3,142b-140b,146f-145f - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr1 -144: - \newinstr2 -145: - \newinstr3 -146: - .popsection -.endm - -.macro N_ALTERNATIVE_3 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, newinstr3, ft_flags3 __N_ALTERNATIVE(N_ALTERNATIVE_2(\oldinstr, \newinstr1, \ft_flags1, \newinstr2, \ft_flags2), \newinstr3, \ft_flags3) .endm
Signature · 1 line
2.43.0
PATCH 14/14 x86/alternative: Replace the old macros
In reply to Borislav PetkovFrom: "Borislav Petkov (AMD)" <[email protected]>
Now that the new macros have been gradually put in place, replace the old ones. Leave the new label numbers starting at 7xx as a hint that the new nested alternatives are being used now.
--- arch/x86/include/asm/alternative.h | 144 +++++++---------------------- 1 file changed, 32 insertions(+), 112 deletions(-)
arch/x86/include/asm/alternative.h +32 -112
@@ -156,131 +156,51 @@ static inline int alternatives_text_reserved(void *start, void *end)#define ALT_CALL_INSTR "call BUG_func" -#define b_replacement (num) "664"#num -#define e_replacement (num) "665"#num +#define alt_slen "772b-771b" +#define alt_total_slen "773b-771b" +#define alt_rlen "775f-774f" -#define alt_end_marker "663" -#define alt_slen "662b-661b" -#define n_alt_slen "772b-771b" - -#define alt_total_slen alt_end_marker"b-661b" -#define n_alt_total_slen "773b-771b" - -#define alt_rlen (num) e_replacement(num)"f-"b_replacement(num)"f" -#define n_alt_rlen "775f-774f" - -#define OLDINSTR (oldinstr, num) \ - "# ALT: oldnstr\n" \ - "661:\n\t" oldinstr "\n662:\n" \ - "# ALT: padding\n" \ - ".skip -(((" alt_rlen(num) ")-(" alt_slen ")) > 0) * " \ - "((" alt_rlen(num) ")-(" alt_slen ")),0x90\n" \ - alt_end_marker ":\n" - -#define N_OLDINSTR (oldinstr) \ - "# N_ALT: oldinstr\n" \ +#define OLDINSTR (oldinstr) \ + "# ALT: oldinstr\n" \ "771:\n\t" oldinstr "\n772:\n" \ - "# N_ALT: padding\n" \ - ".skip -(((" n_alt_rlen ")-(" n_alt_slen ")) > 0) * " \ - "((" n_alt_rlen ")-(" n_alt_slen ")),0x90\n" \ + "# ALT: padding\n" \ + ".skip -(((" alt_rlen ")-(" alt_slen ")) > 0) * " \ + "((" alt_rlen ")-(" alt_slen ")),0x90\n" \ "773:\n" -/* -* gas compatible max based on the idea from: -* http://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax -* -* The additional "-" is needed because gas uses a "true" value of -1. -*/ -#define alt_max_short (a, b) "((" a ") ^ (((" a ") ^ (" b ")) & -(-((" a ") < (" b ")))))" - -/* -* Pad the second replacement alternative with additional NOPs if it is -* additionally longer than the first replacement alternative. -*/ -#define OLDINSTR_2 (oldinstr, num1, num2) \ - "# ALT: oldinstr2\n" \ - "661:\n\t" oldinstr "\n662:\n" \ - "# ALT: padding2\n" \ - ".skip -((" alt_max_short(alt_rlen(num1), alt_rlen(num2)) " - (" alt_slen ")) > 0) * " \ - "(" alt_max_short(alt_rlen(num1), alt_rlen(num2)) " - (" alt_slen ")), 0x90\n" \ - alt_end_marker ":\n" - -#define OLDINSTR_3 (oldinsn, n1, n2, n3) \ - "# ALT: oldinstr3\n" \ - "661:\n\t" oldinsn "\n662:\n" \ - "# ALT: padding3\n" \ - ".skip -((" alt_max_short(alt_max_short(alt_rlen(n1), alt_rlen(n2)), alt_rlen(n3)) \ - " - (" alt_slen ")) > 0) * " \ - "(" alt_max_short(alt_max_short(alt_rlen(n1), alt_rlen(n2)), alt_rlen(n3)) \ - " - (" alt_slen ")), 0x90\n" \ - alt_end_marker ":\n" - -#define ALTINSTR_ENTRY (ft_flags, num) \ - " .long 661b - .\n"/* label */ \ - " .long " b_replacement(num)"f - .\n" /* new instruction */ \ - " .4byte " __stringify(ft_flags) "\n" /* feature + flags */ \ -" .byte " alt_total_slen"\n" /* source len */ \ - " .byte " alt_rlen(num) "\n" /* replacement len */ - -#define N_ALTINSTR_ENTRY (ft_flags) \ +#define ALTINSTR_ENTRY (ft_flags) \ ".pushsection .altinstructions,\"a\"\n" \ " .long 771b - .\n"/* label */ \" .long 774f - .\n" /* new instruction */ \ " .4byte " __stringify(ft_flags) "\n" /* feature + flags */ \ -" .byte " n_alt_total_slen"\n" /* source len */ \ -" .byte " n_alt_rlen"\n" /* replacement len */ \ +" .byte " alt_total_slen"\n" /* source len */ \ +" .byte " alt_rlen"\n" /* replacement len */ \ ".popsection\n" -#define ALTINSTR_REPLACEMENT (newinstr, num)/* replacement */ \ - "# ALT: replacement " #num "\n" \ - b_replacement(num)":\n\t" newinstr "\n" e_replacement(num) ":\n" - -#define N_ALTINSTR_REPLACEMENT (newinstr)/* replacement */ \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - "# N_ALT: replacement\n" \ - "774:\n\t" newinstr "\n775:\n" \ +#define ALTINSTR_REPLACEMENT (newinstr)/* replacement */ \ + ".pushsection .altinstr_replacement, \"ax\"\n" \ + "# ALT: replacement\n" \ + "774:\n\t" newinstr "\n775:\n" \ ".popsection\n"/* alternative assembly primitive: */ #define ALTERNATIVE (oldinstr, newinstr, ft_flags) \ - OLDINSTR(oldinstr, 1) \ - ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY(ft_flags, 1) \ - ".popsection\n" \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - ALTINSTR_REPLACEMENT(newinstr, 1) \ - ".popsection\n" - -/* Nested alternatives macro variant */ -#define N_ALTERNATIVE (oldinstr, newinstr, ft_flags) \ - N_OLDINSTR(oldinstr) \ - N_ALTINSTR_ENTRY(ft_flags) \ - N_ALTINSTR_REPLACEMENT(newinstr) + OLDINSTR(oldinstr) \ + ALTINSTR_ENTRY(ft_flags) \ + ALTINSTR_REPLACEMENT(newinstr)#define ALTERNATIVE_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ - OLDINSTR_2(oldinstr, 1, 2) \ - ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY(ft_flags1, 1) \ - ALTINSTR_ENTRY(ft_flags2, 2) \ - ".popsection\n" \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - ALTINSTR_REPLACEMENT(newinstr1, 1) \ - ALTINSTR_REPLACEMENT(newinstr2, 2) \ - ".popsection\n" - -#define N_ALTERNATIVE_2 (oldinst, newinst1, flag1, newinst2, flag2) \ - N_ALTERNATIVE(N_ALTERNATIVE(oldinst, newinst1, flag1), \ - newinst2, flag2) + ALTERNATIVE(ALTERNATIVE(oldinstr, newinstr1, ft_flags1), newinstr2, ft_flags2)/* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ #define ALTERNATIVE_TERNARY (oldinstr, ft_flags, newinstr_yes, newinstr_no) \ - N_ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ - newinstr_yes, ft_flags) + ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, newinstr_yes, ft_flags)#define ALTERNATIVE_3 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, \ newinstr3, ft_flags3) \ - N_ALTERNATIVE(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2), \ + ALTERNATIVE(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2), \ newinstr3, ft_flags3) +/* * Alternative instructions for different CPU types or capabilities. *
@@ -294,10 +214,10 @@ static inline int alternatives_text_reserved(void *start, void *end)* without volatile and memory clobber. */ #define alternative (oldinstr, newinstr, ft_flags) \ - asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory") + asm_inline volatile(ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory")#define alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ - asm_inline volatile(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory") + asm_inline volatile(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory")/* * Alternative inline assembly with input.
@@ -308,12 +228,12 @@ static inline int alternatives_text_reserved(void *start, void *end)* Leaving an unused argument 0 to keep API compatibility. */ #define alternative_input(oldinstr, newinstr, ft_flags, input...) \ - asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + asm_inline volatile(ALTERNATIVE(oldinstr, newinstr, ft_flags) \ : : "i" (0), ## input)/* Like alternative_input, but with a single output argument */ #define alternative_io(oldinstr, newinstr, ft_flags, output, input...) \ - asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + asm_inline volatile(ALTERNATIVE(oldinstr, newinstr, ft_flags) \ : output : "i" (0), ## input)/*
@@ -327,7 +247,7 @@ static inline int alternatives_text_reserved(void *start, void *end)* suffix. */ #define alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ - asm_inline volatile(N_ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ + asm_inline volatile(ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input)/*
@@ -338,7 +258,7 @@ static inline int alternatives_text_reserved(void *start, void *end)*/ #define alternative_call_2 (oldfunc, newfunc1, ft_flags1, newfunc2, ft_flags2, \ output, input...) \ - asm_inline volatile(N_ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1, \ + asm_inline volatile(ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1, \ "call %c[new2]", ft_flags2) \ : output, ASM_CALL_CONSTRAINT \ : [old] "i" (oldfunc), [new1] "i" (newfunc1), \
@@ -422,7 +342,7 @@ void nop_func(void);* @newinstr. ".skip" directive takes care of proper instruction padding * in case @newinstr is longer than @oldinstr. */ -#define __N_ALTERNATIVE (oldinst, newinst, flag) \ +#define __ALTERNATIVE (oldinst, newinst, flag) \ 740: \ oldinst ; \ 741: \
@@ -438,7 +358,7 @@ void nop_func(void); .popsection ; .macro ALTERNATIVE oldinstr, newinstr, ft_flags - __N_ALTERNATIVE(\oldinstr, \newinstr, \ft_flags) + __ALTERNATIVE(\oldinstr, \newinstr, \ft_flags) .endm#define old_len 141b-140b
@@ -461,12 +381,12 @@ void nop_func(void);* @feature2, it replaces @oldinstr with @feature2. */ .macro ALTERNATIVE_2 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2 - __N_ALTERNATIVE(__N_ALTERNATIVE(\oldinstr, \newinstr1, \ft_flags1), + __ALTERNATIVE(__ALTERNATIVE(\oldinstr, \newinstr1, \ft_flags1), \newinstr2, \ft_flags2) .endm .macro ALTERNATIVE_3 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, newinstr3, ft_flags3 - __N_ALTERNATIVE(N_ALTERNATIVE_2(\oldinstr, \newinstr1, \ft_flags1, \newinstr2, \ft_flags2), + __ALTERNATIVE(ALTERNATIVE_2(\oldinstr, \newinstr1, \ft_flags1, \newinstr2, \ft_flags2), \newinstr3, \ft_flags3) .endm
Signature · 1 line
2.43.0
Peter Zijlstra
maintainerRe: [PATCH v1 14/14] x86/alternative: Replace the old macros
Quoted 5 lines · Borislav Petkov
From: "Borislav Petkov (AMD)" <[email protected]>
Now that the new macros have been gradually put in place, replace the old ones. Leave the new label numbers starting at 7xx as a hint that the new nested alternatives are being used now.
You forgot to kill alt_max_[23]().
Quoted 18 lines · Borislav Petkov
Borislav Petkov (AMD) (13):
x86/alternative: Zapalternative_ternary()x86/alternative: Convertalternative()x86/alternative: Convertalternative_2()x86/alternative: Convertalternative_input()x86/alternative: Convertalternative_io()x86/alternative: Convertalternative_call()x86/alternative: Convertalternative_call_2()x86/alternative: ConvertALTERNATIVE_TERNARY()x86/alternative: ConvertALTERNATIVE_3()x86/alternative: Convert the asmALTERNATIVE()macro x86/alternative: Convert the asmALTERNATIVE_2()macro x86/alternative: Convert the asmALTERNATIVE_3()macro x86/alternative: Replace the old macrosPeter Zijlstra (1):
x86/alternatives: Add nested alternatives macros
Quoted 6 lines
arch/x86/include/asm/alternative.h | 225 +++++++++-------------------- arch/x86/kernel/alternative.c | 20 ++- arch/x86/kernel/fpu/xstate.h | 14 +- tools/objtool/arch/x86/special.c | 23 +++ tools/objtool/special.c | 16 +- 5 files changed, 125 insertions(+), 173 deletions(-)
The whole back and forth with the n_foo things ia weird, but sure, have at.
Borislav Petkov
maintainerQuoted 27 lines · Peter Zijlstra
Quoted 18 lines · Borislav Petkov
Borislav Petkov (AMD) (13):
x86/alternative: Zapalternative_ternary()x86/alternative: Convertalternative()x86/alternative: Convertalternative_2()x86/alternative: Convertalternative_input()x86/alternative: Convertalternative_io()x86/alternative: Convertalternative_call()x86/alternative: Convertalternative_call_2()x86/alternative: ConvertALTERNATIVE_TERNARY()x86/alternative: ConvertALTERNATIVE_3()x86/alternative: Convert the asmALTERNATIVE()macro x86/alternative: Convert the asmALTERNATIVE_2()macro x86/alternative: Convert the asmALTERNATIVE_3()macro x86/alternative: Replace the old macrosPeter Zijlstra (1):
x86/alternatives: Add nested alternatives macrosQuoted 6 lines
arch/x86/include/asm/alternative.h | 225 +++++++++-------------------- arch/x86/kernel/alternative.c | 20 ++- arch/x86/kernel/fpu/xstate.h | 14 +- tools/objtool/arch/x86/special.c | 23 +++ tools/objtool/special.c | 16 +- 5 files changed, 125 insertions(+), 173 deletions(-)
Quoted 4 lines
The whole back and forth with the
n_foothings ia weird, but sure, have at.
Thanks, yeah it is in the 0th mail:
"For ease of bisection, the old macros are converted to the new, nested variants in a step-by-step manner so that in case an issue is encountered during testing, one can pinpoint the place where it fails easier. Because debugging alternatives is a serious pain."
If you remember, your original patch had this issue where it would confuse two back-to-back .altinstr entries for nested because you were comparing only the offsets and they happened to be equal, by l
Now debugging *that* was insane and I managed to find the issue only when I started converting each place one-by-one and booting each one step.
That's why this superficial split is all about. We will appreciate it later. I hope not but if something explodes, bisection will be a lot easier this way.
I hope this makes sense.
Signature · 1 line
Sent from a small device: formatting sucks and brevity is inevitable.
[tip: x86/alternatives] x86/alternative: Convert the asm ALTERNATIVE_2() macro
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: 08a621fcf4a4a4d765315ffe6a28f5d31e8237b2 Gitweb: https://git.kernel.org/tip/08a621fcf4a4a4d765315ffe6a28f5d31e8237b2 Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:59 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:27:14 +02:00
x86/alternative: Convert the asm ALTERNATIVE_2() macro
--- arch/x86/include/asm/alternative.h | 22 ---------------------- 1 file changed, 22 deletions(-)
arch/x86/include/asm/alternative.h +0 -22
@@ -461,28 +461,6 @@ void nop_func(void);* @feature2, it replaces @oldinstr with @feature2. */ .macro ALTERNATIVE_2 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2 -140: - \oldinstr -141: - .skip -((alt_max_2(new_len1, new_len2) - (old_len)) > 0) * \ - (alt_max_2(new_len1, new_len2) - (old_len)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstr_entry 140b,143f,\ft_flags1,142b-140b,144f-143f - altinstr_entry 140b,144f,\ft_flags2,142b-140b,145f-144f - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr1 -144: - \newinstr2 -145: - .popsection -.endm - -.macro N_ALTERNATIVE_2 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2 __N_ALTERNATIVE(__N_ALTERNATIVE(\oldinstr, \newinstr1, \ft_flags1), \newinstr2, \ft_flags2) .endm
[tip: x86/alternatives] x86/alternative: Convert the asm ALTERNATIVE() macro
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: a6c7a6a18b10a4ac0913e7abdbdce928a2601bdb Gitweb: https://git.kernel.org/tip/a6c7a6a18b10a4ac0913e7abdbdce928a2601bdb Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:58 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:25:00 +02:00
x86/alternative: Convert the asm ALTERNATIVE() macro
--- arch/x86/include/asm/alternative.h | 22 +--------------------- 1 file changed, 1 insertion(+), 21 deletions(-)
arch/x86/include/asm/alternative.h +1 -21
@@ -422,24 +422,6 @@ void nop_func(void);* @newinstr. ".skip" directive takes care of proper instruction padding * in case @newinstr is longer than @oldinstr. */ -.macro ALTERNATIVE oldinstr, newinstr, ft_flags -140: - \oldinstr -141: - .skip -(((144f-143f)-(141b-140b)) > 0) * ((144f-143f)-(141b-140b)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstr_entry 140b,143f,\ft_flags,142b-140b,144f-143f - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr -144: - .popsection -.endm -#define __N_ALTERNATIVE (oldinst, newinst, flag) \ 740: \ oldinst ; \
@@ -455,12 +437,10 @@ void nop_func(void); 744: \ .popsection ; - -.macro N_ALTERNATIVE oldinstr, newinstr, ft_flags +.macro ALTERNATIVE oldinstr, newinstr, ft_flags __N_ALTERNATIVE(\oldinstr, \newinstr, \ft_flags) .endm -#define old_len 141b-140b#define new_len1 144f-143f#define new_len2 145f-144f
[tip: x86/alternatives] x86/alternative: Convert the asm ALTERNATIVE_3() macro
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: 1a6ade825079243f08846870561aca0e1fdfb803 Gitweb: https://git.kernel.org/tip/1a6ade825079243f08846870561aca0e1fdfb803 Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:17:00 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:29:26 +02:00
x86/alternative: Convert the asm ALTERNATIVE_3() macro
--- arch/x86/include/asm/alternative.h | 25 ------------------------- 1 file changed, 25 deletions(-)
arch/x86/include/asm/alternative.h +0 -25
@@ -466,31 +466,6 @@ void nop_func(void); .endm .macro ALTERNATIVE_3 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, newinstr3, ft_flags3 -140: - \oldinstr -141: - .skip -((alt_max_3(new_len1, new_len2, new_len3) - (old_len)) > 0) * \ - (alt_max_3(new_len1, new_len2, new_len3) - (old_len)),0x90 -142: - - .pushsection .altinstructions,"a" - altinstr_entry 140b,143f,\ft_flags1,142b-140b,144f-143f - altinstr_entry 140b,144f,\ft_flags2,142b-140b,145f-144f - altinstr_entry 140b,145f,\ft_flags3,142b-140b,146f-145f - .popsection - - .pushsection .altinstr_replacement,"ax" -143: - \newinstr1 -144: - \newinstr2 -145: - \newinstr3 -146: - .popsection -.endm - -.macro N_ALTERNATIVE_3 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, newinstr3, ft_flags3 __N_ALTERNATIVE(N_ALTERNATIVE_2(\oldinstr, \newinstr1, \ft_flags1, \newinstr2, \ft_flags2), \newinstr3, \ft_flags3) .endm
[tip: x86/alternatives] x86/alternative: Replace the old macros
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: f776e41fdcc4141876ef6f297318ab04c2382eb7 Gitweb: https://git.kernel.org/tip/f776e41fdcc4141876ef6f297318ab04c2382eb7 Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:17:01 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:33:23 +02:00
x86/alternative: Replace the old macros
Now that the new macros have been gradually put in place, replace the old ones. Leave the new label numbers starting at 7xx as a hint that the new nested alternatives are being used now.
--- arch/x86/include/asm/alternative.h | 153 +++++----------------------- 1 file changed, 32 insertions(+), 121 deletions(-)
arch/x86/include/asm/alternative.h +32 -121
@@ -156,131 +156,51 @@ static inline int alternatives_text_reserved(void *start, void *end)#define ALT_CALL_INSTR "call BUG_func" -#define b_replacement (num) "664"#num -#define e_replacement (num) "665"#num +#define alt_slen "772b-771b" +#define alt_total_slen "773b-771b" +#define alt_rlen "775f-774f" -#define alt_end_marker "663" -#define alt_slen "662b-661b" -#define n_alt_slen "772b-771b" - -#define alt_total_slen alt_end_marker"b-661b" -#define n_alt_total_slen "773b-771b" - -#define alt_rlen (num) e_replacement(num)"f-"b_replacement(num)"f" -#define n_alt_rlen "775f-774f" - -#define OLDINSTR (oldinstr, num) \ - "# ALT: oldnstr\n" \ - "661:\n\t" oldinstr "\n662:\n" \ - "# ALT: padding\n" \ - ".skip -(((" alt_rlen(num) ")-(" alt_slen ")) > 0) * " \ - "((" alt_rlen(num) ")-(" alt_slen ")),0x90\n" \ - alt_end_marker ":\n" - -#define N_OLDINSTR (oldinstr) \ - "# N_ALT: oldinstr\n" \ +#define OLDINSTR (oldinstr) \ + "# ALT: oldinstr\n" \ "771:\n\t" oldinstr "\n772:\n" \ - "# N_ALT: padding\n" \ - ".skip -(((" n_alt_rlen ")-(" n_alt_slen ")) > 0) * " \ - "((" n_alt_rlen ")-(" n_alt_slen ")),0x90\n" \ + "# ALT: padding\n" \ + ".skip -(((" alt_rlen ")-(" alt_slen ")) > 0) * " \ + "((" alt_rlen ")-(" alt_slen ")),0x90\n" \ "773:\n" -/* -* gas compatible max based on the idea from: -* http://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax -* -* The additional "-" is needed because gas uses a "true" value of -1. -*/ -#define alt_max_short (a, b) "((" a ") ^ (((" a ") ^ (" b ")) & -(-((" a ") < (" b ")))))" - -/* -* Pad the second replacement alternative with additional NOPs if it is -* additionally longer than the first replacement alternative. -*/ -#define OLDINSTR_2 (oldinstr, num1, num2) \ - "# ALT: oldinstr2\n" \ - "661:\n\t" oldinstr "\n662:\n" \ - "# ALT: padding2\n" \ - ".skip -((" alt_max_short(alt_rlen(num1), alt_rlen(num2)) " - (" alt_slen ")) > 0) * " \ - "(" alt_max_short(alt_rlen(num1), alt_rlen(num2)) " - (" alt_slen ")), 0x90\n" \ - alt_end_marker ":\n" - -#define OLDINSTR_3 (oldinsn, n1, n2, n3) \ - "# ALT: oldinstr3\n" \ - "661:\n\t" oldinsn "\n662:\n" \ - "# ALT: padding3\n" \ - ".skip -((" alt_max_short(alt_max_short(alt_rlen(n1), alt_rlen(n2)), alt_rlen(n3)) \ - " - (" alt_slen ")) > 0) * " \ - "(" alt_max_short(alt_max_short(alt_rlen(n1), alt_rlen(n2)), alt_rlen(n3)) \ - " - (" alt_slen ")), 0x90\n" \ - alt_end_marker ":\n" - -#define ALTINSTR_ENTRY (ft_flags, num) \ - " .long 661b - .\n"/* label */ \ - " .long " b_replacement(num)"f - .\n" /* new instruction */ \ - " .4byte " __stringify(ft_flags) "\n" /* feature + flags */ \ -" .byte " alt_total_slen"\n" /* source len */ \ - " .byte " alt_rlen(num) "\n" /* replacement len */ - -#define N_ALTINSTR_ENTRY (ft_flags) \ +#define ALTINSTR_ENTRY (ft_flags) \ ".pushsection .altinstructions,\"a\"\n" \ " .long 771b - .\n"/* label */ \" .long 774f - .\n" /* new instruction */ \ " .4byte " __stringify(ft_flags) "\n" /* feature + flags */ \ -" .byte " n_alt_total_slen"\n" /* source len */ \ -" .byte " n_alt_rlen"\n" /* replacement len */ \ +" .byte " alt_total_slen"\n" /* source len */ \ +" .byte " alt_rlen"\n" /* replacement len */ \ ".popsection\n" -#define ALTINSTR_REPLACEMENT (newinstr, num)/* replacement */ \ - "# ALT: replacement " #num "\n" \ - b_replacement(num)":\n\t" newinstr "\n" e_replacement(num) ":\n" - -#define N_ALTINSTR_REPLACEMENT (newinstr)/* replacement */ \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - "# N_ALT: replacement\n" \ - "774:\n\t" newinstr "\n775:\n" \ +#define ALTINSTR_REPLACEMENT (newinstr)/* replacement */ \ + ".pushsection .altinstr_replacement, \"ax\"\n" \ + "# ALT: replacement\n" \ + "774:\n\t" newinstr "\n775:\n" \ ".popsection\n"/* alternative assembly primitive: */ #define ALTERNATIVE (oldinstr, newinstr, ft_flags) \ - OLDINSTR(oldinstr, 1) \ - ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY(ft_flags, 1) \ - ".popsection\n" \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - ALTINSTR_REPLACEMENT(newinstr, 1) \ - ".popsection\n" - -/* Nested alternatives macro variant */ -#define N_ALTERNATIVE (oldinstr, newinstr, ft_flags) \ - N_OLDINSTR(oldinstr) \ - N_ALTINSTR_ENTRY(ft_flags) \ - N_ALTINSTR_REPLACEMENT(newinstr) + OLDINSTR(oldinstr) \ + ALTINSTR_ENTRY(ft_flags) \ + ALTINSTR_REPLACEMENT(newinstr)#define ALTERNATIVE_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ - OLDINSTR_2(oldinstr, 1, 2) \ - ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY(ft_flags1, 1) \ - ALTINSTR_ENTRY(ft_flags2, 2) \ - ".popsection\n" \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - ALTINSTR_REPLACEMENT(newinstr1, 1) \ - ALTINSTR_REPLACEMENT(newinstr2, 2) \ - ".popsection\n" - -#define N_ALTERNATIVE_2 (oldinst, newinst1, flag1, newinst2, flag2) \ - N_ALTERNATIVE(N_ALTERNATIVE(oldinst, newinst1, flag1), \ - newinst2, flag2) + ALTERNATIVE(ALTERNATIVE(oldinstr, newinstr1, ft_flags1), newinstr2, ft_flags2)/* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ #define ALTERNATIVE_TERNARY (oldinstr, ft_flags, newinstr_yes, newinstr_no) \ - N_ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ - newinstr_yes, ft_flags) + ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, newinstr_yes, ft_flags)#define ALTERNATIVE_3 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, \ newinstr3, ft_flags3) \ - N_ALTERNATIVE(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2), \ + ALTERNATIVE(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2), \ newinstr3, ft_flags3) +/* * Alternative instructions for different CPU types or capabilities. *
@@ -294,10 +214,10 @@ static inline int alternatives_text_reserved(void *start, void *end)* without volatile and memory clobber. */ #define alternative (oldinstr, newinstr, ft_flags) \ - asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory") + asm_inline volatile(ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory")#define alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ - asm_inline volatile(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory") + asm_inline volatile(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory")/* * Alternative inline assembly with input.
@@ -308,12 +228,12 @@ static inline int alternatives_text_reserved(void *start, void *end)* Leaving an unused argument 0 to keep API compatibility. */ #define alternative_input(oldinstr, newinstr, ft_flags, input...) \ - asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + asm_inline volatile(ALTERNATIVE(oldinstr, newinstr, ft_flags) \ : : "i" (0), ## input)/* Like alternative_input, but with a single output argument */ #define alternative_io(oldinstr, newinstr, ft_flags, output, input...) \ - asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + asm_inline volatile(ALTERNATIVE(oldinstr, newinstr, ft_flags) \ : output : "i" (0), ## input)/*
@@ -327,7 +247,7 @@ static inline int alternatives_text_reserved(void *start, void *end)* suffix. */ #define alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ - asm_inline volatile(N_ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ + asm_inline volatile(ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input)/*
@@ -338,7 +258,7 @@ static inline int alternatives_text_reserved(void *start, void *end)*/ #define alternative_call_2 (oldfunc, newfunc1, ft_flags1, newfunc2, ft_flags2, \ output, input...) \ - asm_inline volatile(N_ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1, \ + asm_inline volatile(ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1, \ "call %c[new2]", ft_flags2) \ : output, ASM_CALL_CONSTRAINT \ : [old] "i" (oldfunc), [new1] "i" (newfunc1), \
@@ -422,7 +342,7 @@ void nop_func(void);* @newinstr. ".skip" directive takes care of proper instruction padding * in case @newinstr is longer than @oldinstr. */ -#define __N_ALTERNATIVE (oldinst, newinst, flag) \ +#define __ALTERNATIVE (oldinst, newinst, flag) \ 740: \ oldinst ; \ 741: \
@@ -438,7 +358,7 @@ void nop_func(void); .popsection ; .macro ALTERNATIVE oldinstr, newinstr, ft_flags - __N_ALTERNATIVE(\oldinstr, \newinstr, \ft_flags) + __ALTERNATIVE(\oldinstr, \newinstr, \ft_flags) .endm#define old_len 141b-140b
@@ -447,26 +367,17 @@ void nop_func(void);#define new_len3 146f-145f/* -* gas compatible max based on the idea from: -* http://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax -* -* The additional "-" is needed because gas uses a "true" value of -1. -*/ -#define alt_max_2 (a, b) ((a) ^ (((a) ^ (b)) & -(-((a) < (b))))) -#define alt_max_3 (a, b, c) (alt_max_2(alt_max_2(a, b), c)) - -/* * Same as ALTERNATIVE macro above but for two alternatives. If CPU * has @feature1, it replaces @oldinstr with @newinstr1. If CPU has * @feature2, it replaces @oldinstr with @feature2. */ .macro ALTERNATIVE_2 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2 - __N_ALTERNATIVE(__N_ALTERNATIVE(\oldinstr, \newinstr1, \ft_flags1), + __ALTERNATIVE(__ALTERNATIVE(\oldinstr, \newinstr1, \ft_flags1), \newinstr2, \ft_flags2) .endm .macro ALTERNATIVE_3 oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, newinstr3, ft_flags3 - __N_ALTERNATIVE(N_ALTERNATIVE_2(\oldinstr, \newinstr1, \ft_flags1, \newinstr2, \ft_flags2), + __ALTERNATIVE(ALTERNATIVE_2(\oldinstr, \newinstr1, \ft_flags1, \newinstr2, \ft_flags2), \newinstr3, \ft_flags3) .endm
[tip: x86/alternatives] x86/alternative: Convert ALTERNATIVE_3()
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: 93694129c6e84d3013f5b2787e2ff88dd706c4f0 Gitweb: https://git.kernel.org/tip/93694129c6e84d3013f5b2787e2ff88dd706c4f0 Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:57 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:22:41 +02:00
x86/alternative: Convert ALTERNATIVE_3()
Zap the hack of using an ALTERNATIVE_3() internal label, as suggested by bgerst:
https://lore.kernel.org/r/CAMzpN2i4oJ-Dv0qO46Fd-DxNv5z9=x%2BvO%[email protected]
in favor of a label local to this macro only, as it should be done.
--- arch/x86/include/asm/alternative.h | 24 ++++-------------------- arch/x86/kernel/fpu/xstate.h | 14 +++++--------- 2 files changed, 9 insertions(+), 29 deletions(-)
arch/x86/include/asm/alternative.h +4 -20
@@ -277,26 +277,10 @@ static inline int alternatives_text_reserved(void *start, void *end)N_ALTERNATIVE_2 (oldinstr, newinstr_no,X86_FEATURE_ALWAYS , \ newinstr_yes, ft_flags) -#define ALTERNATIVE_3 (oldinsn, newinsn1, ft_flags1, newinsn2, ft_flags2, \ - newinsn3, ft_flags3) \ - OLDINSTR_3(oldinsn, 1, 2, 3) \ - ".pushsection .altinstructions,\"a\"\n" \ - ALTINSTR_ENTRY(ft_flags1, 1) \ - ALTINSTR_ENTRY(ft_flags2, 2) \ - ALTINSTR_ENTRY(ft_flags3, 3) \ - ".popsection\n" \ - ".pushsection .altinstr_replacement, \"ax\"\n" \ - ALTINSTR_REPLACEMENT(newinsn1, 1) \ - ALTINSTR_REPLACEMENT(newinsn2, 2) \ - ALTINSTR_REPLACEMENT(newinsn3, 3) \ - ".popsection\n" - - -#define N_ALTERNATIVE_3 (oldinst, newinst1, flag1, newinst2, flag2, \ - newinst3, flag3) \ - N_ALTERNATIVE(N_ALTERNATIVE_2(oldinst, newinst1, flag1, newinst2, flag2), \ - newinst3, flag3) - +#define ALTERNATIVE_3 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2, \ + newinstr3, ft_flags3) \ + N_ALTERNATIVE(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2), \ + newinstr3, ft_flags3)/* * Alternative instructions for different CPU types or capabilities. *
arch/x86/kernel/fpu/xstate.h +5 -9
@@ -106,21 +106,17 @@ static inline u64 xfeatures_mask_independent(void)* Otherwise, if XSAVEOPT is enabled, XSAVEOPT replaces XSAVE because XSAVEOPT * supports modified optimization which is not supported by XSAVE. * -* We use XSAVE as a fallback. -* -* The 661 label is defined in the ALTERNATIVE* macros as the address of the -* original instruction which gets replaced. We need to use it here as the -* address of the instruction where we might get an exception at. +* Use XSAVE as a fallback. */ #define XSTATE_XSAVE (st, lmask, hmask, err) \ - asm volatile(ALTERNATIVE_3(XSAVE, \ + asm volatile("1: " ALTERNATIVE_3(XSAVE, \ XSAVEOPT, X86_FEATURE_XSAVEOPT, \ XSAVEC, X86_FEATURE_XSAVEC, \ XSAVES, X86_FEATURE_XSAVES) \ "\n" \ "xor %[err], %[err]\n" \ "3:\n" \ - _ASM_EXTABLE_TYPE_REG(661b, 3b, EX_TYPE_EFAULT_REG, %[err]) \ + _ASM_EXTABLE_TYPE_REG(1b, 3b, EX_TYPE_EFAULT_REG, %[err]) \ : [err] "=r" (err) \ : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \ : "memory")
@@ -130,11 +126,11 @@ static inline u64 xfeatures_mask_independent(void)* XSAVE area format. */ #define XSTATE_XRESTORE (st, lmask, hmask) \ - asm volatile(ALTERNATIVE(XRSTOR, \ + asm volatile("1: " ALTERNATIVE(XRSTOR, \ XRSTORS, X86_FEATURE_XSAVES) \ "\n" \ "3:\n" \ - _ASM_EXTABLE_TYPE(661b, 3b, EX_TYPE_FPU_RESTORE) \ + _ASM_EXTABLE_TYPE(1b, 3b, EX_TYPE_FPU_RESTORE) \ : \ : "D" (st), "m" (*st), "a" (lmask), "d" (hmask) \ : "memory")
[tip: x86/alternatives] x86/alternative: Convert alternative_input()
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: bb91576965e79d17e8e23fccdef91c4bf09d1a74 Gitweb: https://git.kernel.org/tip/bb91576965e79d17e8e23fccdef91c4bf09d1a74 Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:52 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:09:27 +02:00
x86/alternative: Convert alternative_input()
--- arch/x86/include/asm/alternative.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
arch/x86/include/asm/alternative.h +1 -5
@@ -330,7 +330,7 @@ static inline int alternatives_text_reserved(void *start, void *end)* Leaving an unused argument 0 to keep API compatibility. */ #define alternative_input(oldinstr, newinstr, ft_flags, input...) \ - asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ : : "i" (0), ## input)/* Like alternative_input, but with a single output argument */
@@ -356,10 +356,6 @@ static inline int alternatives_text_reserved(void *start, void *end) asm_inline volatile (ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input) -#define n_alternative_input(oldinstr, newinstr, ft_flags, input...) \ - asm_inline volatile (N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ - : : "i" (0), ## input) - #define n_alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ asm_inline volatile (N_ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input)
[tip: x86/alternatives] x86/alternative: Convert ALTERNATIVE_TERNARY()
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: d2d302b1bbe28dba3bd8da855ac9c16aa5dbd00e Gitweb: https://git.kernel.org/tip/d2d302b1bbe28dba3bd8da855ac9c16aa5dbd00e Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:56 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:20:41 +02:00
x86/alternative: Convert ALTERNATIVE_TERNARY()
The C macro.
--- arch/x86/include/asm/alternative.h | 6 ------ 1 file changed, 6 deletions(-)
arch/x86/include/asm/alternative.h +0 -6
@@ -274,15 +274,9 @@ static inline int alternatives_text_reserved(void *start, void *end)/* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ #define ALTERNATIVE_TERNARY (oldinstr, ft_flags, newinstr_yes, newinstr_no) \ - ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ - newinstr_yes, ft_flags) - -/* If @feature is set, patch in @newinstr_yes, otherwise @newinstr_no. */ -#define N_ALTERNATIVE_TERNARY (oldinstr, ft_flags, newinstr_yes, newinstr_no) \ N_ALTERNATIVE_2(oldinstr, newinstr_no, X86_FEATURE_ALWAYS, \ newinstr_yes, ft_flags) -#define ALTERNATIVE_3 (oldinsn, newinsn1, ft_flags1, newinsn2, ft_flags2, \ newinsn3, ft_flags3) \ OLDINSTR_3(oldinsn, 1, 2, 3) \
[tip: x86/alternatives] x86/alternative: Convert alternative_call_2()
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: 719ac02347ee5f94a9f3d5c2fe84640f855432a9 Gitweb: https://git.kernel.org/tip/719ac02347ee5f94a9f3d5c2fe84640f855432a9 Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:55 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:17:10 +02:00
x86/alternative: Convert alternative_call_2()
--- arch/x86/include/asm/alternative.h | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-)
arch/x86/include/asm/alternative.h +6 -15
@@ -358,23 +358,14 @@ static inline int alternatives_text_reserved(void *start, void *end)* Otherwise, if CPU has feature1, function1 is used. * Otherwise, old function is used. */ -#define alternative_call_2 (oldfunc, newfunc1, ft_flags1, newfunc2, ft_flags2, \ - output, input...) \ - asm_inline volatile (ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1, \ - "call %c[new2]", ft_flags2) \ - : output, ASM_CALL_CONSTRAINT \ - : [old] "i" (oldfunc), [new1] "i" (newfunc1), \ +#define alternative_call_2 (oldfunc, newfunc1, ft_flags1, newfunc2, ft_flags2, \ + output, input...) \ + asm_inline volatile(N_ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1, \ + "call %c[new2]", ft_flags2) \ + : output, ASM_CALL_CONSTRAINT \ + : [old] "i" (oldfunc), [new1] "i" (newfunc1), \ [new2] "i" (newfunc2), ## input) -#define n_alternative_call_2 (oldfunc, newfunc1, ft_flags1, newfunc2, ft_flags2, \ - output, input...) \ - asm_inline volatile (N_ALTERNATIVE_2("call %c[old]", "call %c[new1]", ft_flags1,\ - "call %c[new2]", ft_flags2) \ - : output, ASM_CALL_CONSTRAINT \ - : [old] "i" (oldfunc), [new1] "i" (newfunc1), \ - [new2] "i" (newfunc2), ## input) - -/* * use this macro(s) if you need more than one output parameter * in alternative_io
[tip: x86/alternatives] x86/alternative: Convert alternative_2()
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: 8cb1f14b707d3d9b7d2b330c0acd62537576c6a7 Gitweb: https://git.kernel.org/tip/8cb1f14b707d3d9b7d2b330c0acd62537576c6a7 Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:51 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:06:58 +02:00
x86/alternative: Convert alternative_2()
--- arch/x86/include/asm/alternative.h | 3 --- 1 file changed, 3 deletions(-)
arch/x86/include/asm/alternative.h +0 -3
@@ -319,9 +319,6 @@ static inline int alternatives_text_reserved(void *start, void *end) asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory")#define alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ - asm_inline volatile(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory") - -#define n_alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ asm_inline volatile(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory")/*
[tip: x86/alternatives] x86/alternative: Convert alternative_call()
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: a880f9ef6bf7edc36753a9e40388d42ae8d7e099 Gitweb: https://git.kernel.org/tip/a880f9ef6bf7edc36753a9e40388d42ae8d7e099 Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:54 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:15:03 +02:00
x86/alternative: Convert alternative_call()
--- arch/x86/include/asm/alternative.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
arch/x86/include/asm/alternative.h +1 -5
@@ -349,11 +349,7 @@ static inline int alternatives_text_reserved(void *start, void *end)* suffix. */ #define alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ - asm_inline volatile (ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ - : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input) - -#define n_alternative_call(oldfunc, newfunc, ft_flags, output, input...) \ - asm_inline volatile (N_ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ + asm_inline volatile(N_ALTERNATIVE("call %c[old]", "call %c[new]", ft_flags) \ : output : [old] "i" (oldfunc), [new] "i" (newfunc), ## input)/*
[tip: x86/alternatives] x86/alternative: Zap alternative_ternary()
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: 9cbf2643b3ec7866e688df35d5b28c8b07ecbe6c Gitweb: https://git.kernel.org/tip/9cbf2643b3ec7866e688df35d5b28c8b07ecbe6c Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:48 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 17:07:35 +02:00
x86/alternative: Zap alternative_ternary()
Unused.
--- arch/x86/include/asm/alternative.h | 3 --- 1 file changed, 3 deletions(-)
arch/x86/include/asm/alternative.h +0 -3
@@ -271,9 +271,6 @@ static inline int alternatives_text_reserved(void *start, void *end)#define alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ asm_inline volatile(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory") -#define alternative_ternary (oldinstr, ft_flags, newinstr_yes, newinstr_no) \ - asm_inline volatile(ALTERNATIVE_TERNARY(oldinstr, ft_flags, newinstr_yes, newinstr_no) ::: "memory") -/* * Alternative inline assembly with input. *
[tip: x86/alternatives] x86/alternative: Convert alternative()
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: b94c1fe10be544070f53e4a1e5164087e446945c Gitweb: https://git.kernel.org/tip/b94c1fe10be544070f53e4a1e5164087e446945c Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:50 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 17:14:26 +02:00
x86/alternative: Convert alternative()
Split conversion deliberately into minimal pieces to ease bisection because debugging alternatives is a nightmare.
--- arch/x86/include/asm/alternative.h | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
arch/x86/include/asm/alternative.h +1 -4
@@ -316,14 +316,11 @@ static inline int alternatives_text_reserved(void *start, void *end)* without volatile and memory clobber. */ #define alternative (oldinstr, newinstr, ft_flags) \ - asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory") + asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory")#define alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ asm_inline volatile(ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory") -#define n_alternative (oldinstr, newinstr, ft_flags) \ - asm_inline volatile (N_ALTERNATIVE(oldinstr, newinstr, ft_flags) : : : "memory") -#define n_alternative_2 (oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) \ asm_inline volatile(N_ALTERNATIVE_2(oldinstr, newinstr1, ft_flags1, newinstr2, ft_flags2) ::: "memory")
[tip: x86/alternatives] x86/alternative: Convert alternative_io()
In reply to Borislav PetkovThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: ad36085ee3563001cf1720eeb93c2e85715eb5cd Gitweb: https://git.kernel.org/tip/ad36085ee3563001cf1720eeb93c2e85715eb5cd Author: Borislav Petkov (AMD) <[email protected]> AuthorDate: Fri, 07 Jun 2024 13:16:53 +02:00 Committer: Borislav Petkov (AMD) <[email protected]> CommitterDate: Tue, 11 Jun 2024 18:12:29 +02:00
x86/alternative: Convert alternative_io()
--- arch/x86/include/asm/alternative.h | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
arch/x86/include/asm/alternative.h +1 -5
@@ -335,11 +335,7 @@ static inline int alternatives_text_reserved(void *start, void *end)/* Like alternative_input, but with a single output argument */ #define alternative_io(oldinstr, newinstr, ft_flags, output, input...) \ - asm_inline volatile (ALTERNATIVE(oldinstr, newinstr, ft_flags) \ - : output : "i" (0), ## input) - -#define n_alternative_io(oldinstr, newinstr, ft_flags, output, input...) \ - asm_inline volatile (N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ + asm_inline volatile(N_ALTERNATIVE(oldinstr, newinstr, ft_flags) \ : output : "i" (0), ## input)/*