Skip to contentSkip to conversation

x86: allow Kconfig control over psABI level

PATCH In review Jan Beulich 2 participants 14 messages Jul 12, 2023 – Jul 18, 2023

lore
Share & tools
lore
#0 lore raw

cover x86: allow Kconfig control over psABI level

As discussed in Prague, with some minor add-ons (patch 2 replacing a much older patch under the same title).

1: allow Kconfig control over psABI level
2: use POPCNT for hweight<N>() when available
3: short-circuit certain cpu_has_* when x86-64-v{2,3} are in effect

Jan

#1 lore raw

PATCH 1/3 x86: allow Kconfig control over psABI level

Newer hardware offers more efficient and/or flexible and/or capable instructions, some of which we can make good use of in the hypervisor as well. Allow a basic way (no alternatives patching) of enabling their use. Of course this means that hypervisors thus built won't work anymore on older, less capable hardware.

---
TBD: Should we, just like for NX, add an early check in assembly code,
to have a "clean" failure rather than a random crash?

Whereas the baseline -> v2 step isn't much of a difference (we'll gain more there by a subsequent patch), v2 -> v3, while presumably (or shall I say hopefully) faster, yields an overall growth of .text size by (in my build) about 2k. The primary reason for this appear to be conversions of SHL-by-immediate to SHLX.

xen/arch/x86/Kconfig +30 -0
@@ -118,6 +118,36 @@ config HVM
 
 	  If unsure, say Y.
 
+choice
+	prompt "base psABI level"
+	default X86_64_BASELINE
+	help
+	  The psABI defines 4 levels of ISA extension sets as a coarse granular
+	  way of identifying advanced functionality that would be uniformly
+	  available in respectively newer hardware.  While v4 is not really of
+	  interest for Xen, the others can be selected here, making the
+	  resulting Xen no longer work on older hardware.  This option won't
+	  have any effect if the toolchain doesn't support the distinction.
+
+	  If unsure, stick to the default.
+
+config X86_64_BASELINE
+	bool "baseline"
+
+config X86_64_V2
+	bool "v2"
+	help
+	  This enables POPCNT and CX16, besides other extensions which are of
+	  no interest here.
+
+config X86_64_V3
+	bool "v3"
+	help
+	  This enables BMI, BMI2, LZCNT, and MOVBE, besides other extensions
+	  which are of no interest here.
+
+endchoice
+
 config XEN_SHSTK
 	bool "Supervisor Shadow Stacks"
 	depends on HAS_AS_CET_SS
xen/arch/x86/arch.mk +4 -0
@@ -36,6 +36,10 @@ CFLAGS += -mno-red-zone -fpic
 # the SSE setup for variadic function calls.
 CFLAGS += -mno-mmx -mno-sse $(call cc-option,$(CC),-mskip-rax-setup)
 
+# Enable the selected baseline ABI, if supported by the compiler.
+CFLAGS-$(CONFIG_X86_64_V2) += $(call cc-option,$(CC),-march=x86-64-v2)
+CFLAGS-$(CONFIG_X86_64_V3) += $(call cc-option,$(CC),-march=x86-64-v3)
+
 ifeq ($(CONFIG_INDIRECT_THUNK),y)
 # Compile with gcc thunk-extern, indirect-branch-register if available.
 CFLAGS-$(CONFIG_CC_IS_GCC) += -mindirect-branch=thunk-extern
#2 lore raw

xen/arch/x86/arch.mk CFLAGS += -mno-red-zone -fpic · line 41

Quoted 9 lines · Jan Beulich
xen/arch/x86/arch.mk +3 -0
@@ -36,6 +36,10 @@ CFLAGS += -mno-red-zone -fpic
 # the SSE setup for variadic function calls.
 CFLAGS += -mno-mmx -mno-sse $(call cc-option,$(CC),-mskip-rax-setup)
 
+# Enable the selected baseline ABI, if supported by the compiler.
+CFLAGS-$(CONFIG_X86_64_V2) += $(call cc-option,$(CC),-march=x86-64-v2)
+CFLAGS-$(CONFIG_X86_64_V3) += $(call cc-option,$(CC),-march=x86-64-v3)

Hmm, I should have remembered that this won't work with older gcc, because of how $(cc-option ...) works. I recall someone else already fell into this trap a few years ago, but then they weren't following up on the suggestions on how to make their somewhat fragile workaround more robust. Now I'm on the hook ...

Jan

Reviewed-by on v1 1/3
#3 lore raw
In reply to Jan Beulich
Quoted 8 lines · Jan Beulich

Newer hardware offers more efficient and/or flexible and/or capable instructions, some of which we can make good use of in the hypervisor as well. Allow a basic way (no alternatives patching) of enabling their use. Of course this means that hypervisors thus built won't work anymore on older, less capable hardware.

With two comments.

Quoted 3 lines

---
TBD: Should we, just like for NX, add an early check in assembly code,
to have a "clean" failure rather than a random crash?

That is less confusing than a random crash.

Quoted 29 lines

Whereas the baseline -> v2 step isn't much of a difference (we'll gain more there by a subsequent patch), v2 -> v3, while presumably (or shall I say hopefully) faster, yields an overall growth of .text size by (in my build) about 2k. The primary reason for this appear to be conversions of SHL-by-immediate to SHLX.

xen/arch/x86/Kconfig +17 -0
@@ -118,6 +118,36 @@ config HVM
 
          If unsure, say Y.
 
+choice
+       prompt "base psABI level"
+       default X86_64_BASELINE
+       help
+         The psABI defines 4 levels of ISA extension sets as a coarse granular
+         way of identifying advanced functionality that would be uniformly
+         available in respectively newer hardware.  While v4 is not really of
+         interest for Xen, the others can be selected here, making the
+         resulting Xen no longer work on older hardware.  This option won't
+         have any effect if the toolchain doesn't support the distinction.
+
+         If unsure, stick to the default.
+
+config X86_64_BASELINE
+       bool "baseline"
+
+config X86_64_V2

xen/arch/x86/Kconfig config HVM · line 147

Quoted 10 lines
xen/arch/x86/Kconfig +10 -0
+       bool "v2"
+       help
+         This enables POPCNT and CX16, besides other extensions which are of
+         no interest here.
+
+config X86_64_V3
+       bool "v3"
+       help
+         This enables BMI, BMI2, LZCNT, and MOVBE, besides other extensions
+         which are of no interest here.

Would it make sense to try and specify models? According to this:
https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels

x86-64-v2
(circa 2009: Nehalem and Jaguar)
Atom Silvermont (2013)
VIA Nano and Eden "C" (2015)

x86-64-v3
(circa 2015: Haswell and Excavator)
Atom Gracemont (2021)

Regards,
Jason

#4 lore raw
Quoted 11 lines · Jason Andryuk
Quoted 8 lines · Jan Beulich

Newer hardware offers more efficient and/or flexible and/or capable instructions, some of which we can make good use of in the hypervisor as well. Allow a basic way (no alternatives patching) of enabling their use. Of course this means that hypervisors thus built won't work anymore on older, less capable hardware.

Thanks.

Quoted 7 lines

With two comments.

Quoted 3 lines

---
TBD: Should we, just like for NX, add an early check in assembly code,
to have a "clean" failure rather than a random crash?

That is less confusing than a random crash.

Of course. My thinking behind not doing this right away was that people aren't supposed to use such a hypervisor on incapable hardware. So we would likely add code for something which ought to never happen. Still for v2 I guess I'll add such code (then of course invalidating your R-b again); in the worst case it should be easy to rip out again from the patch.

Quoted 37 lines
Quoted 23 lines
xen/arch/x86/Kconfig +17 -0
@@ -118,6 +118,36 @@ config HVM
 
          If unsure, say Y.
 
+choice
+       prompt "base psABI level"
+       default X86_64_BASELINE
+       help
+         The psABI defines 4 levels of ISA extension sets as a coarse granular
+         way of identifying advanced functionality that would be uniformly
+         available in respectively newer hardware.  While v4 is not really of
+         interest for Xen, the others can be selected here, making the
+         resulting Xen no longer work on older hardware.  This option won't
+         have any effect if the toolchain doesn't support the distinction.
+
+         If unsure, stick to the default.
+
+config X86_64_BASELINE
+       bool "baseline"
+
+config X86_64_V2

xen/arch/x86/Kconfig config HVM · line 147

Quoted 10 lines
xen/arch/x86/Kconfig +10 -0
+       bool "v2"
+       help
+         This enables POPCNT and CX16, besides other extensions which are of
+         no interest here.
+
+config X86_64_V3
+       bool "v3"
+       help
+         This enables BMI, BMI2, LZCNT, and MOVBE, besides other extensions
+         which are of no interest here.

Would it make sense to try and specify models? According to this:
https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels

Quoted 8 lines

x86-64-v2
(circa 2009: Nehalem and Jaguar)
Atom Silvermont (2013)
VIA Nano and Eden "C" (2015)

x86-64-v3
(circa 2015: Haswell and Excavator)
Atom Gracemont (2021)

I'm not sure how useful this would be without supplying a reasonably complete list of models. I don't think we can expect people to be aware of the sequence of model names, without which specifying just the first respective models isn't very useful.

Jan

#5 lore raw
In reply to Jan Beulich
Quoted 10 lines · Jan Beulich

Newer hardware offers more efficient and/or flexible and/or capable instructions, some of which we can make good use of in the hypervisor as well. Allow a basic way (no alternatives patching) of enabling their use. Of course this means that hypervisors thus built won't work anymore on older, less capable hardware.

---
TBD: Should we, just like for NX, add an early check in assembly code,
to have a "clean" failure rather than a random crash?

TBD: While older compilers (apparently gcc10 and older; not sure about
clang) won't recognize -march=x86-64-v2 etc, we could fall back to passing -mpopcnt and alike explicitly.

Jan

Quoted 60 lines

Whereas the baseline -> v2 step isn't much of a difference (we'll gain more there by a subsequent patch), v2 -> v3, while presumably (or shall I say hopefully) faster, yields an overall growth of .text size by (in my build) about 2k. The primary reason for this appear to be conversions of SHL-by-immediate to SHLX.

xen/arch/x86/Kconfig +30 -0
@@ -118,6 +118,36 @@ config HVM
 
 	  If unsure, say Y.
 
+choice
+	prompt "base psABI level"
+	default X86_64_BASELINE
+	help
+	  The psABI defines 4 levels of ISA extension sets as a coarse granular
+	  way of identifying advanced functionality that would be uniformly
+	  available in respectively newer hardware.  While v4 is not really of
+	  interest for Xen, the others can be selected here, making the
+	  resulting Xen no longer work on older hardware.  This option won't
+	  have any effect if the toolchain doesn't support the distinction.
+
+	  If unsure, stick to the default.
+
+config X86_64_BASELINE
+	bool "baseline"
+
+config X86_64_V2
+	bool "v2"
+	help
+	  This enables POPCNT and CX16, besides other extensions which are of
+	  no interest here.
+
+config X86_64_V3
+	bool "v3"
+	help
+	  This enables BMI, BMI2, LZCNT, and MOVBE, besides other extensions
+	  which are of no interest here.
+
+endchoice
+
 config XEN_SHSTK
 	bool "Supervisor Shadow Stacks"
 	depends on HAS_AS_CET_SS
xen/arch/x86/arch.mk +4 -0
@@ -36,6 +36,10 @@ CFLAGS += -mno-red-zone -fpic
 # the SSE setup for variadic function calls.
 CFLAGS += -mno-mmx -mno-sse $(call cc-option,$(CC),-mskip-rax-setup)
 
+# Enable the selected baseline ABI, if supported by the compiler.
+CFLAGS-$(CONFIG_X86_64_V2) += $(call cc-option,$(CC),-march=x86-64-v2)
+CFLAGS-$(CONFIG_X86_64_V3) += $(call cc-option,$(CC),-march=x86-64-v3)
+
 ifeq ($(CONFIG_INDIRECT_THUNK),y)
 # Compile with gcc thunk-extern, indirect-branch-register if available.
 CFLAGS-$(CONFIG_CC_IS_GCC) += -mindirect-branch=thunk-extern
#6 lore raw

PATCH 2/3 x86: use POPCNT for hweight<N>() when available

In reply to Jan Beulich

This is faster than using the software implementation, and the insn is available on all half-way recent hardware. Use the respective compiler builtins when available.

---
v3: Re-do in a much simpler manner, without alternatives patching.
v2: Also suppress UB sanitizer instrumentation. Reduce macroization in
hweight.c. Exclude clang builds.

xen/arch/x86/include/asm/bitops.h +7 -0
@@ -475,9 +475,16 @@ static inline int fls(unsigned int x)
  *
  * The Hamming Weight of a number is the total number of bits set in it.
  */
+#ifdef __POPCNT__
+#define hweight64(x) __builtin_popcountll(x)
+#define hweight32(x) __builtin_popcount(x)
+#define hweight16(x) __builtin_popcount((uint16_t)(x))
+#define hweight8(x)  __builtin_popcount((uint8_t)(x))
+#else
 #define hweight64(x) generic_hweight64(x)
 #define hweight32(x) generic_hweight32(x)
 #define hweight16(x) generic_hweight16(x)
 #define hweight8(x) generic_hweight8(x)
+#endif
 
 #endif /* _X86_BITOPS_H */
#8 lore raw

PATCH 3/3 x86: short-circuit certain cpu_has_* when x86-64-v{2,3} are in effect

In reply to Jan Beulich

Certain fallback code can be made subject to DCE this way. Note that CX16 has no compiler provided manifest constant, so CONFIG_* are used there instead. Note also that we don't have cpu_has_movbe nor cpu_has_lzcnt (aka cpu_has_abm).

---
Of course we could use IS_ENABLED(CONFIG_X86_64_V<n>) everywhere, but as CX16 shows this isn't necessarily better than the #if/#else approach based on compiler-provided manifest constants. While not really intended to be used that way, it looks as if we could also use IS_ENABLED(__POPCNT__) and alike.

We could go further and also short-circuit SSE*, AVX and alike, which we don't use outside of the emulator.

xen/arch/x86/include/asm/cpufeature.h +15 -1
@@ -76,13 +76,19 @@ static inline bool boot_cpu_has(unsigned
 #define cpu_has_eist            boot_cpu_has(X86_FEATURE_EIST)
 #define cpu_has_ssse3           boot_cpu_has(X86_FEATURE_SSSE3)
 #define cpu_has_fma             boot_cpu_has(X86_FEATURE_FMA)
-#define cpu_has_cx16            boot_cpu_has(X86_FEATURE_CX16)
+#define cpu_has_cx16            (IS_ENABLED(CONFIG_X86_64_V2) || \
+                                 IS_ENABLED(CONFIG_X86_64_V3) || \
+                                 boot_cpu_has(X86_FEATURE_CX16))
 #define cpu_has_pdcm            boot_cpu_has(X86_FEATURE_PDCM)
 #define cpu_has_pcid            boot_cpu_has(X86_FEATURE_PCID)
 #define cpu_has_sse4_1          boot_cpu_has(X86_FEATURE_SSE4_1)
 #define cpu_has_sse4_2          boot_cpu_has(X86_FEATURE_SSE4_2)
 #define cpu_has_x2apic          boot_cpu_has(X86_FEATURE_X2APIC)
+#ifdef __POPCNT__
+#define cpu_has_popcnt          true
+#else
 #define cpu_has_popcnt          boot_cpu_has(X86_FEATURE_POPCNT)
+#endif
 #define cpu_has_aesni           boot_cpu_has(X86_FEATURE_AESNI)
 #define cpu_has_xsave           boot_cpu_has(X86_FEATURE_XSAVE)
 #define cpu_has_avx             boot_cpu_has(X86_FEATURE_AVX)
@@ -114,11 +120,19 @@ static inline bool boot_cpu_has(unsigned
 #define cpu_has_xsaves          boot_cpu_has(X86_FEATURE_XSAVES)
 
 /* CPUID level 0x00000007:0.ebx */
+#ifdef __BMI__
+#define cpu_has_bmi1            true
+#else
 #define cpu_has_bmi1            boot_cpu_has(X86_FEATURE_BMI1)
+#endif
 #define cpu_has_hle             boot_cpu_has(X86_FEATURE_HLE)
 #define cpu_has_avx2            boot_cpu_has(X86_FEATURE_AVX2)
 #define cpu_has_smep            boot_cpu_has(X86_FEATURE_SMEP)
+#ifdef __BMI2__
+#define cpu_has_bmi2            true
+#else
 #define cpu_has_bmi2            boot_cpu_has(X86_FEATURE_BMI2)
+#endif
 #define cpu_has_invpcid         boot_cpu_has(X86_FEATURE_INVPCID)
 #define cpu_has_rtm             boot_cpu_has(X86_FEATURE_RTM)
 #define cpu_has_pqe             boot_cpu_has(X86_FEATURE_PQE)
Reviewed-by on v1 3/3
#9 lore raw

Re: [PATCH 3/3] x86: short-circuit certain cpu_has_* when x86-64-v{2,3} are in effect

Quoted 7 lines · Jan Beulich

Certain fallback code can be made subject to DCE this way. Note that CX16 has no compiler provided manifest constant, so CONFIG_* are used there instead. Note also that we don't have cpu_has_movbe nor cpu_has_lzcnt (aka cpu_has_abm).

One thought below.

Quoted 13 lines

---
Of course we could use IS_ENABLED(CONFIG_X86_64_V<n>) everywhere, but as CX16 shows this isn't necessarily better than the #if/#else approach based on compiler-provided manifest constants. While not really intended to be used that way, it looks as if we could also use IS_ENABLED(__POPCNT__) and alike.

We could go further and also short-circuit SSE*, AVX and alike, which we don't use outside of the emulator.

xen/arch/x86/include/asm/cpufeature.h static inline bool boot_cpu_has(unsigned · line 81

Quoted 7 lines
xen/arch/x86/include/asm/cpufeature.h +3 -1
@@ -76,13 +76,19 @@ static inline bool boot_cpu_has(unsigned
 #define cpu_has_eist            boot_cpu_has(X86_FEATURE_EIST)
 #define cpu_has_ssse3           boot_cpu_has(X86_FEATURE_SSSE3)
 #define cpu_has_fma             boot_cpu_has(X86_FEATURE_FMA)
-#define cpu_has_cx16            boot_cpu_has(X86_FEATURE_CX16)
+#define cpu_has_cx16            (IS_ENABLED(CONFIG_X86_64_V2) || \
+                                 IS_ENABLED(CONFIG_X86_64_V3) || \
+                                 boot_cpu_has(X86_FEATURE_CX16))

If you think there may be more ABI selections in the future, it might be better to express the "V$N" numerically and check >= 2. Or you can add a Kconfig CONFIG_X86_64_CX16 and select that as appropriate. But if there aren't going to be more of these, then this is fine.

Regards,
Jason

#10 lore raw

Re: [PATCH 3/3] x86: short-circuit certain cpu_has_* when x86-64-v{2,3} are in effect

Quoted 10 lines · Jason Andryuk
Quoted 7 lines · Jan Beulich

Certain fallback code can be made subject to DCE this way. Note that CX16 has no compiler provided manifest constant, so CONFIG_* are used there instead. Note also that we don't have cpu_has_movbe nor cpu_has_lzcnt (aka cpu_has_abm).

Thanks.

Quoted 7 lines
Quoted 7 lines
xen/arch/x86/include/asm/cpufeature.h +0 -1
@@ -76,13 +76,19 @@ static inline bool boot_cpu_has(unsigned
 #define cpu_has_eist            boot_cpu_has(X86_FEATURE_EIST)
 #define cpu_has_ssse3           boot_cpu_has(X86_FEATURE_SSSE3)
 #define cpu_has_fma             boot_cpu_has(X86_FEATURE_FMA)
-#define cpu_has_cx16            boot_cpu_has(X86_FEATURE_CX16)
Quoted 8 lines

xen/arch/x86/include/asm/cpufeature.h static inline bool boot_cpu_has(unsigned · line 81

Quoted 3 lines
xen/arch/x86/include/asm/cpufeature.h +3 -0
+#define cpu_has_cx16            (IS_ENABLED(CONFIG_X86_64_V2) || \
+                                 IS_ENABLED(CONFIG_X86_64_V3) || \
+                                 boot_cpu_has(X86_FEATURE_CX16))

If you think there may be more ABI selections in the future, it might be better to express the "V$N" numerically and check >= 2. Or you can add a Kconfig CONFIG_X86_64_CX16 and select that as appropriate. But if there aren't going to be more of these, then this is fine.

I was thinking this same way: If more appear (which aren't SIMD-only, like v4 is), we can use a numeric CONFIG_*, but for now it's good enough (and slightly simpler) this way.

Jan

#11 lore raw

PATCH 1/1 build: make cc-option properly deal with unrecognized sub-options

In reply to Jan Beulich

In options like -march=, it may be only the sub-option which is unrecognized by the compiler. In such an event the error message often splits option and argument, typically saying something like "bad value '<argument>' for '<option>'. Extend the grep invocation to check for any of the three resulting variants.

To keep things halfway readable, re-wrap and re-indent the entire construct.

Config.mk +8 -3
@@ -90,9 +90,14 @@ PYTHON_PREFIX_ARG ?= --prefix="$(prefix)
 # of which would indicate an "unrecognized command-line option" warning/error.
 #
 # Usage: cflags-y += $(call cc-option,$(CC),-march=winchip-c6,-march=i586)
-cc-option = $(shell if test -z "`echo 'void*p=1;' | \
-              $(1) $(2) -c -o /dev/null -x c - 2>&1 | grep -- $(2:-Wa$(comma)%=%) -`"; \
-              then echo "$(2)"; else echo "$(3)"; fi ;)
+cc-option = $(shell pat='$(2:-Wa$(comma)%=%)'; \
+                    opt="$${pat%%=*}" arg="$${pat\#*=}"; \
+                    if test -z "`echo 'void*p=1;' | \
+                                 $(1) $(2) -c -o /dev/null -x c - 2>&1 | \
+                                 grep -e "$$pat" -e "$$arg.*$$opt" -e "$$opt.*$$arg" -`"; \
+                    then echo "$(2)"; \
+                    else echo "$(3)"; \
+                    fi;)
 
 # cc-option-add: Add an option to compilation flags, but only if supported.
 # Usage: $(call cc-option-add CFLAGS,CC,-march=winchip-c6)
#12 lore raw

PATCH 1/1 build: make cc-option properly deal with unrecognized sub-options

In reply to Jan Beulich

In options like -march=, it may be only the sub-option which is unrecognized by the compiler. In such an event the error message often splits option and argument, typically saying something like "bad value '<argument>' for '<option>'. Extend the grep invocation to check for any of the three resulting variants.

To keep things halfway readable, re-wrap and re-indent the entire construct.

---
I'm sorry, re-sending with wider Cc list (as was intended originally).

Config.mk +8 -3
@@ -90,9 +90,14 @@ PYTHON_PREFIX_ARG ?= --prefix="$(prefix)
 # of which would indicate an "unrecognized command-line option" warning/error.
 #
 # Usage: cflags-y += $(call cc-option,$(CC),-march=winchip-c6,-march=i586)
-cc-option = $(shell if test -z "`echo 'void*p=1;' | \
-              $(1) $(2) -c -o /dev/null -x c - 2>&1 | grep -- $(2:-Wa$(comma)%=%) -`"; \
-              then echo "$(2)"; else echo "$(3)"; fi ;)
+cc-option = $(shell pat='$(2:-Wa$(comma)%=%)'; \
+                    opt="$${pat%%=*}" arg="$${pat\#*=}"; \
+                    if test -z "`echo 'void*p=1;' | \
+                                 $(1) $(2) -c -o /dev/null -x c - 2>&1 | \
+                                 grep -e "$$pat" -e "$$arg.*$$opt" -e "$$opt.*$$arg" -`"; \
+                    then echo "$(2)"; \
+                    else echo "$(3)"; \
+                    fi;)
 
 # cc-option-add: Add an option to compilation flags, but only if supported.
 # Usage: $(call cc-option-add CFLAGS,CC,-march=winchip-c6)
#13 lore raw

Re: [PATCH 0.5/3] build: make cc-option properly deal with unrecognized sub-options

Quoted 4 lines · Jan Beulich
Config.mk +0 -0
@@ -90,9 +90,14 @@ PYTHON_PREFIX_ARG ?= --prefix="$(prefix)
 # of which would indicate an "unrecognized command-line option" warning/error.

Config.mk PYTHON_PREFIX_ARG ?= --prefix="$(prefix) · line 97

Quoted 10 lines
Config.mk +5 -3
 #
 # Usage: cflags-y += $(call cc-option,$(CC),-march=winchip-c6,-march=i586)
-cc-option = $(shell if test -z "`echo 'void*p=1;' | \
-              $(1) $(2) -c -o /dev/null -x c - 2>&1 | grep -- $(2:-Wa$(comma)%=%) -`"; \
-              then echo "$(2)"; else echo "$(3)"; fi ;)
+cc-option = $(shell pat='$(2:-Wa$(comma)%=%)'; \
+                    opt="$${pat%%=*}" arg="$${pat\#*=}"; \
+                    if test -z "`echo 'void*p=1;' | \
+                                 $(1) $(2) -c -o /dev/null -x c - 2>&1 | \
+                                 grep -e "$$pat" -e "$$arg.*$$opt" -e "$$opt.*$$arg" -`"; \

Sigh. At least some versions of Clang (I'm looking at 7.0.1 right now) don't quote "-march" in their resulting error message. So I guess all we can do (without finding a different approach to the checking altogether) is further relax the grep pattern, to

grep -e "$$pat" -e "$$opt" -e "$$arg" -

Of course this has the increasing risk of getting false positives (i.e. false negatives for the overall check result).

Jan