Skip to contentSkip to conversation

x86: allow Kconfig control over psABI level

PATCH In review Jan Beulich 3 participants 15 messages Jul 26, 2023 – Aug 22, 2023

lore
Share & tools
lore
#0 lore raw

v3 cover x86: allow Kconfig control over psABI level

As discussed in Prague, with some minor add-ons (patch 2 replacing a much older patch under the same title).

v3 addresses issues found as well as review feedback; see individual patches for details. Patch 3 in particular continues to raises several questions as well, though.

1: build: make cc-option properly deal with unrecognized sub-options 2: build: permit Kconfig control over how to deal with unsatisfiable choices 3: allow Kconfig control over psABI level
4: use POPCNT for hweight<N>() when available
5: short-circuit certain cpu_has_* when x86-64-v{2,3} are in effect

Jan

#1 lore raw

PATCH v3 1/5 build: make cc-option properly deal with unrecognized sub-options

In options like -march=, it may be only the sub-option which is unrecognized by the compiler. In such an event the error message often splits option and argument, typically saying something like "bad value '<argument>' for '<option>'. Extend the grep invocation accordingly, also accounting for Clang to not mention e.g. -march at all when an incorrect argument was given for it.

To keep things halfway readable, re-wrap and re-indent the entire construct.

---
In principle -e "$$pat" could now be omitted from the grep invocation, since if that matches, both $$opt and $$arg will, too. But I thought I'd leave it for completeness.
---
v3: Fix build with make 4.3 and newer, where the treatment of \# has
changed.
v2: Further relax grep patterns for clang, which doesn't mention -march
when complaining about an invalid argument to it.

Config.mk +9 -3
@@ -8,6 +8,7 @@ endif
 comma   := ,
 open    := (
 close   := )
+sharp   := \#
 squote  := '
 #' Balancing squote, to help syntax highlighting
 empty   :=
@@ -90,9 +91,14 @@ PYTHON_PREFIX_ARG ?= --prefix="$(prefix)
 # of which would indicate an "unrecognized command-line option" warning/error.
 #
 # Usage: cflags-y += $(call cc-option,$(CC),-march=winchip-c6,-march=i586)
-cc-option = $(shell if test -z "`echo 'void*p=1;' | \
-              $(1) $(2) -c -o /dev/null -x c - 2>&1 | grep -- $(2:-Wa$(comma)%=%) -`"; \
-              then echo "$(2)"; else echo "$(3)"; fi ;)
+cc-option = $(shell pat='$(2:-Wa$(comma)%=%)'; \
+                    opt="$${pat%%=*}" arg="$${pat$(sharp)*=}"; \
+                    if test -z "`echo 'void*p=1;' | \
+                                 $(1) $(2) -c -o /dev/null -x c - 2>&1 | \
+                                 grep -e "$$pat" -e "$$opt" -e "$$arg" -`"; \
+                    then echo "$(2)"; \
+                    else echo "$(3)"; \
+                    fi;)
 
 # cc-option-add: Add an option to compilation flags, but only if supported.
 # Usage: $(call cc-option-add CFLAGS,CC,-march=winchip-c6)
Reviewed-by on v3 1/5
#2 lore raw

Re: [PATCH v3 1/5] build: make cc-option properly deal with unrecognized sub-options

Quoted 36 lines · Jan Beulich

In options like -march=, it may be only the sub-option which is unrecognized by the compiler. In such an event the error message often splits option and argument, typically saying something like "bad value '<argument>' for '<option>'. Extend the grep invocation accordingly, also accounting for Clang to not mention e.g. -march at all when an incorrect argument was given for it.

To keep things halfway readable, re-wrap and re-indent the entire construct.

---
In principle -e "$$pat" could now be omitted from the grep invocation, since if that matches, both $$opt and $$arg will, too. But I thought I'd leave it for completeness.
---
v3: Fix build with make 4.3 and newer, where the treatment of \# has
changed.
v2: Further relax grep patterns for clang, which doesn't mention -march
when complaining about an invalid argument to it.

Config.mk +1 -1
@@ -8,6 +8,7 @@ endif
 comma   := ,
 open    := (
 close   := )
+sharp   := \#
 squote  := '
 #' Balancing squote, to help syntax highlighting
 empty   :=
@@ -90,9 +91,14 @@ PYTHON_PREFIX_ARG ?= --prefix="$(prefix)
 # of which would indicate an "unrecognized command-line option" warning/error.
 #
 # Usage: cflags-y += $(call cc-option,$(CC),-march=winchip-c6,-march=i586)
-cc-option = $(shell if test -z "`echo 'void*p=1;' | \

Config.mk PYTHON_PREFIX_ARG ?= --prefix="$(prefix) · line 101

Quoted 10 lines
Config.mk +8 -2
-              $(1) $(2) -c -o /dev/null -x c - 2>&1 | grep -- $(2:-Wa$(comma)%=%) -`"; \
-              then echo "$(2)"; else echo "$(3)"; fi ;)
+cc-option = $(shell pat='$(2:-Wa$(comma)%=%)'; \
+                    opt="$${pat%%=*}" arg="$${pat$(sharp)*=}"; \
+                    if test -z "`echo 'void*p=1;' | \
+                                 $(1) $(2) -c -o /dev/null -x c - 2>&1 | \
+                                 grep -e "$$pat" -e "$$opt" -e "$$arg" -`"; \
+                    then echo "$(2)"; \
+                    else echo "$(3)"; \
+                    fi;)

This patch looks fine. Shouldn't the comment been updated as well? At the moment, it only discuss about -Wno-*, which it seems is why grep was introduced in the first place.

But isn't it doing doing pattern matching on an error message going to lead sometime to false positive? Linux's build system seems to works fine by just using the exit value. They've got a few trick to deal with -Wno-* and with clang.

For -Wno-$(warning), they test -W$(warning) instead. For clang, they've enable additional warnings:
-Werror=unknown-warning-option -Werror=ignored-optimization-argument -Werror=option-ignored -Werror=unused-command-line-argument

In any case, the patch is fine.

Thanks,

Signature · 1 line
Anthony PERARD
#3 lore raw

Re: [PATCH v3 1/5] build: make cc-option properly deal with unrecognized sub-options

Quoted 43 lines · Anthony PERARD
Quoted 42 lines · Jan Beulich

In options like -march=, it may be only the sub-option which is unrecognized by the compiler. In such an event the error message often splits option and argument, typically saying something like "bad value '<argument>' for '<option>'. Extend the grep invocation accordingly, also accounting for Clang to not mention e.g. -march at all when an incorrect argument was given for it.

To keep things halfway readable, re-wrap and re-indent the entire construct.

---
In principle -e "$$pat" could now be omitted from the grep invocation, since if that matches, both $$opt and $$arg will, too. But I thought I'd leave it for completeness.
---
v3: Fix build with make 4.3 and newer, where the treatment of \# has
changed.
v2: Further relax grep patterns for clang, which doesn't mention -march
when complaining about an invalid argument to it.

Config.mk +5 -3
@@ -8,6 +8,7 @@ endif
 comma   := ,
 open    := (
 close   := )
+sharp   := \#
 squote  := '
 #' Balancing squote, to help syntax highlighting
 empty   :=
@@ -90,9 +91,14 @@ PYTHON_PREFIX_ARG ?= --prefix="$(prefix)
 # of which would indicate an "unrecognized command-line option" warning/error.
 #
 # Usage: cflags-y += $(call cc-option,$(CC),-march=winchip-c6,-march=i586)
-cc-option = $(shell if test -z "`echo 'void*p=1;' | \
-              $(1) $(2) -c -o /dev/null -x c - 2>&1 | grep -- $(2:-Wa$(comma)%=%) -`"; \
-              then echo "$(2)"; else echo "$(3)"; fi ;)
+cc-option = $(shell pat='$(2:-Wa$(comma)%=%)'; \
+                    opt="$${pat%%=*}" arg="$${pat$(sharp)*=}"; \
+                    if test -z "`echo 'void*p=1;' | \
+                                 $(1) $(2) -c -o /dev/null -x c - 2>&1 | \
Quoted 8 lines

Config.mk PYTHON_PREFIX_ARG ?= --prefix="$(prefix) · line 101

Quoted 4 lines
Config.mk +4 -0
+                                 grep -e "$$pat" -e "$$opt" -e "$$arg" -`"; \
+                    then echo "$(2)"; \
+                    else echo "$(3)"; \
+                    fi;)

This patch looks fine. Shouldn't the comment been updated as well? At the moment, it only discuss about -Wno-*, which it seems is why grep was introduced in the first place.

Right, but that has been an issue already before.

Quoted 2 lines

But isn't it doing doing pattern matching on an error message going to lead sometime to false positive?

There's a certain risk, of course.

Quoted 10 lines

Linux's build system seems to works
fine by just using the exit value. They've got a few trick to deal with -Wno-* and with clang.

For -Wno-$(warning), they test -W$(warning) instead. For clang, they've enable additional warnings:
-Werror=unknown-warning-option -Werror=ignored-optimization-argument -Werror=option-ignored -Werror=unused-command-line-argument

Ah, yes, this would likely be a better way to test things. Time to redo what was done 12 years ago. I guess for the purpose of this series I'll keep what I have, but take note to rework things afterwards (which now would likely mean post-4.18, as the new-submissions deadline has passed).

Quoted 2 lines

In any case, the patch is fine.

Thanks.

Jan

#4 lore raw

Re: [PATCH v3 1/5] build: make cc-option properly deal with unrecognized sub-options

Quoted 71 lines · Jan Beulich
Quoted 43 lines · Anthony PERARD
Quoted 42 lines · Jan Beulich

In options like -march=, it may be only the sub-option which is unrecognized by the compiler. In such an event the error message often splits option and argument, typically saying something like "bad value '<argument>' for '<option>'. Extend the grep invocation accordingly, also accounting for Clang to not mention e.g. -march at all when an incorrect argument was given for it.

To keep things halfway readable, re-wrap and re-indent the entire construct.

---
In principle -e "$$pat" could now be omitted from the grep invocation, since if that matches, both $$opt and $$arg will, too. But I thought I'd leave it for completeness.
---
v3: Fix build with make 4.3 and newer, where the treatment of \# has
changed.
v2: Further relax grep patterns for clang, which doesn't mention -march
when complaining about an invalid argument to it.

Config.mk +5 -3
@@ -8,6 +8,7 @@ endif
 comma   := ,
 open    := (
 close   := )
+sharp   := \#
 squote  := '
 #' Balancing squote, to help syntax highlighting
 empty   :=
@@ -90,9 +91,14 @@ PYTHON_PREFIX_ARG ?= --prefix="$(prefix)
 # of which would indicate an "unrecognized command-line option" warning/error.
 #
 # Usage: cflags-y += $(call cc-option,$(CC),-march=winchip-c6,-march=i586)
-cc-option = $(shell if test -z "`echo 'void*p=1;' | \
-              $(1) $(2) -c -o /dev/null -x c - 2>&1 | grep -- $(2:-Wa$(comma)%=%) -`"; \
-              then echo "$(2)"; else echo "$(3)"; fi ;)
+cc-option = $(shell pat='$(2:-Wa$(comma)%=%)'; \
+                    opt="$${pat%%=*}" arg="$${pat$(sharp)*=}"; \
+                    if test -z "`echo 'void*p=1;' | \
+                                 $(1) $(2) -c -o /dev/null -x c - 2>&1 | \
Quoted 8 lines

Config.mk PYTHON_PREFIX_ARG ?= --prefix="$(prefix) · line 101

Quoted 4 lines
Config.mk +4 -0
+                                 grep -e "$$pat" -e "$$opt" -e "$$arg" -`"; \
+                    then echo "$(2)"; \
+                    else echo "$(3)"; \
+                    fi;)

This patch looks fine. Shouldn't the comment been updated as well? At the moment, it only discuss about -Wno-*, which it seems is why grep was introduced in the first place.

Right, but that has been an issue already before.

Quoted 2 lines

But isn't it doing doing pattern matching on an error message going to lead sometime to false positive?

There's a certain risk, of course.

Quoted 10 lines

Linux's build system seems to works
fine by just using the exit value. They've got a few trick to deal with -Wno-* and with clang.

For -Wno-$(warning), they test -W$(warning) instead. For clang, they've enable additional warnings:
-Werror=unknown-warning-option -Werror=ignored-optimization-argument -Werror=option-ignored -Werror=unused-command-line-argument

Quoted 4 lines

Ah, yes, this would likely be a better way to test things. Time to redo what was done 12 years ago. I guess for the purpose of this series I'll keep what I have, but take note to rework things afterwards (which now would likely mean post-4.18, as the new-submissions deadline has passed).

Hmm, or maybe I could simply call this v4 then, especially when directly integrating -Wno-* handling right here (by suitably using $(patsubst ...). The extra Clang aspects (if indeed needed; didn't check yet) may not easily be coverable that way, though.

Jan

#5 lore raw

Re: [PATCH v3 1/5] build: make cc-option properly deal with unrecognized sub-options

In reply to Anthony PERARD
Quoted 57 lines · Anthony PERARD
Quoted 36 lines · Jan Beulich

In options like -march=, it may be only the sub-option which is unrecognized by the compiler. In such an event the error message often splits option and argument, typically saying something like "bad value '<argument>' for '<option>'. Extend the grep invocation accordingly, also accounting for Clang to not mention e.g. -march at all when an incorrect argument was given for it.

To keep things halfway readable, re-wrap and re-indent the entire construct.

---
In principle -e "$$pat" could now be omitted from the grep invocation, since if that matches, both $$opt and $$arg will, too. But I thought I'd leave it for completeness.
---
v3: Fix build with make 4.3 and newer, where the treatment of \# has
changed.
v2: Further relax grep patterns for clang, which doesn't mention -march
when complaining about an invalid argument to it.

Config.mk +1 -1
@@ -8,6 +8,7 @@ endif
 comma   := ,
 open    := (
 close   := )
+sharp   := \#
 squote  := '
 #' Balancing squote, to help syntax highlighting
 empty   :=
@@ -90,9 +91,14 @@ PYTHON_PREFIX_ARG ?= --prefix="$(prefix)
 # of which would indicate an "unrecognized command-line option" warning/error.
 #
 # Usage: cflags-y += $(call cc-option,$(CC),-march=winchip-c6,-march=i586)
-cc-option = $(shell if test -z "`echo 'void*p=1;' | \

Config.mk PYTHON_PREFIX_ARG ?= --prefix="$(prefix) · line 101

Quoted 10 lines
Config.mk +8 -2
-              $(1) $(2) -c -o /dev/null -x c - 2>&1 | grep -- $(2:-Wa$(comma)%=%) -`"; \
-              then echo "$(2)"; else echo "$(3)"; fi ;)
+cc-option = $(shell pat='$(2:-Wa$(comma)%=%)'; \
+                    opt="$${pat%%=*}" arg="$${pat$(sharp)*=}"; \
+                    if test -z "`echo 'void*p=1;' | \
+                                 $(1) $(2) -c -o /dev/null -x c - 2>&1 | \
+                                 grep -e "$$pat" -e "$$opt" -e "$$arg" -`"; \
+                    then echo "$(2)"; \
+                    else echo "$(3)"; \
+                    fi;)

This patch looks fine. Shouldn't the comment been updated as well? At the moment, it only discuss about -Wno-*, which it seems is why grep was introduced in the first place.

But isn't it doing doing pattern matching on an error message going to lead sometime to false positive? Linux's build system seems to works fine by just using the exit value. They've got a few trick to deal with -Wno-* and with clang.

Quoted 6 lines

For -Wno-$(warning), they test -W$(warning) instead. For clang, they've enable additional warnings:
-Werror=unknown-warning-option -Werror=ignored-optimization-argument -Werror=option-ignored -Werror=unused-command-line-argument

I think using just -Werror is going to be enough. The completely changed patch below appears to be doing fine, but of course requires me to drop
...

Quoted 2 lines

In any case, the patch is fine.

... this.

Jan

Config.mk +7 -7
@@ -81,17 +81,17 @@ PYTHON_PREFIX_ARG ?= --prefix="$(prefix)
 
 # cc-option: Check if compiler supports first option, else fall back to second.
 #
-# This is complicated by the fact that unrecognised -Wno-* options:
+# This is complicated by the fact that with most gcc versions unrecognised
+# -Wno-* options:
 #   (a) are ignored unless the compilation emits a warning; and
 #   (b) even then produce a warning rather than an error
-# To handle this we do a test compile, passing the option-under-test, on a code
-# fragment that will always produce a warning (integer assigned to pointer).
-# We then grep for the option-under-test in the compiler's output, the presence
-# of which would indicate an "unrecognized command-line option" warning/error.
+# Further Clang also only warns for unrecognised -W* options.  To handle this
+# we do a test compile, substituting -Wno-* by -W* and adding -Werror.  This
+# way all unrecognised options are diagnosed uniformly, allowing us to merely
+# check exit status.
 #
 # Usage: cflags-y += $(call cc-option,$(CC),-march=winchip-c6,-march=i586)
-cc-option = $(shell if test -z "`echo 'void*p=1;' | \
-              $(1) $(2) -c -o /dev/null -x c - 2>&1 | grep -- $(2:-Wa$(comma)%=%) -`"; \
+cc-option = $(shell if $(1) $(2:-Wno-%=-W%) -Werror -c -o /dev/null -x c /dev/null >/dev/null 2>&1; \
               then echo "$(2)"; else echo "$(3)"; fi ;)
 
 # cc-option-add: Add an option to compilation flags, but only if supported.
Reviewed-by
#6 lore raw

Re: [PATCH v3 1/5] build: make cc-option properly deal with unrecognized sub-options

Quoted 6 lines · Jan Beulich
Quoted 5 lines · Anthony PERARD

But isn't it doing doing pattern matching on an error message going to lead sometime to false positive? Linux's build system seems to works fine by just using the exit value. They've got a few trick to deal with -Wno-* and with clang.

Quoted 8 lines
Quoted 6 lines

For -Wno-$(warning), they test -W$(warning) instead. For clang, they've enable additional warnings:
-Werror=unknown-warning-option -Werror=ignored-optimization-argument -Werror=option-ignored -Werror=unused-command-line-argument

I think using just -Werror is going to be enough. The completely changed

Yes, looks like -Werror is enough. I'm not sure why Linux has them because they tests flags with -Werror in most cases.

Quoted 25 lines

patch below appears to be doing fine, but of course requires me to drop
...

Quoted 2 lines

In any case, the patch is fine.

... this.

Jan

Config.mk +2 -5
@@ -81,17 +81,17 @@ PYTHON_PREFIX_ARG ?= --prefix="$(prefix)
 
 # cc-option: Check if compiler supports first option, else fall back to second.
 #
-# This is complicated by the fact that unrecognised -Wno-* options:
+# This is complicated by the fact that with most gcc versions unrecognised
+# -Wno-* options:
 #   (a) are ignored unless the compilation emits a warning; and
 #   (b) even then produce a warning rather than an error
-# To handle this we do a test compile, passing the option-under-test, on a code
-# fragment that will always produce a warning (integer assigned to pointer).
-# We then grep for the option-under-test in the compiler's output, the presence
-# of which would indicate an "unrecognized command-line option" warning/error.

Config.mk PYTHON_PREFIX_ARG ?= --prefix="$(prefix) · line 95

Quoted 10 lines
Config.mk +5 -2
+# Further Clang also only warns for unrecognised -W* options.  To handle this
+# we do a test compile, substituting -Wno-* by -W* and adding -Werror.  This
+# way all unrecognised options are diagnosed uniformly, allowing us to merely
+# check exit status.
 #
 # Usage: cflags-y += $(call cc-option,$(CC),-march=winchip-c6,-march=i586)
-cc-option = $(shell if test -z "`echo 'void*p=1;' | \
-              $(1) $(2) -c -o /dev/null -x c - 2>&1 | grep -- $(2:-Wa$(comma)%=%) -`"; \
+cc-option = $(shell if $(1) $(2:-Wno-%=-W%) -Werror -c -o /dev/null -x c /dev/null >/dev/null 2>&1; \
               then echo "$(2)"; else echo "$(3)"; fi ;)

I've try to compare the result of cc-option with and without this change in the gitlab CI, and it seems that the result is the same for the flags we tests. So this change looks fine:

Thanks,

Signature · 1 line
Anthony PERARD
#7 lore raw

PATCH v3 2/5 build: permit Kconfig control over how to deal with unsatisfiable choices

In reply to Jan Beulich

Some options we allow the build admin to select may require new enough tool chain components to fulfill (partly or entirely). Provide yet another control to pick what action to take at the end of the build process - be silent about this, warn, or fail the build.

---
This may not be fine grained enough: Optimization settings (like added by "x86: allow Kconfig control over psABI level") may want dealing with differently than security relevant ones (like XEN_SHSTK or XEN_IBT).

Whether to do this uniformly at the end of the build is up for
discussion: In the "warn" case we will want the resulting output late, so it is more likely to be noticed. In the "fail build" case though we may want the failure to occur early.
---
v3: New.

xen/Kconfig +19 -0
@@ -64,6 +64,25 @@ config UNSUPPORTED
 	  preview features as defined by SUPPORT.md. (Note that if an option
 	  doesn't depend on UNSUPPORTED it doesn't imply that is supported.)
 
+choice
+	prompt "How to deal with settings which cannot be satisified"
+	default UNSATISFIED_WARNING
+	help
+	  Some selectable options may depend on e.g. tool chain functionality.
+	  Select here how to deal with such when actually building a such
+	  configured hypervisor.
+
+config UNSATISFIED_SILENT
+	bool "silent"
+
+config UNSATISFIED_WARNING
+	bool "emit warnings"
+
+config UNSATISFIED_ERROR
+	bool "fail the build"
+
+endchoice
+
 config LTO
 	bool "Link Time Optimisation"
 	depends on BROKEN
xen/Makefile +4 -0
@@ -588,6 +588,10 @@ $(TARGET): outputmakefile FORCE
 	$(Q)$(MAKE) $(build)=arch/$(SRCARCH) include
 	$(Q)$(MAKE) $(build)=. arch/$(SRCARCH)/include/asm/asm-offsets.h
 	$(Q)$(MAKE) $(build)=. MKRELOC=$(MKRELOC) 'ALL_OBJS=$(ALL_OBJS-y)' 'ALL_LIBS=$(ALL_LIBS-y)' $@
+ifneq ($(CONFIG_UNSATISFIED_SILENT),y)
+	$(Q)$(if $(strip $(XEN_CONFIG_UNSATISFIED)),$(MAKE),:) \
+	    $(build)=. 'XEN_CONFIG_UNSATISFIED=$(XEN_CONFIG_UNSATISFIED)' check_unsatisfied
+endif
 
 SUBDIRS = xsm arch common crypto drivers lib test
 define all_sources
xen/build.mk +8 -0
@@ -88,3 +88,11 @@ targets += prelink.o
 
 $(TARGET): prelink.o FORCE
 	$(Q)$(MAKE) $(build)=arch/$(SRCARCH) $@
+
+PHONY += check_unsatisfied
+check_unsatisfied:
+	$(Q): $(if $(filter y,$(CONFIG_UNSATISFIED_WARNING)), \
+	           $(warning The following selections could not be satisfied:), \
+	           $(shell echo 'The following selections could not be satisfied:' >&2)) \
+	      $(foreach c,$(sort $(XEN_CONFIG_UNSATISFIED)),$(shell echo ' - CONFIG_$c' >&2)) \
+	      $(if $(filter y,$(CONFIG_UNSATISFIED_ERROR)),$(error Failing build))
#8 lore raw

PATCH v3 3/5 x86: allow Kconfig control over psABI level

In reply to Jan Beulich

Newer hardware offers more efficient and/or flexible and/or capable instructions, some of which we can make good use of in the hypervisor as well. Allow a basic way (no alternatives patching) of enabling their use. Of course this means that hypervisors thus built won't work anymore on older, less capable hardware.

Since older compilers (apparently gcc10 / clang11 and older) won't recognize -march=x86-64-v2 etc, also addd fallback logic passing -mpopcnt and alike explicitly.

Note that in efi_arch_cpu() the filling of 7c0 and 7d0 are forward- looking; we only require 7b0, but we need to use cpuid_count() anyway.

---
TBD: While the added assembly code goes strictly from the CONFIG_*
settings, EFI code uses the compiler provided manifest constants where available (just like the subsequent "x86: short-circuit certain cpu_has_* when x86-64-v{2,3} are in effect"). While I generally prefer it that way, this comes with a downside: If we ever started to use one of the ISA extensions in assembly code (and then keyed to the CONFIG_* settings), things could break. I don't view us doing this as very likely though, as then we'd need to have two (or more) variants of such assembly code, which we would likely want to avoid. (What I'd like to avoid is using the compiler manifest constants in assembly code: In principle we ought to filter out any -march= when constructing AFLAGS, and perhaps at least all -m... and -f... options. Compilers might rightfully complain about their use as inapplicable, much like we've seen for -Wl,... when no linking is done.)

TBD: While we don't document most "cpuid=no-*" options (and hence imply
their use to be unsupported), using e.g. "cpuid=no-popcnt" with a V2=y hypervisor clearly can't have the intended effect, and hence might perhaps better be flagged in some way.

TBD: v2 also includes LAHF/SAHF. Since we don't use floating point and
hence FPU insns, we ought to be okay not explicitly checking for it. But there is a certain amount of risk of the compiler finding some "smart" use for one or both of the insns. However, if we were to check the feature, we'd need to account for the quirk that init_amd() also works around.

Whereas the baseline -> v2 step isn't much of a difference (we'll gain more there by a subsequent patch), v2 -> v3, while presumably (or shall I say hopefully) faster, yields an overall growth of .text size by (in my build) about 2k. The primary reason for this appear to be conversions of SHL-by-immediate to SHLX.

The VGA output attempt in early (MB/MB2) boot code does not appear to work (anymore?). The serial output may work, but only if - without any setup in Xen - both sides agree on the serial settings (baud rate etc). Hence the feature checks added on the legacy paths are of limited use.
---
v3: Add fallback logic and use logic from new prereq change.
v2: Also cover XSAVE. Add early boot feature checking.

xen/arch/x86/Kconfig +30 -0
@@ -118,6 +118,36 @@ config HVM
 
 	  If unsure, say Y.
 
+choice
+	prompt "base psABI level"
+	default X86_64_BASELINE
+	help
+	  The psABI defines 4 levels of ISA extension sets as a coarse granular
+	  way of identifying advanced functionality that would be uniformly
+	  available in respectively newer hardware.  While v4 is not really of
+	  interest for Xen, the others can be selected here, making the
+	  resulting Xen no longer work on older hardware.  This option won't
+	  have any effect if the toolchain doesn't support the distinction.
+
+	  If unsure, stick to the default.
+
+config X86_64_BASELINE
+	bool "baseline"
+
+config X86_64_V2
+	bool "v2"
+	help
+	  This enables POPCNT and CX16, besides other extensions which are of
+	  no interest here.
+
+config X86_64_V3
+	bool "v3"
+	help
+	  This enables BMI, BMI2, LZCNT, MOVBE, and XSAVE, besides other
+	  extensions which are of no interest here.
+
+endchoice
+
 config XEN_SHSTK
 	bool "Supervisor Shadow Stacks"
 	depends on HAS_AS_CET_SS
xen/arch/x86/arch.mk +23 -0
@@ -36,6 +36,29 @@ CFLAGS += -mno-red-zone -fpic
 # the SSE setup for variadic function calls.
 CFLAGS += -mno-mmx -mno-sse $(call cc-option,$(CC),-mskip-rax-setup)
 
+# Enable the selected baseline ABI, if supported by the compiler.
+x86-64-$(CONFIG_X86_64_BASELINE) :=
+x86-64-$(CONFIG_X86_64_V2) := x86-64-v2
+x86-64-$(CONFIG_X86_64_V3) := x86-64-v3
+
+ifneq ($(x86-64-y),)
+CFLAGS-psabi := $(call cc-option,$(CC),-march=$(x86-64-y))
+
+ifeq ($(CFLAGS-psabi),)
+# Fall back to using the subset of -m... options that are of interest.
+x86-64-v2 := -mpopcnt -mcx16
+x86-64-v3 := $(x86-64-v2) -mbmi -mbmi2 -mlzcnt -mmovbe -mxsave
+$(call cc-options-add,CFLAGS-psabi,CC,$($(x86-64-y)))
+
+ifneq ($(strip $(CFLAGS-psabi)),$($(x86-64-y)))
+$(warning Options not recognized by $(CC): $(filter-out $(CFLAGS-psabi),$($(x86-64-y))))
+XEN_CONFIG_UNSATISFIED += X86_64_Vn
+endif
+endif # CFLAGS-psabi
+
+CFLAGS += $(CFLAGS-psabi)
+endif # x86-64-y
+
 ifeq ($(CONFIG_INDIRECT_THUNK),y)
 # Compile with gcc thunk-extern, indirect-branch-register if available.
 CFLAGS-$(CONFIG_CC_IS_GCC) += -mindirect-branch=thunk-extern
xen/arch/x86/boot/head.S +51 -0

84 lines not on this page · raw message

xen/arch/x86/efi/efi-boot.h +42 -18

71 lines not on this page · raw message

Reviewed-by on v3 3/5
#9 lore raw
Quoted 7 lines · Jan Beulich

Newer hardware offers more efficient and/or flexible and/or capable instructions, some of which we can make good use of in the hypervisor as well. Allow a basic way (no alternatives patching) of enabling their use. Of course this means that hypervisors thus built won't work anymore on older, less capable hardware.

Quoted 8 lines

Since older compilers (apparently gcc10 / clang11 and older) won't recognize -march=x86-64-v2 etc, also addd fallback logic passing -mpopcnt and alike explicitly.

Note that in efi_arch_cpu() the filling of 7c0 and 7d0 are forward- looking; we only require 7b0, but we need to use cpuid_count() anyway.

Thanks,
Jason

#10 lore raw
In reply to Jan Beulich
Quoted 52 lines · Jan Beulich
xen/arch/x86/Kconfig +30 -0
@@ -118,6 +118,36 @@ config HVM
 
 	  If unsure, say Y.
 
+choice
+	prompt "base psABI level"
+	default X86_64_BASELINE
+	help
+	  The psABI defines 4 levels of ISA extension sets as a coarse granular
+	  way of identifying advanced functionality that would be uniformly
+	  available in respectively newer hardware.  While v4 is not really of
+	  interest for Xen, the others can be selected here, making the
+	  resulting Xen no longer work on older hardware.  This option won't
+	  have any effect if the toolchain doesn't support the distinction.
+
+	  If unsure, stick to the default.
+
+config X86_64_BASELINE
+	bool "baseline"
+
+config X86_64_V2
+	bool "v2"
+	help
+	  This enables POPCNT and CX16, besides other extensions which are of
+	  no interest here.
+
+config X86_64_V3
+	bool "v3"
+	help
+	  This enables BMI, BMI2, LZCNT, MOVBE, and XSAVE, besides other
+	  extensions which are of no interest here.
+
+endchoice
+
 config XEN_SHSTK
 	bool "Supervisor Shadow Stacks"
 	depends on HAS_AS_CET_SS
xen/arch/x86/arch.mk +7 -0
@@ -36,6 +36,29 @@ CFLAGS += -mno-red-zone -fpic
 # the SSE setup for variadic function calls.
 CFLAGS += -mno-mmx -mno-sse $(call cc-option,$(CC),-mskip-rax-setup)
 
+# Enable the selected baseline ABI, if supported by the compiler.
+x86-64-$(CONFIG_X86_64_BASELINE) :=
+x86-64-$(CONFIG_X86_64_V2) := x86-64-v2
+x86-64-$(CONFIG_X86_64_V3) := x86-64-v3
+
+ifneq ($(x86-64-y),)
+CFLAGS-psabi := $(call cc-option,$(CC),-march=$(x86-64-y))

xen/arch/x86/arch.mk CFLAGS += -mno-red-zone -fpic · line 55

Quoted 10 lines
xen/arch/x86/arch.mk +10 -0
+
+ifeq ($(CFLAGS-psabi),)
+# Fall back to using the subset of -m... options that are of interest.
+x86-64-v2 := -mpopcnt -mcx16
+x86-64-v3 := $(x86-64-v2) -mbmi -mbmi2 -mlzcnt -mmovbe -mxsave
+$(call cc-options-add,CFLAGS-psabi,CC,$($(x86-64-y)))
+
+ifneq ($(strip $(CFLAGS-psabi)),$($(x86-64-y)))
+$(warning Options not recognized by $(CC): $(filter-out $(CFLAGS-psabi),$($(x86-64-y))))
+XEN_CONFIG_UNSATISFIED += X86_64_Vn

I think I understand Andrew's point on the previous version of this patch. Here, we barely just print a warning if the compiler isn't capable to do what we ask, then do the build and maybe print something after the fact. So we can end up with a build of Xen with an embedded .config which says X86_64_V3 where in fact it's only a build with the baseline ABI.

Can't we just fail the build right here instead?

I don't see any need to let Kconfig know if the compiler can do x86-64-vX or not, beside only presenting actual available options in make *config, and it would override options selected by editing .config directly.

xen/arch/x86/arch.mk CFLAGS += -mno-red-zone -fpic · line 60

Quoted 5 lines
xen/arch/x86/arch.mk +5 -0
+endif
+endif # CFLAGS-psabi
+
+CFLAGS += $(CFLAGS-psabi)
+endif # x86-64-y

Cheers,

Signature · 1 line
Anthony PERARD
#11 lore raw
Quoted 23 lines · Anthony PERARD
Quoted 22 lines · Jan Beulich
xen/arch/x86/arch.mk +16 -0
@@ -36,6 +36,29 @@ CFLAGS += -mno-red-zone -fpic
 # the SSE setup for variadic function calls.
 CFLAGS += -mno-mmx -mno-sse $(call cc-option,$(CC),-mskip-rax-setup)
 
+# Enable the selected baseline ABI, if supported by the compiler.
+x86-64-$(CONFIG_X86_64_BASELINE) :=
+x86-64-$(CONFIG_X86_64_V2) := x86-64-v2
+x86-64-$(CONFIG_X86_64_V3) := x86-64-v3
+
+ifneq ($(x86-64-y),)
+CFLAGS-psabi := $(call cc-option,$(CC),-march=$(x86-64-y))
+
+ifeq ($(CFLAGS-psabi),)
+# Fall back to using the subset of -m... options that are of interest.
+x86-64-v2 := -mpopcnt -mcx16
+x86-64-v3 := $(x86-64-v2) -mbmi -mbmi2 -mlzcnt -mmovbe -mxsave
+$(call cc-options-add,CFLAGS-psabi,CC,$($(x86-64-y)))
+
+ifneq ($(strip $(CFLAGS-psabi)),$($(x86-64-y)))
+$(warning Options not recognized by $(CC): $(filter-out $(CFLAGS-psabi),$($(x86-64-y))))
Quoted 8 lines

xen/arch/x86/arch.mk CFLAGS += -mno-red-zone -fpic · line 55

Quoted 1 line
xen/arch/x86/arch.mk +1 -0
+XEN_CONFIG_UNSATISFIED += X86_64_Vn

I think I understand Andrew's point on the previous version of this patch. Here, we barely just print a warning if the compiler isn't capable to do what we ask, then do the build and maybe print something after the fact. So we can end up with a build of Xen with an embedded .config which says X86_64_V3 where in fact it's only a build with the baseline ABI.

Which isn't really a problem, is it? Even if passed the respective option, the compiler may equally choose to use none of the thus available insns. (It won't typically, but it still could.)

Quoted 1 line

Can't we just fail the build right here instead?

Whether to do so is not the user's choice (see patch 2). See also both post-commit-message remarks there.

Quoted 4 lines

I don't see any need to let Kconfig know if the compiler can do x86-64-vX or not, beside only presenting actual available options in make *config, and it would override options selected by editing .config directly.

I'm afraid I don't understand what you're suggesting here that I do differently.

Jan

#12 lore raw

PATCH v3 4/5 x86: use POPCNT for hweight<N>() when available

In reply to Jan Beulich

This is faster than using the software implementation, and the insn is available on all half-way recent hardware. Use the respective compiler builtins when available.

xen/arch/x86/include/asm/bitops.h +7 -0
@@ -475,9 +475,16 @@ static inline int fls(unsigned int x)
  *
  * The Hamming Weight of a number is the total number of bits set in it.
  */
+#ifdef __POPCNT__
+#define hweight64(x) __builtin_popcountll(x)
+#define hweight32(x) __builtin_popcount(x)
+#define hweight16(x) __builtin_popcount((uint16_t)(x))
+#define hweight8(x)  __builtin_popcount((uint8_t)(x))
+#else
 #define hweight64(x) generic_hweight64(x)
 #define hweight32(x) generic_hweight32(x)
 #define hweight16(x) generic_hweight16(x)
 #define hweight8(x) generic_hweight8(x)
+#endif
 
 #endif /* _X86_BITOPS_H */
#13 lore raw

PATCH v3 5/5 x86: short-circuit certain cpu_has_* when x86-64-v{2,3} are in effect

In reply to Jan Beulich

Certain fallback code can be made subject to DCE this way. Note that CX16 has no compiler provided manifest constant, so CONFIG_* are used there instead. Note also that we don't have cpu_has_movbe nor cpu_has_lzcnt (aka cpu_has_abm).

---
Of course we could use IS_ENABLED(CONFIG_X86_64_V<n>) everywhere, but as CX16 shows this isn't necessarily better than the #if/#else approach based on compiler-provided manifest symbols. While not really intended to be used that way, it looks as if we could also use IS_ENABLED(__POPCNT__) and alike if we thought this would end up neater (because of avoiding the #ifdef).

We could go further and also short-circuit SSE*, AVX and alike, which we don't use outside of the emulator. This then would of course call for also having a way to select x86-64-v4.
---
v2: Also cover XSAVE.

xen/arch/x86/include/asm/cpufeature.h +19 -1
@@ -76,15 +76,25 @@ static inline bool boot_cpu_has(unsigned
 #define cpu_has_eist            boot_cpu_has(X86_FEATURE_EIST)
 #define cpu_has_ssse3           boot_cpu_has(X86_FEATURE_SSSE3)
 #define cpu_has_fma             boot_cpu_has(X86_FEATURE_FMA)
-#define cpu_has_cx16            boot_cpu_has(X86_FEATURE_CX16)
+#define cpu_has_cx16            (IS_ENABLED(CONFIG_X86_64_V2) || \
+                                 IS_ENABLED(CONFIG_X86_64_V3) || \
+                                 boot_cpu_has(X86_FEATURE_CX16))
 #define cpu_has_pdcm            boot_cpu_has(X86_FEATURE_PDCM)
 #define cpu_has_pcid            boot_cpu_has(X86_FEATURE_PCID)
 #define cpu_has_sse4_1          boot_cpu_has(X86_FEATURE_SSE4_1)
 #define cpu_has_sse4_2          boot_cpu_has(X86_FEATURE_SSE4_2)
 #define cpu_has_x2apic          boot_cpu_has(X86_FEATURE_X2APIC)
+#ifdef __POPCNT__
+#define cpu_has_popcnt          true
+#else
 #define cpu_has_popcnt          boot_cpu_has(X86_FEATURE_POPCNT)
+#endif
 #define cpu_has_aesni           boot_cpu_has(X86_FEATURE_AESNI)
+#ifdef __XSAVE__
+#define cpu_has_xsave           true
+#else
 #define cpu_has_xsave           boot_cpu_has(X86_FEATURE_XSAVE)
+#endif
 #define cpu_has_avx             boot_cpu_has(X86_FEATURE_AVX)
 #define cpu_has_f16c            boot_cpu_has(X86_FEATURE_F16C)
 #define cpu_has_rdrand          boot_cpu_has(X86_FEATURE_RDRAND)
@@ -114,11 +124,19 @@ static inline bool boot_cpu_has(unsigned
 #define cpu_has_xsaves          boot_cpu_has(X86_FEATURE_XSAVES)
 
 /* CPUID level 0x00000007:0.ebx */
+#ifdef __BMI__
+#define cpu_has_bmi1            true
+#else
 #define cpu_has_bmi1            boot_cpu_has(X86_FEATURE_BMI1)
+#endif
 #define cpu_has_hle             boot_cpu_has(X86_FEATURE_HLE)
 #define cpu_has_avx2            boot_cpu_has(X86_FEATURE_AVX2)
 #define cpu_has_smep            boot_cpu_has(X86_FEATURE_SMEP)
+#ifdef __BMI2__
+#define cpu_has_bmi2            true
+#else
 #define cpu_has_bmi2            boot_cpu_has(X86_FEATURE_BMI2)
+#endif
 #define cpu_has_invpcid         boot_cpu_has(X86_FEATURE_INVPCID)
 #define cpu_has_rtm             boot_cpu_has(X86_FEATURE_RTM)
 #define cpu_has_pqe             boot_cpu_has(X86_FEATURE_PQE)
Reviewed-by on v3 5/5
#14 lore raw

Re: [PATCH v3 5/5] x86: short-circuit certain cpu_has_* when x86-64-v{2,3} are in effect

Quoted 7 lines · Jan Beulich

Certain fallback code can be made subject to DCE this way. Note that CX16 has no compiler provided manifest constant, so CONFIG_* are used there instead. Note also that we don't have cpu_has_movbe nor cpu_has_lzcnt (aka cpu_has_abm).

Thanks,
Jason