Skip to contentSkip to conversation

improve late microcode loading

PATCH Superseded Chao Gao 5 participants 53 messages May 27, 2019 – Jun 14, 2019

v7 of 11 Superseded Latest is v11 →

lore
Share & tools
lore

Chao Gao

author
#0 lore raw

v7 cover improve late microcode loading

Changes in version 7:
- cache one microcode update rather than a list of it. Assuming that all CPUs (including those will be plugged in later) in the system have the same signature, one update matches with one CPU should match with others. Thus, one update is enough for microcode updating during CPU hot-plug and resuming.
- To handle load failure, microcode update is cached after it is applied to avoid a broken update overriding a validated one. Unvalidated microcode updates are passed by arguments rather than another global variable, where this series slightly differs from Roger's suggestion in:
https://lists.xen.org/archives/html/xen-devel/2019-03/msg00776.html
- incorporate Sergey's patch (patch 10) to fix a bug: we maintain a variable to reflect current microcode revision. But in some cases, this variable isn't initialized during system boot time, which results in falsely reporting that processor is susceptible to some known vulnerabilities.
- fix issues reported by Sergey:
https://lists.xenproject.org/archives/html/xen-devel/2019-03/msg00901.html
- Responses to Sergey/Roger/Wei/Ashok's other comments.

Major changes in version 6:
- run wbinvd before updating microcode (patch 10)
- add an userspace tool for late microcode update (patch 1)
- scale time to wait by the number of remaining CPUs to respond
- remove 'cpu' parameters from some related callbacks and functins
- save an ucode patch only if its supported CPU is allowed to mix with current cpu.

Changes in version 5:
- support parallel microcode updates for all cores (see patch 8)
- Address Roger's comments on the last version.

The intention of this series is to make the late microcode loading more reliable by rendezvousing all cpus in stop_machine context. This idea comes from Ashok. I am porting his linux patch to Xen (see patch 7 and 8 for more details).

This series includes five changes:
1. Patch 1: an userspace tool for late microcode update
2. Patch 2-6: introduce a global microcode cache and some cleanup
3. Patch 7: writeback and invalidate cache before updating microcode
3. Patch 8: synchronize late microcode loading
4. Patch 9: support parallel microcodes update on different cores
5. Patch 10: always read microcode revision at boot time

Currently, late microcode loading does a lot of things including parsing microcode blob, checking the signature/revision and performing update. Putting all of them into stop_machine context is a bad idea because of complexity (one issue I observed is memory allocation triggered one assertion in stop_machine context). To simplify the load process, parsing microcode is moved out of the load process. Remaining parts of load process is put to stop_machine context.

Regarding changes to AMD side, I didn't do any test for them due to lack of hardware. Sergey, could you help to test this series on an AMD machine again?
At least, two basic tests are needed:
* do a microcode update after system bootup
* don't bring all pCPUs up at bootup by specifying maxcpus option in xen command line and then do a microcode update and online all offlined CPUs via 'xen-hptool'.

Chao Gao (9):
misc/xen-ucode: Upload a microcode blob to the hypervisor microcode/intel: extend microcode_update_match()
microcode: introduce a global cache of ucode patch
microcode: remove struct ucode_cpu_info
microcode: remove pointless 'cpu' parameter
microcode: split out apply_microcode() from cpu_request_microcode() microcode/intel: Writeback and invalidate caches before updating
microcode
x86/microcode: Synchronize late microcode loading
microcode: remove microcode_update_lock

Sergey Dyasli (1):
x86/microcode: always collect_cpu_info() during boot

 tools/libxc/include/xenctrl.h   |   1 +
 tools/libxc/xc_misc.c           |  23 +++
 tools/misc/Makefile             |   4 +
 tools/misc/xen-ucode.c          |  78 ++++++++
 xen/arch/x86/acpi/power.c       |   2 +-
 xen/arch/x86/apic.c             |   2 +-
 xen/arch/x86/microcode.c        | 401 ++++++++++++++++++++++++++++------------
 xen/arch/x86/microcode_amd.c    | 245 ++++++++++++------------
 xen/arch/x86/microcode_intel.c  | 202 ++++++++++----------
 xen/arch/x86/smpboot.c          |   5 +-
 xen/arch/x86/spec_ctrl.c        |   2 +-
 xen/include/asm-x86/microcode.h |  39 ++--
 xen/include/asm-x86/processor.h |   3 +-
 13 files changed, 639 insertions(+), 368 deletions(-)
 create mode 100644 tools/misc/xen-ucode.c
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#1 lore raw

PATCH v7 1/10 misc/xen-ucode: Upload a microcode blob to the hypervisor

This patch provides a tool for late microcode update.

---
Changes in v7:
- introduce xc_microcode_update() rather than xc_platform_op()
- avoid creating bounce buffer twice
- rename xenmicrocode to xen-ucode, following naming tradition of other tools there.

---
 tools/libxc/include/xenctrl.h |  1 +
 tools/libxc/xc_misc.c         | 23 +++++++++++++
 tools/misc/Makefile           |  4 +++
 tools/misc/xen-ucode.c        | 78 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 106 insertions(+)
 create mode 100644 tools/misc/xen-ucode.c
tools/libxc/include/xenctrl.h +1 -0
index 538007a..6d80ae5 100644
@@ -1244,6 +1244,7 @@ typedef uint32_t xc_node_to_node_dist_t;
 int xc_physinfo(xc_interface *xch, xc_physinfo_t *info);
 int xc_cputopoinfo(xc_interface *xch, unsigned *max_cpus,
                    xc_cputopo_t *cputopo);
+int xc_microcode_update(xc_interface *xch, const void *buf, size_t len);
 int xc_numainfo(xc_interface *xch, unsigned *max_nodes,
                 xc_meminfo_t *meminfo, uint32_t *distance);
 int xc_pcitopoinfo(xc_interface *xch, unsigned num_devs,
tools/libxc/xc_misc.c +23 -0
index 5e6714a..85538e0 100644
@@ -226,6 +226,29 @@ int xc_physinfo(xc_interface *xch,
     return 0;
 }
 
+int xc_microcode_update(xc_interface *xch, const void *buf, size_t len)
+{
+    int ret;
+    DECLARE_PLATFORM_OP;
+    DECLARE_HYPERCALL_BUFFER(struct xenpf_microcode_update, uc);
+
+    uc = xc_hypercall_buffer_alloc(xch, uc, len);
+    if (uc == NULL)
+        return -1;
+
+    memcpy(uc, buf, len);
+
+    platform_op.cmd = XENPF_microcode_update;
+    platform_op.u.microcode.length = len;
+    set_xen_guest_handle(platform_op.u.microcode.data, uc);
+
+    ret = do_platform_op(xch, &platform_op);
+
+    xc_hypercall_buffer_free(xch, uc);
+
+    return ret;
+}
+
 int xc_cputopoinfo(xc_interface *xch, unsigned *max_cpus,
                    xc_cputopo_t *cputopo)
 {
tools/misc/Makefile +4 -0
index d4320dc..63947bf 100644
@@ -22,6 +22,7 @@ INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmcrash
 INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmctx
 INSTALL_SBIN-$(CONFIG_X86)     += xen-lowmemd
 INSTALL_SBIN-$(CONFIG_X86)     += xen-mfndump
+INSTALL_SBIN-$(CONFIG_X86)     += xen-ucode
 INSTALL_SBIN                   += xencov
 INSTALL_SBIN                   += xenlockprof
 INSTALL_SBIN                   += xenperf
@@ -113,4 +114,7 @@ xen-lowmemd: xen-lowmemd.o
 xencov: xencov.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
 
+xen-ucode: xen-ucode.o
+	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
+
 -include $(DEPS_INCLUDE)
tools/misc/xen-ucode.c new file +78 -0

78 lines not on this page · raw message

Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Acked-by on v7 1/10
#2 lore raw

Re: [Xen-devel] [PATCH v7 01/10] misc/xen-ucode: Upload a microcode blob to the hypervisor

Quoted 37 lines · Chao Gao

This patch provides a tool for late microcode update.

---
Changes in v7:
- introduce xc_microcode_update() rather than xc_platform_op()
- avoid creating bounce buffer twice
- rename xenmicrocode to xen-ucode, following naming tradition of other tools there.

---
 tools/libxc/include/xenctrl.h |  1 +
 tools/libxc/xc_misc.c         | 23 +++++++++++++
 tools/misc/Makefile           |  4 +++
 tools/misc/xen-ucode.c        | 78 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 106 insertions(+)
 create mode 100644 tools/misc/xen-ucode.c
tools/libxc/include/xenctrl.h +1 -0
index 538007a..6d80ae5 100644
@@ -1244,6 +1244,7 @@ typedef uint32_t xc_node_to_node_dist_t;
 int xc_physinfo(xc_interface *xch, xc_physinfo_t *info);
 int xc_cputopoinfo(xc_interface *xch, unsigned *max_cpus,
                    xc_cputopo_t *cputopo);
+int xc_microcode_update(xc_interface *xch, const void *buf, size_t len);
 int xc_numainfo(xc_interface *xch, unsigned *max_nodes,
                 xc_meminfo_t *meminfo, uint32_t *distance);
 int xc_pcitopoinfo(xc_interface *xch, unsigned num_devs,
tools/libxc/xc_misc.c +0 -0
index 5e6714a..85538e0 100644
@@ -226,6 +226,29 @@ int xc_physinfo(xc_interface *xch,
     return 0;

tools/libxc/xc_misc.c int xc_physinfo(xc_interface *xch, · line 236

Quoted 10 lines
tools/libxc/xc_misc.c +8 -0
 }
 
+int xc_microcode_update(xc_interface *xch, const void *buf, size_t len)
+{
+    int ret;
+    DECLARE_PLATFORM_OP;
+    DECLARE_HYPERCALL_BUFFER(struct xenpf_microcode_update, uc);
+
+    uc = xc_hypercall_buffer_alloc(xch, uc, len);
+    if (uc == NULL)

Xen style.  Extra space please.

Quoted 53 lines
tools/libxc/xc_misc.c +15 -0
+        return -1;
+
+    memcpy(uc, buf, len);
+
+    platform_op.cmd = XENPF_microcode_update;
+    platform_op.u.microcode.length = len;
+    set_xen_guest_handle(platform_op.u.microcode.data, uc);
+
+    ret = do_platform_op(xch, &platform_op);
+
+    xc_hypercall_buffer_free(xch, uc);
+
+    return ret;
+}
+
 int xc_cputopoinfo(xc_interface *xch, unsigned *max_cpus,
                    xc_cputopo_t *cputopo)
 {
tools/misc/Makefile +4 -0
index d4320dc..63947bf 100644
@@ -22,6 +22,7 @@ INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmcrash
 INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmctx
 INSTALL_SBIN-$(CONFIG_X86)     += xen-lowmemd
 INSTALL_SBIN-$(CONFIG_X86)     += xen-mfndump
+INSTALL_SBIN-$(CONFIG_X86)     += xen-ucode
 INSTALL_SBIN                   += xencov
 INSTALL_SBIN                   += xenlockprof
 INSTALL_SBIN                   += xenperf
@@ -113,4 +114,7 @@ xen-lowmemd: xen-lowmemd.o
 xencov: xencov.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
 
+xen-ucode: xen-ucode.o
+	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
+
 -include $(DEPS_INCLUDE)
tools/misc/xen-ucode.c new file +9 -0
new file mode 100644
index 0000000..da668ca
@@ -0,0 +1,78 @@
+#define _GNU_SOURCE
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mman.h>
+#include <errno.h>
+#include <string.h>
+#include <inttypes.h>
+#include <unistd.h>

tools/misc/xen-ucode.c line 19

Quoted 10 lines
tools/misc/xen-ucode.c new file +10 -0
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <xenctrl.h>
+
+void show_help(void)
+{
+    fprintf(stderr,
+            "xenmicrocode: Xen microcode updating tool\n"
+            "Usage: xenmicrocode <microcode blob>\n");

s/xenmicrocode/xen-ucode/

Both can be fixed on commit

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

Re: [Xen-devel] [PATCH v7 01/10] misc/xen-ucode: Upload a microcode blob to the hypervisor

Quoted 18 lines
Quoted 1 line
Quoted 1 line · On 04.06.19 at 18:14, <[email protected]> wrote:
Quoted 16 lines · Chao Gao
tools/misc/xen-ucode.c new file +13 -0
@@ -0,0 +1,78 @@
+#define _GNU_SOURCE
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mman.h>
+#include <errno.h>
+#include <string.h>
+#include <inttypes.h>
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <xenctrl.h>
Quoted 8 lines

tools/misc/xen-ucode.c line 19

Quoted 6 lines
tools/misc/xen-ucode.c new file +6 -0
+
+void show_help(void)
+{
+    fprintf(stderr,
+            "xenmicrocode: Xen microcode updating tool\n"
+            "Usage: xenmicrocode <microcode blob>\n");

s/xenmicrocode/xen-ucode/

Isn't it common practice to use argv[0] for such?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#4 lore raw

Re: [Xen-devel] [PATCH v7 01/10] misc/xen-ucode: Upload a microcode blob to the hypervisor

In reply to Andrew Cooper
Quoted 117 lines · Andrew Cooper
Quoted 37 lines · Chao Gao

This patch provides a tool for late microcode update.

---
Changes in v7:
- introduce xc_microcode_update() rather than xc_platform_op()
- avoid creating bounce buffer twice
- rename xenmicrocode to xen-ucode, following naming tradition of other tools there.

---
 tools/libxc/include/xenctrl.h |  1 +
 tools/libxc/xc_misc.c         | 23 +++++++++++++
 tools/misc/Makefile           |  4 +++
 tools/misc/xen-ucode.c        | 78 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 106 insertions(+)
 create mode 100644 tools/misc/xen-ucode.c
tools/libxc/include/xenctrl.h +1 -0
index 538007a..6d80ae5 100644
@@ -1244,6 +1244,7 @@ typedef uint32_t xc_node_to_node_dist_t;
 int xc_physinfo(xc_interface *xch, xc_physinfo_t *info);
 int xc_cputopoinfo(xc_interface *xch, unsigned *max_cpus,
                    xc_cputopo_t *cputopo);
+int xc_microcode_update(xc_interface *xch, const void *buf, size_t len);
 int xc_numainfo(xc_interface *xch, unsigned *max_nodes,
                 xc_meminfo_t *meminfo, uint32_t *distance);
 int xc_pcitopoinfo(xc_interface *xch, unsigned num_devs,
tools/libxc/xc_misc.c +0 -0
index 5e6714a..85538e0 100644
@@ -226,6 +226,29 @@ int xc_physinfo(xc_interface *xch,
     return 0;

tools/libxc/xc_misc.c int xc_physinfo(xc_interface *xch, · line 236

Quoted 10 lines
tools/libxc/xc_misc.c +8 -0
 }
 
+int xc_microcode_update(xc_interface *xch, const void *buf, size_t len)
+{
+    int ret;
+    DECLARE_PLATFORM_OP;
+    DECLARE_HYPERCALL_BUFFER(struct xenpf_microcode_update, uc);
+
+    uc = xc_hypercall_buffer_alloc(xch, uc, len);
+    if (uc == NULL)

Xen style.  Extra space please.

Quoted 53 lines
tools/libxc/xc_misc.c +15 -0
+        return -1;
+
+    memcpy(uc, buf, len);
+
+    platform_op.cmd = XENPF_microcode_update;
+    platform_op.u.microcode.length = len;
+    set_xen_guest_handle(platform_op.u.microcode.data, uc);
+
+    ret = do_platform_op(xch, &platform_op);
+
+    xc_hypercall_buffer_free(xch, uc);
+
+    return ret;
+}
+
 int xc_cputopoinfo(xc_interface *xch, unsigned *max_cpus,
                    xc_cputopo_t *cputopo)
 {
tools/misc/Makefile +4 -0
index d4320dc..63947bf 100644
@@ -22,6 +22,7 @@ INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmcrash
 INSTALL_SBIN-$(CONFIG_X86)     += xen-hvmctx
 INSTALL_SBIN-$(CONFIG_X86)     += xen-lowmemd
 INSTALL_SBIN-$(CONFIG_X86)     += xen-mfndump
+INSTALL_SBIN-$(CONFIG_X86)     += xen-ucode
 INSTALL_SBIN                   += xencov
 INSTALL_SBIN                   += xenlockprof
 INSTALL_SBIN                   += xenperf
@@ -113,4 +114,7 @@ xen-lowmemd: xen-lowmemd.o
 xencov: xencov.o
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
 
+xen-ucode: xen-ucode.o
+	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
+
 -include $(DEPS_INCLUDE)
tools/misc/xen-ucode.c new file +9 -0
new file mode 100644
index 0000000..da668ca
@@ -0,0 +1,78 @@
+#define _GNU_SOURCE
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/mman.h>
+#include <errno.h>
+#include <string.h>
+#include <inttypes.h>
+#include <unistd.h>

tools/misc/xen-ucode.c line 19

Quoted 10 lines
tools/misc/xen-ucode.c new file +10 -0
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <xenctrl.h>
+
+void show_help(void)
+{
+    fprintf(stderr,
+            "xenmicrocode: Xen microcode updating tool\n"
+            "Usage: xenmicrocode <microcode blob>\n");

s/xenmicrocode/xen-ucode/

Quoted 3 lines

Both can be fixed on commit

Thanks.

As Jan said, it is better to use argv[0] here.

Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#5 lore raw

PATCH v7 2/10 microcode/intel: extend microcode_update_match()

In reply to Chao Gao

to a more generic function. Then, this function can compare two given microcodes' signature/revision as well. Comparing two microcodes is used to update the global microcode cache (introduced by the later patches in this series) when a new microcode is given.

Note that enum microcode_match_result will be used in common code (aka microcode.c), it has been placed in the common header.

---
Changes in v6:
- eliminate unnecessary type casting in microcode_update_match
- check if a patch has an extend header

Changes in v5:
- constify the extended_signature
- use named enum type for the return value of microcode_update_match

---
 xen/arch/x86/microcode_intel.c  | 48 +++++++++++++++++++----------------------
 xen/include/asm-x86/microcode.h |  6 ++++++
 2 files changed, 28 insertions(+), 26 deletions(-)
xen/arch/x86/microcode_intel.c +22 -26
index 22fdeca..ecec83b 100644
@@ -134,14 +134,28 @@ static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig)
     return 0;
 }
 
-static inline int microcode_update_match(
-    unsigned int cpu_num, const struct microcode_header_intel *mc_header,
-    int sig, int pf)
+static enum microcode_match_result microcode_update_match(
+    const struct microcode_header_intel *mc_header, unsigned int sig,
+    unsigned int pf, unsigned int rev)
 {
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu_num);
+    const struct extended_sigtable *ext_header;
+    const struct extended_signature *ext_sig;
+    unsigned long data_size = get_datasize(mc_header);
+    unsigned int i;
+
+    if ( sigmatch(sig, mc_header->sig, pf, mc_header->pf) )
+        return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;
 
-    return (sigmatch(sig, uci->cpu_sig.sig, pf, uci->cpu_sig.pf) &&
-            (mc_header->rev > uci->cpu_sig.rev));
+    if ( get_totalsize(mc_header) == (data_size + MC_HEADER_SIZE) )
+        return MIS_UCODE;
+
+    ext_header = (const void *)(mc_header + 1) + data_size;
+    ext_sig = (const void *)(ext_header + 1);
+    for ( i = 0; i < ext_header->count; i++ )
+        if ( sigmatch(sig, ext_sig[i].sig, pf, ext_sig[i].pf) )
+            return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;
+
+    return MIS_UCODE;
 }
 
 static int microcode_sanity_check(void *mc)
@@ -243,31 +257,13 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)
 {
     struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu);
     const struct microcode_header_intel *mc_header = mc;
-    const struct extended_sigtable *ext_header;
     unsigned long total_size = get_totalsize(mc_header);
-    int ext_sigcount, i;
-    struct extended_signature *ext_sig;
     void *new_mc;
 
-    if ( microcode_update_match(cpu, mc_header,
-                                mc_header->sig, mc_header->pf) )
-        goto find;
-
-    if ( total_size <= (get_datasize(mc_header) + MC_HEADER_SIZE) )
+    if ( microcode_update_match(mc, uci->cpu_sig.sig, uci->cpu_sig.pf,
+                                uci->cpu_sig.rev) != NEW_UCODE )
         return 0;
 
-    ext_header = mc + get_datasize(mc_header) + MC_HEADER_SIZE;
-    ext_sigcount = ext_header->count;
-    ext_sig = (void *)ext_header + EXT_HEADER_SIZE;
-    for ( i = 0; i < ext_sigcount; i++ )
-    {
-        if ( microcode_update_match(cpu, mc_header,
-                                    ext_sig->sig, ext_sig->pf) )
-            goto find;
-        ext_sig++;
-    }
-    return 0;
- find:
     pr_debug("microcode: CPU%d found a matching microcode update with"
              " version %#x (current=%#x)\n",
              cpu, mc_header->rev, uci->cpu_sig.rev);
xen/include/asm-x86/microcode.h +6 -0
index 23ea954..73ebe9a 100644
@@ -3,6 +3,12 @@
 
 #include <xen/percpu.h>
 
+enum microcode_match_result {
+    OLD_UCODE, /* signature matched, but revision id isn't newer */
+    NEW_UCODE, /* signature matched, but revision id is newer */
+    MIS_UCODE, /* signature mismatched */
+};
+
 struct cpu_signature;
 struct ucode_cpu_info;
 
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#6 lore raw

Re: [Xen-devel] [PATCH v7 02/10] microcode/intel: extend microcode_update_match()

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +2 -3
@@ -134,14 +134,28 @@ static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig)
     return 0;
 }
 
-static inline int microcode_update_match(
-    unsigned int cpu_num, const struct microcode_header_intel *mc_header,
-    int sig, int pf)
+static enum microcode_match_result microcode_update_match(
+    const struct microcode_header_intel *mc_header, unsigned int sig,

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 147

Quoted 10 lines
xen/arch/x86/microcode_intel.c +8 -1
+    unsigned int pf, unsigned int rev)
 {
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu_num);
+    const struct extended_sigtable *ext_header;
+    const struct extended_signature *ext_sig;
+    unsigned long data_size = get_datasize(mc_header);
+    unsigned int i;
+
+    if ( sigmatch(sig, mc_header->sig, pf, mc_header->pf) )
+        return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;

As indicated before, I think you would better also provide an "equal" indication. Iirc I've told you that I have one system where the cores get handed over from the BIOS in an inconsistent state (only core has ucode loaded). Hence we'd want to be able to also _store_ ucode matching that found on CPU 0, without actually want to _load_ it there.

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 150

Quoted 4 lines
xen/arch/x86/microcode_intel.c +2 -2
-    return (sigmatch(sig, uci->cpu_sig.sig, pf, uci->cpu_sig.pf) &&
-            (mc_header->rev > uci->cpu_sig.rev));
+    if ( get_totalsize(mc_header) == (data_size + MC_HEADER_SIZE) )
+        return MIS_UCODE;

Okay, you're tightening the original <= to == here. But if you're already tightening things, why don't you make sure you actually have enough data to ...

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 152

Quoted 1 line
xen/arch/x86/microcode_intel.c +1 -0
+    ext_header = (const void *)(mc_header + 1) + data_size;

... hold an extended header, and then also to hold ...

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 156

Quoted 4 lines
xen/arch/x86/microcode_intel.c +4 -0
+    ext_sig = (const void *)(ext_header + 1);
+    for ( i = 0; i < ext_header->count; i++ )
+        if ( sigmatch(sig, ext_sig[i].sig, pf, ext_sig[i].pf) )
+            return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;

... enough array elements?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw

Re: [Xen-devel] [PATCH v7 02/10] microcode/intel: extend microcode_update_match()

Quoted 21 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +9 -4
@@ -134,14 +134,28 @@ static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig)
     return 0;
 }
 
-static inline int microcode_update_match(
-    unsigned int cpu_num, const struct microcode_header_intel *mc_header,
-    int sig, int pf)
+static enum microcode_match_result microcode_update_match(
+    const struct microcode_header_intel *mc_header, unsigned int sig,
+    unsigned int pf, unsigned int rev)
 {
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu_num);
+    const struct extended_sigtable *ext_header;
+    const struct extended_signature *ext_sig;
+    unsigned long data_size = get_datasize(mc_header);
+    unsigned int i;
+
+    if ( sigmatch(sig, mc_header->sig, pf, mc_header->pf) )
Quoted 8 lines

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 147

Quoted 1 line
xen/arch/x86/microcode_intel.c +1 -0
+        return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;

As indicated before, I think you would better also provide an "equal" indication. Iirc I've told you that I have one system where the cores get handed over from the BIOS in an inconsistent state (only core has ucode loaded). Hence we'd want to be able to also _store_ ucode matching that found on CPU 0, without actually want to _load_ it there.

Hm, without me being an expert on microcode, isn't such a system utterly broken?

I'm not against making Xen capable of booting in this scenario where firmware leaves the CPUs with different microcode versions, but this is something that should be reported to the vendor in order to get it fixed IMO?

What happens when you don't load any microcode at all, is the system capable of operating normally with such mixed microcode?

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

Re: [Xen-devel] [PATCH v7 02/10] microcode/intel: extend microcode_update_match()

Quoted 33 lines
Quoted 1 line
Quoted 1 line · On 05.06.19 at 15:22, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num,

Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +0 -0
@@ -134,14 +134,28 @@ static int collect_cpu_info(unsigned int cpu_num, 

struct cpu_signature *csig)

Quoted 17 lines
Quoted 17 lines
xen/arch/x86/microcode_intel.c +9 -4
     return 0;
 }
 
-static inline int microcode_update_match(
-    unsigned int cpu_num, const struct microcode_header_intel *mc_header,
-    int sig, int pf)
+static enum microcode_match_result microcode_update_match(
+    const struct microcode_header_intel *mc_header, unsigned int sig,
+    unsigned int pf, unsigned int rev)
 {
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu_num);
+    const struct extended_sigtable *ext_header;
+    const struct extended_signature *ext_sig;
+    unsigned long data_size = get_datasize(mc_header);
+    unsigned int i;
+
+    if ( sigmatch(sig, mc_header->sig, pf, mc_header->pf) )
Quoted 8 lines

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, · line 147

Quoted 1 line
xen/arch/x86/microcode_intel.c +1 -0
+        return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;

As indicated before, I think you would better also provide an "equal" indication. Iirc I've told you that I have one system where the cores get handed over from the BIOS in an inconsistent state (only core has ucode loaded). Hence we'd want to be able to also _store_ ucode matching that found on CPU 0, without actually want to _load_ it there.

Quoted 2 lines

Hm, without me being an expert on microcode, isn't such a system utterly broken?

It's working fine (from all I can tell). It really depends on what exactly the ucode update changes.

Quoted 4 lines

I'm not against making Xen capable of booting in this scenario where firmware leaves the CPUs with different microcode versions, but this is something that should be reported to the vendor in order to get it fixed IMO?

I did report it, years ago.

Quoted 2 lines

What happens when you don't load any microcode at all, is the system capable of operating normally with such mixed microcode?

Yes. Obviously there are anomalies (like fixed errata of particular insns showing up or not depending on what core a process executes), but the system has been stable this way for many years. (I've always been doing early loading of ucode when running it with Xen, but the distro that's on it never managed to arrange for early ucode loading when booting a bare metal kernel).

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#9 lore raw

Re: [Xen-devel] [PATCH v7 02/10] microcode/intel: extend microcode_update_match()

In reply to Jan Beulich
Quoted 21 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +9 -4
@@ -134,14 +134,28 @@ static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig)
     return 0;
 }
 
-static inline int microcode_update_match(
-    unsigned int cpu_num, const struct microcode_header_intel *mc_header,
-    int sig, int pf)
+static enum microcode_match_result microcode_update_match(
+    const struct microcode_header_intel *mc_header, unsigned int sig,
+    unsigned int pf, unsigned int rev)
 {
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu_num);
+    const struct extended_sigtable *ext_header;
+    const struct extended_signature *ext_sig;
+    unsigned long data_size = get_datasize(mc_header);
+    unsigned int i;
+
+    if ( sigmatch(sig, mc_header->sig, pf, mc_header->pf) )
Quoted 8 lines

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 147

Quoted 1 line
xen/arch/x86/microcode_intel.c +1 -0
+        return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;

As indicated before, I think you would better also provide an "equal" indication. Iirc I've told you that I have one system where the cores get handed over from the BIOS in an inconsistent state (only core has ucode loaded). Hence we'd want to be able to also _store_ ucode matching that found on CPU 0, without actually want to _load_ it there.

Will do. What if no microcode update is provided in this case? Shall we refuse to boot? If we allow different microcode revisions in the system, it would complicate late microcode loading.

Quoted 12 lines

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 149

Quoted 5 lines
xen/arch/x86/microcode_intel.c +2 -2
-    return (sigmatch(sig, uci->cpu_sig.sig, pf, uci->cpu_sig.pf) &&
-            (mc_header->rev > uci->cpu_sig.rev));
+    if ( get_totalsize(mc_header) == (data_size + MC_HEADER_SIZE) )
+        return MIS_UCODE;

Okay, you're tightening the original <= to == here. But if you're already tightening things, why don't you make sure you actually have enough data to ...

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 150

Quoted 1 line
xen/arch/x86/microcode_intel.c +1 -0
+    ext_header = (const void *)(mc_header + 1) + data_size;
Quoted 8 lines

... hold an extended header, and then also to hold ...

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 154

Quoted 4 lines
xen/arch/x86/microcode_intel.c +4 -0
+    ext_sig = (const void *)(ext_header + 1);
+    for ( i = 0; i < ext_header->count; i++ )
+        if ( sigmatch(sig, ext_sig[i].sig, pf, ext_sig[i].pf) )
+            return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;

... enough array elements?

Do you think below incremental change is fine?

xen/arch/x86/microcode_intel.c +13 -6
index 94a1561..3dcbd28 100644
@@ -138,18 +138,25 @@ static enum microcode_match_result microcode_update_match(
     const struct extended_signature *ext_sig;
     unsigned long data_size = get_datasize(mc_header);
     unsigned int i;
+    const void *end = (const void *)mc_header + get_totalsize(mc_header);
 
     if ( sigmatch(sig, mc_header->sig, pf, mc_header->pf) )
         return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;
 
-    if ( get_totalsize(mc_header) == (data_size + MC_HEADER_SIZE) )
-        return MIS_UCODE;
-
     ext_header = (const void *)(mc_header + 1) + data_size;
     ext_sig = (const void *)(ext_header + 1);
-    for ( i = 0; i < ext_header->count; i++ )
-        if ( sigmatch(sig, ext_sig[i].sig, pf, ext_sig[i].pf) )
-            return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;
+
+    /*
+     * Make sure there is enough space to hold an extended header and enough
+     * array elements.
+     */
+    if ( (end >= (const void *)ext_sig) &&
+         (end >= (const void *)(ext_sig + ext_header->count)) )
+    {
+        for ( i = 0; i < ext_header->count; i++ )
+            if ( sigmatch(sig, ext_sig[i].sig, pf, ext_sig[i].pf) )
+                return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;
+    }
 
     return MIS_UCODE;
 }

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

Re: [Xen-devel] [PATCH v7 02/10] microcode/intel: extend microcode_update_match()

Quoted 33 lines
Quoted 1 line
Quoted 1 line · On 06.06.19 at 10:26, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num,

Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +0 -0
@@ -134,14 +134,28 @@ static int collect_cpu_info(unsigned int cpu_num, 

struct cpu_signature *csig)

Quoted 17 lines
Quoted 17 lines
xen/arch/x86/microcode_intel.c +9 -4
     return 0;
 }
 
-static inline int microcode_update_match(
-    unsigned int cpu_num, const struct microcode_header_intel *mc_header,
-    int sig, int pf)
+static enum microcode_match_result microcode_update_match(
+    const struct microcode_header_intel *mc_header, unsigned int sig,
+    unsigned int pf, unsigned int rev)
 {
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu_num);
+    const struct extended_sigtable *ext_header;
+    const struct extended_signature *ext_sig;
+    unsigned long data_size = get_datasize(mc_header);
+    unsigned int i;
+
+    if ( sigmatch(sig, mc_header->sig, pf, mc_header->pf) )
Quoted 8 lines

xen/arch/x86/microcode_intel.c static enum microcode_match_result microcode_update_match( · line 158

Quoted 1 line
xen/arch/x86/microcode_intel.c +1 -0
+        return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;

As indicated before, I think you would better also provide an "equal" indication. Iirc I've told you that I have one system where the cores get handed over from the BIOS in an inconsistent state (only core has ucode loaded). Hence we'd want to be able to also _store_ ucode matching that found on CPU 0, without actually want to _load_ it there.

Quoted 3 lines

Will do. What if no microcode update is provided in this case? Shall we refuse to boot? If we allow different microcode revisions in the system, it would complicate late microcode loading.

No, I don't think we should refuse to boot in such a case. We may want to warn about the situation, but should continue booting in a best effort manner. And no, I also don't think it'll complicate late loading meaningfully. Late loading, after all, is then the only way to get the system into "proper" shape again (other than rebooting after making available ucode to early boot).

Quoted 13 lines
Quoted 13 lines

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, · line 149

Quoted 4 lines
xen/arch/x86/microcode_intel.c +2 -2
-    return (sigmatch(sig, uci->cpu_sig.sig, pf, uci->cpu_sig.pf) &&
-            (mc_header->rev > uci->cpu_sig.rev));
+    if ( get_totalsize(mc_header) == (data_size + MC_HEADER_SIZE) )
+        return MIS_UCODE;

Okay, you're tightening the original <= to == here. But if you're already tightening things, why don't you make sure you actually have enough data to ...

xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, · line 150

Quoted 1 line
xen/arch/x86/microcode_intel.c +1 -0
+    ext_header = (const void *)(mc_header + 1) + data_size;

... hold an extended header, and then also to hold ...

Quoted 8 lines
Quoted 6 lines

xen/arch/x86/microcode_intel.c static enum microcode_match_result microcode_update_match( · line 158

Quoted 4 lines
xen/arch/x86/microcode_intel.c +4 -0
+    ext_sig = (const void *)(ext_header + 1);
+    for ( i = 0; i < ext_header->count; i++ )
+        if ( sigmatch(sig, ext_sig[i].sig, pf, ext_sig[i].pf) )
+            return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;

... enough array elements?

Do you think below incremental change is fine?

Something along these lines, yes. I don't think ...

Quoted 21 lines
xen/arch/x86/microcode_intel.c +3 -6
@@ -138,18 +138,25 @@ static enum microcode_match_result microcode_update_match(
     const struct extended_signature *ext_sig;
     unsigned long data_size = get_datasize(mc_header);
     unsigned int i;
+    const void *end = (const void *)mc_header + get_totalsize(mc_header);
 
     if ( sigmatch(sig, mc_header->sig, pf, mc_header->pf) )
         return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;
 
-    if ( get_totalsize(mc_header) == (data_size + MC_HEADER_SIZE) )
-        return MIS_UCODE;
-
     ext_header = (const void *)(mc_header + 1) + data_size;
     ext_sig = (const void *)(ext_header + 1);
-    for ( i = 0; i < ext_header->count; i++ )
-        if ( sigmatch(sig, ext_sig[i].sig, pf, ext_sig[i].pf) )
-            return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;
+
+    /*

xen/arch/x86/microcode_intel.c static enum microcode_match_result microcode_update_match( · line 159

Quoted 10 lines
xen/arch/x86/microcode_intel.c +10 -0
+     * Make sure there is enough space to hold an extended header and enough
+     * array elements.
+     */
+    if ( (end >= (const void *)ext_sig) &&
+         (end >= (const void *)(ext_sig + ext_header->count)) )
+    {
+        for ( i = 0; i < ext_header->count; i++ )
+            if ( sigmatch(sig, ext_sig[i].sig, pf, ext_sig[i].pf) )
+                return (mc_header->rev > rev) ? NEW_UCODE : OLD_UCODE;
+    }

... this re-indentation of the for() is needed. Just like the function was previously coded, the if() condition could be inverted and its body could be "return MIS_UCODE;". But it's a style question, and hence largely up to you.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#11 lore raw

PATCH v7 3/10 microcode: introduce a global cache of ucode patch

In reply to Chao Gao

to replace the current per-cpu cache 'uci->mc'.

With the assumption that all CPUs in the system have the same signature (family, model, stepping and 'pf'), one microcode update matches with one cpu should match with others. And Having multiple microcode revisions on different cpus would cause system unstable and is what should be avoided. Hence, caching only one microcode update is good enough for all cases.

Introduce a global variable, microcode_cache, to store the newest matching microcode update. Whenever we get a new valid microcode update, its revision id is compared against that of the microcode update to determine whether the "microcode_cache" needs to be replaced. And now this global cache is loaded to cpu in apply_microcode().

All operations on the cache is expected to be done with the 'microcode_mutex' hold.

Note that I deliberately avoid touching 'uci->mc' as I am going to remove it completely in the next patch.

---
Changes in v7:
- reworked to cache only one microcode patch rather than a list of microcode patches.

Changes in v6:
- constify local variables and function parameters if possible
- comment that the global cache is protected by 'microcode_mutex'. and add assertions to catch violations in microcode_{save/find}_patch()

Changes in v5:
- reword the commit description
- find_patch() and save_patch() are abstracted into common functions with some hooks for AMD and Intel

---
 xen/arch/x86/microcode.c        | 36 ++++++++++++++++
 xen/arch/x86/microcode_amd.c    | 91 +++++++++++++++++++++++++++++++++++++----
 xen/arch/x86/microcode_intel.c  | 77 +++++++++++++++++++++++++++-------
 xen/include/asm-x86/microcode.h | 15 +++++++
 4 files changed, 197 insertions(+), 22 deletions(-)
xen/arch/x86/microcode.c +36 -0
index 4163f50..cff86a9 100644
@@ -61,6 +61,9 @@ static struct ucode_mod_blob __initdata ucode_blob;
  */
 static bool_t __initdata ucode_scan;
 
+/* Protected by microcode_mutex */
+static struct microcode_patch *microcode_cache;
+
 void __init microcode_set_module(unsigned int idx)
 {
     ucode_mod_idx = idx;
@@ -262,6 +265,39 @@ int microcode_resume_cpu(unsigned int cpu)
     return err;
 }
 
+const struct microcode_patch *microcode_get_cache(void)
+{
+    ASSERT(spin_is_locked(&microcode_mutex));
+
+    return microcode_cache;
+}
+
+/* Return true if cache gets updated. Otherwise, return false */
+bool microcode_update_cache(struct microcode_patch *patch)
+{
+
+    ASSERT(spin_is_locked(&microcode_mutex));
+
+    if ( !microcode_ops->match_cpu(patch) )
+        return false;
+
+    if ( !microcode_cache )
+        microcode_cache = patch;
+    else if ( microcode_ops->compare_patch(patch, microcode_cache) ==
+                  NEW_UCODE )
+    {
+        microcode_ops->free_patch(microcode_cache);
+        microcode_cache = patch;
+    }
+    else
+    {
+        microcode_ops->free_patch(patch);
+        return false;
+    }
+
+    return true;
+}
+
 static int microcode_update_cpu(const void *buf, size_t size)
 {
     int err;
xen/arch/x86/microcode_amd.c +84 -7

121 lines not on this page · raw message

xen/arch/x86/microcode_intel.c +62 -15

151 lines not on this page · raw message

xen/include/asm-x86/microcode.h +15 -0
index 73ebe9a..6541c58 100644
@@ -12,6 +12,13 @@ enum microcode_match_result {
 struct cpu_signature;
 struct ucode_cpu_info;
 
+struct microcode_patch {
+    union {
+        struct microcode_intel *mc_intel;
+        struct microcode_amd *mc_amd;
+    };
+};
+
 struct microcode_ops {
     int (*microcode_resume_match)(unsigned int cpu, const void *mc);
     int (*cpu_request_microcode)(unsigned int cpu, const void *buf,
@@ -19,6 +26,11 @@ struct microcode_ops {
     int (*collect_cpu_info)(unsigned int cpu, struct cpu_signature *csig);
     int (*apply_microcode)(unsigned int cpu);
     int (*start_update)(void);
+    void (*free_patch)(struct microcode_patch *patch);
+    bool (*match_cpu)(const struct microcode_patch *patch);
+    enum microcode_match_result (*compare_patch)(
+            const struct microcode_patch *new,
+            const struct microcode_patch *old);
 };
 
 struct cpu_signature {
@@ -39,4 +51,7 @@ struct ucode_cpu_info {
 DECLARE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info);
 extern const struct microcode_ops *microcode_ops;
 
+const struct microcode_patch *microcode_get_cache(void);
+bool microcode_update_cache(struct microcode_patch *patch);
+
 #endif /* ASM_X86__MICROCODE_H */
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#12 lore raw

Re: [Xen-devel] [PATCH v7 03/10] microcode: introduce a global cache of ucode patch

Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode.c +6 -0
+bool microcode_update_cache(struct microcode_patch *patch)
+{
+
+    ASSERT(spin_is_locked(&microcode_mutex));
+
+    if ( !microcode_ops->match_cpu(patch) )

xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 291

Quoted 10 lines
xen/arch/x86/microcode.c +10 -0
+        return false;
+
+    if ( !microcode_cache )
+        microcode_cache = patch;
+    else if ( microcode_ops->compare_patch(patch, microcode_cache) ==
+                  NEW_UCODE )
+    {
+        microcode_ops->free_patch(microcode_cache);
+        microcode_cache = patch;
+    }

Hmm, okay, the way you do things here three enumeration values may indeed be sufficient. "old" may just be a little misleading then. (As to my respective comment on the previous patch.)

Quoted 6 lines
xen/arch/x86/microcode_amd.c +6 -0
+static struct microcode_patch *alloc_microcode_patch(
+    const struct microcode_amd *mc_amd)
+{
+    struct microcode_patch *microcode_patch = xmalloc(struct microcode_patch);
+    struct microcode_amd *cache = xmalloc(struct microcode_amd);
+    void *mpb = xmalloc_bytes(mc_amd->mpb_size);

xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 215

Quoted 10 lines
xen/arch/x86/microcode_amd.c +10 -0
+    struct equiv_cpu_entry *equiv_cpu_table =
+                                xmalloc_bytes(mc_amd->equiv_cpu_table_size);
+
+    if ( !microcode_patch || !cache || !mpb || !equiv_cpu_table )
+    {
+        xfree(microcode_patch);
+        xfree(cache);
+        xfree(mpb);
+        xfree(equiv_cpu_table);
+        printk(XENLOG_ERR "microcode: Can not allocate memory\n");

I'm not convinced this needs logging.

xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 221

Quoted 6 lines
xen/arch/x86/microcode_amd.c +6 -0
+        return ERR_PTR(-ENOMEM);
+    }
+
+    cache->equiv_cpu_table = equiv_cpu_table;
+    cache->mpb = mpb;
+    memcpy(cache->equiv_cpu_table, mc_amd->equiv_cpu_table,

Why not use the local variable here and ...

xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 223

Quoted 2 lines
xen/arch/x86/microcode_amd.c +2 -0
+           mc_amd->equiv_cpu_table_size);
+    memcpy(cache->mpb, mc_amd->mpb, mc_amd->mpb_size);

here? Less source code and presumably also slightly less binary code. In fact I wonder if you wouldn't better memcpy() first anyway, and only then store the values into the fields. It won't matter much with the global lock held, but it's generally good practice to do things in an order that won't risk to confuse hypothetical consumers of the data.

xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 238

Quoted 8 lines
xen/arch/x86/microcode_amd.c +8 -0
+static void free_patch(struct microcode_patch *microcode_patch)
+{
+    struct microcode_amd *mc_amd = microcode_patch->mc_amd;
+
+    xfree(mc_amd->equiv_cpu_table);
+    xfree(mc_amd->mpb);
+    xfree(mc_amd);
+    xfree(microcode_patch);

I think I said so before: Freeing of the generic wrapper struct would probably better be placed in generic code.

Quoted 7 lines
xen/arch/x86/microcode_amd.c +2 -1
@@ -497,7 +558,20 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,
     while ( (error = get_ucode_from_buffer_amd(mc_amd, buf, bufsize,
                                                &offset)) == 0 )
     {
-        if ( microcode_fits(mc_amd, cpu) )
+        struct microcode_patch *new_patch = alloc_microcode_patch(mc_amd);
+

xen/arch/x86/microcode_amd.c static int cpu_request_microcode(unsigned int cpu, const void *buf, · line 572

Quoted 10 lines
xen/arch/x86/microcode_amd.c +10 -0
+        if ( IS_ERR(new_patch) )
+        {
+            error = PTR_ERR(new_patch);
+            break;
+        }
+
+        if ( match_cpu(new_patch) )
+            microcode_update_cache(new_patch);
+        else
+            free_patch(new_patch);

Why do you re-do what microcode_update_cache() already does? It calls ->match_cpu() and ->free_patch() all by itself. It looks as if it would need to gain one more ->free_patch() invocation though.

(These last two comments apply to the respective Intel code as well then.)

xen/arch/x86/microcode_intel.c static int get_matching_microcode(const void *mc, unsigned int cpu) · line 325

Quoted 8 lines
xen/arch/x86/microcode_intel.c +1 -0
@@ -277,6 +319,7 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)
     memcpy(new_mc, mc, total_size);
     xfree(uci->mc.mc_intel);
     uci->mc.mc_intel = new_mc;
+
     return 1;
 }
 

Stray cosmetics?

Quoted 12 lines
xen/arch/x86/microcode_intel.c +2 -2
@@ -309,19 +356,19 @@ static int apply_microcode(unsigned int cpu)
     val[1] = (uint32_t)(msr_content >> 32);
 
     spin_unlock_irqrestore(&microcode_update_lock, flags);
-    if ( val[1] != uci->mc.mc_intel->hdr.rev )
+    if ( val[1] != mc_intel->hdr.rev )
     {
         printk(KERN_ERR "microcode: CPU%d update from revision "
                "%#x to %#x failed. Resulting revision is %#x.\n", cpu_num,
-               uci->cpu_sig.rev, uci->mc.mc_intel->hdr.rev, val[1]);
+               uci->cpu_sig.rev, mc_intel->hdr.rev, val[1]);
         return -EIO;

xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 371

Quoted 10 lines
xen/arch/x86/microcode_intel.c +3 -3
     }
     printk(KERN_INFO "microcode: CPU%d updated from revision "
            "%#x to %#x, date = %04x-%02x-%02x \n",
            cpu_num, uci->cpu_sig.rev, val[1],
-           uci->mc.mc_intel->hdr.year,
-           uci->mc.mc_intel->hdr.month,
-           uci->mc.mc_intel->hdr.day);
+           mc_intel->hdr.year,
+           mc_intel->hdr.month,
+           mc_intel->hdr.day);

The three arguments now look to all fit on a single line.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#13 lore raw

Re: [Xen-devel] [PATCH v7 03/10] microcode: introduce a global cache of ucode patch

Quoted 53 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode.c +6 -0
+bool microcode_update_cache(struct microcode_patch *patch)
+{
+
+    ASSERT(spin_is_locked(&microcode_mutex));
+
+    if ( !microcode_ops->match_cpu(patch) )

xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 291

Quoted 10 lines
xen/arch/x86/microcode.c +10 -0
+        return false;
+
+    if ( !microcode_cache )
+        microcode_cache = patch;
+    else if ( microcode_ops->compare_patch(patch, microcode_cache) ==
+                  NEW_UCODE )
+    {
+        microcode_ops->free_patch(microcode_cache);
+        microcode_cache = patch;
+    }

Hmm, okay, the way you do things here three enumeration values may indeed be sufficient. "old" may just be a little misleading then. (As to my respective comment on the previous patch.)

Quoted 6 lines
xen/arch/x86/microcode_amd.c +6 -0
+static struct microcode_patch *alloc_microcode_patch(
+    const struct microcode_amd *mc_amd)
+{
+    struct microcode_patch *microcode_patch = xmalloc(struct microcode_patch);
+    struct microcode_amd *cache = xmalloc(struct microcode_amd);
+    void *mpb = xmalloc_bytes(mc_amd->mpb_size);

xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 215

Quoted 10 lines
xen/arch/x86/microcode_amd.c +10 -0
+    struct equiv_cpu_entry *equiv_cpu_table =
+                                xmalloc_bytes(mc_amd->equiv_cpu_table_size);
+
+    if ( !microcode_patch || !cache || !mpb || !equiv_cpu_table )
+    {
+        xfree(microcode_patch);
+        xfree(cache);
+        xfree(mpb);
+        xfree(equiv_cpu_table);
+        printk(XENLOG_ERR "microcode: Can not allocate memory\n");

I'm not convinced this needs logging.

xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 221

Quoted 6 lines
xen/arch/x86/microcode_amd.c +6 -0
+        return ERR_PTR(-ENOMEM);
+    }
+
+    cache->equiv_cpu_table = equiv_cpu_table;
+    cache->mpb = mpb;
+    memcpy(cache->equiv_cpu_table, mc_amd->equiv_cpu_table,

Why not use the local variable here and ...

xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 223

Quoted 2 lines
xen/arch/x86/microcode_amd.c +2 -0
+           mc_amd->equiv_cpu_table_size);
+    memcpy(cache->mpb, mc_amd->mpb, mc_amd->mpb_size);
Quoted 6 lines

here? Less source code and presumably also slightly less binary code. In fact I wonder if you wouldn't better memcpy() first anyway, and only then store the values into the fields. It won't matter much with the global lock held, but it's generally good practice to do things in an order that won't risk to confuse hypothetical consumers of the data.

Will do.

Quoted 12 lines

xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 238

Quoted 9 lines
xen/arch/x86/microcode_amd.c +8 -0
+static void free_patch(struct microcode_patch *microcode_patch)
+{
+    struct microcode_amd *mc_amd = microcode_patch->mc_amd;
+
+    xfree(mc_amd->equiv_cpu_table);
+    xfree(mc_amd->mpb);
+    xfree(mc_amd);
+    xfree(microcode_patch);

I think I said so before: Freeing of the generic wrapper struct would probably better be placed in generic code.

Do you mean something as shown below:

/* in generic code */

struct microcode_patch {
    union {
        struct microcode_intel *mc_intel;
	struct microcode_amd *mc_amd;
	void *mc;
    };
};

void microcode_free_patch(struct microcode_patch *microcode_patch)

{
    microcode_ops->free_patch(microcode_patch->mc);
    xfree(microcode_patch);
}

/* in vendor-specific (AMD) code */

static void free_patch(void *mc)

{
    struct microcode_amd *mc_amd = mc;

    xfree(mc_amd->equiv_cpu_table);
    xfree(mc_amd->mpb);
    xfree(mc_amd);
}
Quoted 14 lines
Quoted 14 lines
xen/arch/x86/microcode_amd.c +8 -1
@@ -497,7 +558,20 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,
     while ( (error = get_ucode_from_buffer_amd(mc_amd, buf, bufsize,
                                                &offset)) == 0 )
     {
-        if ( microcode_fits(mc_amd, cpu) )
+        struct microcode_patch *new_patch = alloc_microcode_patch(mc_amd);
+
+        if ( IS_ERR(new_patch) )
+        {
+            error = PTR_ERR(new_patch);
+            break;
+        }
+
Quoted 9 lines

xen/arch/x86/microcode_amd.c static int cpu_request_microcode(unsigned int cpu, const void *buf, · line 572

Quoted 4 lines
xen/arch/x86/microcode_amd.c +4 -0
+        if ( match_cpu(new_patch) )
+            microcode_update_cache(new_patch);
+        else
+            free_patch(new_patch);

Why do you re-do what microcode_update_cache() already does? It calls ->match_cpu() and ->free_patch() all by itself. It looks as if it would need to gain one more ->free_patch() invocation though.

Will remove both invocations of match_cpu().

To support the case (the broken bios) you described, a patch which needs to be stored isn't necessary to be newer than the microcode loaded to current CPU. As long as the processor's signature is covered by the patch, we will store the patch regardless the revision number.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#14 lore raw

Re: [Xen-devel] [PATCH v7 03/10] microcode: introduce a global cache of ucode patch

Quoted 35 lines
Quoted 1 line
Quoted 1 line · On 10.06.19 at 07:33, <[email protected]> wrote:
Quoted 12 lines · Jan Beulich

xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 238

Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode_amd.c +8 -0
+static void free_patch(struct microcode_patch *microcode_patch)
+{
+    struct microcode_amd *mc_amd = microcode_patch->mc_amd;
+
+    xfree(mc_amd->equiv_cpu_table);
+    xfree(mc_amd->mpb);
+    xfree(mc_amd);
+    xfree(microcode_patch);

I think I said so before: Freeing of the generic wrapper struct would probably better be placed in generic code.

Do you mean something as shown below:

/* in generic code */

struct microcode_patch {
    union {
        struct microcode_intel *mc_intel;
	struct microcode_amd *mc_amd;
	void *mc;
    };
};

void microcode_free_patch(struct microcode_patch *microcode_patch)

{
    microcode_ops->free_patch(microcode_patch->mc);
    xfree(microcode_patch);
}

/* in vendor-specific (AMD) code */

Quoted 8 lines

static void free_patch(void *mc)

{
    struct microcode_amd *mc_amd = mc;

    xfree(mc_amd->equiv_cpu_table);
    xfree(mc_amd->mpb);
    xfree(mc_amd);
}

Something along these lines, yes. Whether you do as above or whether instead you continue to pass struct microcode_patch * is secondary (and really up to you). Perhaps the above the (slightly) better.

Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
xen/arch/x86/microcode_amd.c +8 -1
@@ -497,7 +558,20 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,
     while ( (error = get_ucode_from_buffer_amd(mc_amd, buf, bufsize,
                                                &offset)) == 0 )
     {
-        if ( microcode_fits(mc_amd, cpu) )
+        struct microcode_patch *new_patch = alloc_microcode_patch(mc_amd);
+
+        if ( IS_ERR(new_patch) )
+        {
+            error = PTR_ERR(new_patch);
+            break;
+        }
+
Quoted 9 lines

xen/arch/x86/microcode_amd.c static int cpu_request_microcode(unsigned int cpu, const void *buf, · line 572

Quoted 4 lines
xen/arch/x86/microcode_amd.c +4 -0
+        if ( match_cpu(new_patch) )
+            microcode_update_cache(new_patch);
+        else
+            free_patch(new_patch);

Why do you re-do what microcode_update_cache() already does? It calls ->match_cpu() and ->free_patch() all by itself. It looks as if it would need to gain one more ->free_patch() invocation though.

Quoted 6 lines

Will remove both invocations of match_cpu().

To support the case (the broken bios) you described, a patch which needs to be stored isn't necessary to be newer than the microcode loaded to current CPU. As long as the processor's signature is covered by the patch, we will store the patch regardless the revision number.

Well, if so, then fine. I did get the impression that successful application is a required pre-condition for storing.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#15 lore raw

PATCH v7 4/10 microcode: remove struct ucode_cpu_info

In reply to Chao Gao

We can remove the per-cpu cache field in struct ucode_cpu_info since it has been replaced by a global cache. It would leads to only one field remaining in ucode_cpu_info. Then, this struct is removed and the remaining field (cpu signature) is stored in per-cpu area.

Also remove 'microcode_resume_match' from microcode_ops because the check is done in find_patch(). The cpu status notifier is also removed. It was used to free the "mc" field to avoid memory leak.

---
Changes in v6:
- remove the whole struct ucode_cpu_info instead of the per-cpu cache in it.

---
 xen/arch/x86/apic.c             |   2 +-
 xen/arch/x86/microcode.c        |  91 +++---------------------------------
 xen/arch/x86/microcode_amd.c    | 100 +++++-----------------------------------
 xen/arch/x86/microcode_intel.c  |  33 ++++---------
 xen/arch/x86/spec_ctrl.c        |   2 +-
 xen/include/asm-x86/microcode.h |  13 +-----
 6 files changed, 30 insertions(+), 211 deletions(-)
xen/arch/x86/apic.c +1 -1
index fafc0bd..d216455 100644
@@ -1188,7 +1188,7 @@ static void __init check_deadline_errata(void)
     else
         rev = (unsigned long)m->driver_data;
 
-    if ( this_cpu(ucode_cpu_info).cpu_sig.rev >= rev )
+    if ( this_cpu(cpu_sig).rev >= rev )
         return;
 
     setup_clear_cpu_cap(X86_FEATURE_TSC_DEADLINE);
xen/arch/x86/microcode.c +7 -84

137 lines not on this page · raw message

xen/arch/x86/microcode_amd.c +12 -88

198 lines not on this page · raw message

xen/arch/x86/microcode_intel.c +8 -25
index d3405a0..bf6497f 100644
@@ -250,13 +250,13 @@ static int microcode_sanity_check(void *mc)
 
 static bool match_cpu(const struct microcode_patch *patch)
 {
-    const struct ucode_cpu_info *uci = &this_cpu(ucode_cpu_info);
+    const struct cpu_signature *sig = &this_cpu(cpu_sig);
 
     if ( !patch )
         return false;
 
-    return microcode_update_match(&patch->mc_intel->hdr, uci->cpu_sig.sig,
-                                uci->cpu_sig.pf, uci->cpu_sig.rev) == NEW_UCODE;
+    return microcode_update_match(&patch->mc_intel->hdr,
+                                  sig->sig, sig->pf, sig->rev) == NEW_UCODE;
 }
 
 static void free_patch(struct microcode_patch *patch)
@@ -281,7 +281,6 @@ static enum microcode_match_result compare_patch(
  */
 static int get_matching_microcode(const void *mc, unsigned int cpu)
 {
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu);
     const struct microcode_header_intel *mc_header = mc;
     unsigned long total_size = get_totalsize(mc_header);
     void *new_mc = xmalloc_bytes(total_size);
@@ -308,17 +307,7 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)
 
     pr_debug("microcode: CPU%d found a matching microcode update with"
              " version %#x (current=%#x)\n",
-             cpu, mc_header->rev, uci->cpu_sig.rev);
-    new_mc = xmalloc_bytes(total_size);
-    if ( new_mc == NULL )
-    {
-        printk(KERN_ERR "microcode: error! Can not allocate memory\n");
-        return -ENOMEM;
-    }
-
-    memcpy(new_mc, mc, total_size);
-    xfree(uci->mc.mc_intel);
-    uci->mc.mc_intel = new_mc;
+             cpu, mc_header->rev, this_cpu(cpu_sig).rev);
 
     return 1;
 }
@@ -329,7 +318,7 @@ static int apply_microcode(unsigned int cpu)
     uint64_t msr_content;
     unsigned int val[2];
     unsigned int cpu_num = raw_smp_processor_id();
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu_num);
+    struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
     const struct microcode_intel *mc_intel;
     const struct microcode_patch *patch = microcode_get_cache();
 
@@ -360,16 +349,16 @@ static int apply_microcode(unsigned int cpu)
     {
         printk(KERN_ERR "microcode: CPU%d update from revision "
                "%#x to %#x failed. Resulting revision is %#x.\n", cpu_num,
-               uci->cpu_sig.rev, mc_intel->hdr.rev, val[1]);
+               sig->rev, mc_intel->hdr.rev, val[1]);
         return -EIO;
     }
     printk(KERN_INFO "microcode: CPU%d updated from revision "
            "%#x to %#x, date = %04x-%02x-%02x \n",
-           cpu_num, uci->cpu_sig.rev, val[1],
+           cpu_num, sig->rev, val[1],
            mc_intel->hdr.year,
            mc_intel->hdr.month,
            mc_intel->hdr.day);
-    uci->cpu_sig.rev = val[1];
+    sig->rev = val[1];
 
     return 0;
 }
@@ -440,13 +429,7 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,
     return error;
 }
 
-static int microcode_resume_match(unsigned int cpu, const void *mc)
-{
-    return get_matching_microcode(mc, cpu);
-}
-
 static const struct microcode_ops microcode_intel_ops = {
-    .microcode_resume_match           = microcode_resume_match,
     .cpu_request_microcode            = cpu_request_microcode,
     .collect_cpu_info                 = collect_cpu_info,
     .apply_microcode                  = apply_microcode,
xen/arch/x86/spec_ctrl.c +1 -1
index 5d98cac..43128c3 100644
@@ -436,7 +436,7 @@ static bool __init check_smt_enabled(void)
 /* Calculate whether Retpoline is known-safe on this CPU. */
 static bool __init retpoline_safe(uint64_t caps)
 {
-    unsigned int ucode_rev = this_cpu(ucode_cpu_info).cpu_sig.rev;
+    unsigned int ucode_rev = this_cpu(cpu_sig).rev;
 
     if ( boot_cpu_data.x86_vendor == X86_VENDOR_AMD )
         return true;
xen/include/asm-x86/microcode.h +1 -12
index 6541c58..f2ac509 100644
@@ -10,7 +10,6 @@ enum microcode_match_result {
 };
 
 struct cpu_signature;
-struct ucode_cpu_info;
 
 struct microcode_patch {
     union {
@@ -20,7 +19,6 @@ struct microcode_patch {
 };
 
 struct microcode_ops {
-    int (*microcode_resume_match)(unsigned int cpu, const void *mc);
     int (*cpu_request_microcode)(unsigned int cpu, const void *buf,
                                  size_t size);
     int (*collect_cpu_info)(unsigned int cpu, struct cpu_signature *csig);
@@ -39,16 +37,7 @@ struct cpu_signature {
     unsigned int rev;
 };
 
-struct ucode_cpu_info {
-    struct cpu_signature cpu_sig;
-    union {
-        struct microcode_intel *mc_intel;
-        struct microcode_amd *mc_amd;
-        void *mc_valid;
-    } mc;
-};
-
-DECLARE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info);
+DECLARE_PER_CPU(struct cpu_signature, cpu_sig);
 extern const struct microcode_ops *microcode_ops;
 
 const struct microcode_patch *microcode_get_cache(void);
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#16 lore raw

Re: [Xen-devel] [PATCH v7 04/10] microcode: remove struct ucode_cpu_info

Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

We can remove the per-cpu cache field in struct ucode_cpu_info since it has been replaced by a global cache. It would leads to only one field remaining in ucode_cpu_info. Then, this struct is removed and the remaining field (cpu signature) is stored in per-cpu area.

Also remove 'microcode_resume_match' from microcode_ops because the check is done in find_patch(). The cpu status notifier is also removed. It was used to free the "mc" field to avoid memory leak.

There's no find_patch() function anymore afaics. And I also think this should be a separate patch. The above isn't enough imo to justify ...

Quoted 38 lines

int microcode_resume_cpu(unsigned int cpu)

 {
     int err;
xen/arch/x86/microcode.c +1 -28
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu);
-    struct cpu_signature nsig;
-    unsigned int cpu2;
+    struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
 
     if ( !microcode_ops )
         return 0;
 
     spin_lock(&microcode_mutex);
 
-    err = microcode_ops->collect_cpu_info(cpu, &uci->cpu_sig);
-    if ( err )
-    {
-        __microcode_fini_cpu(cpu);
-        spin_unlock(&microcode_mutex);
-        return err;
-    }
-
-    if ( uci->mc.mc_valid )
-    {
-        err = microcode_ops->microcode_resume_match(cpu, uci->mc.mc_valid);
-        if ( err >= 0 )
-        {
-            if ( err )
-                err = microcode_ops->apply_microcode(cpu);
-            spin_unlock(&microcode_mutex);
-            return err;
-        }
-    }
-
-    nsig = uci->cpu_sig;
-    __microcode_fini_cpu(cpu);
-    uci->cpu_sig = nsig;
-
-    err = -EIO;

xen/arch/x86/microcode.c struct microcode_info { · line 260

Quoted 10 lines
xen/arch/x86/microcode.c +0 -10
-    for_each_online_cpu ( cpu2 )
-    {
-        uci = &per_cpu(ucode_cpu_info, cpu2);
-        if ( uci->mc.mc_valid &&
-             microcode_ops->microcode_resume_match(cpu, uci->mc.mc_valid) > 0 )
-        {
-            err = microcode_ops->apply_microcode(cpu);
-            break;
-        }
-    }

... in particular the removal of this loop, the more that both the loop and the code ahead of it also call ->apply_microcode().

xen/arch/x86/microcode_intel.c static enum microcode_match_result compare_patch( · line 284

Quoted 5 lines
xen/arch/x86/microcode_intel.c +0 -1
@@ -281,7 +281,6 @@ static enum microcode_match_result compare_patch(
  */
 static int get_matching_microcode(const void *mc, unsigned int cpu)
 {
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu);

Note how this was using "cpu".

Quoted 6 lines
xen/arch/x86/microcode_intel.c +0 -2
@@ -308,17 +307,7 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)
 
     pr_debug("microcode: CPU%d found a matching microcode update with"
              " version %#x (current=%#x)\n",
-             cpu, mc_header->rev, uci->cpu_sig.rev);
-    new_mc = xmalloc_bytes(total_size);

xen/arch/x86/microcode_intel.c static int get_matching_microcode(const void *mc, unsigned int cpu) · line 310

Quoted 10 lines
xen/arch/x86/microcode_intel.c +1 -9
-    if ( new_mc == NULL )
-    {
-        printk(KERN_ERR "microcode: error! Can not allocate memory\n");
-        return -ENOMEM;
-    }
-
-    memcpy(new_mc, mc, total_size);
-    xfree(uci->mc.mc_intel);
-    uci->mc.mc_intel = new_mc;
+             cpu, mc_header->rev, this_cpu(cpu_sig).rev);

Why "this_cpu()" here?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#17 lore raw

Re: [Xen-devel] [PATCH v7 04/10] microcode: remove struct ucode_cpu_info

Quoted 56 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

We can remove the per-cpu cache field in struct ucode_cpu_info since it has been replaced by a global cache. It would leads to only one field remaining in ucode_cpu_info. Then, this struct is removed and the remaining field (cpu signature) is stored in per-cpu area.

Also remove 'microcode_resume_match' from microcode_ops because the check is done in find_patch(). The cpu status notifier is also removed. It was used to free the "mc" field to avoid memory leak.

There's no find_patch() function anymore afaics. And I also think this should be a separate patch. The above isn't enough imo to justify ...

Quoted 43 lines

int microcode_resume_cpu(unsigned int cpu)

 {
     int err;
xen/arch/x86/microcode.c +1 -33
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu);
-    struct cpu_signature nsig;
-    unsigned int cpu2;
+    struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
 
     if ( !microcode_ops )
         return 0;
 
     spin_lock(&microcode_mutex);
 
-    err = microcode_ops->collect_cpu_info(cpu, &uci->cpu_sig);
-    if ( err )
-    {
-        __microcode_fini_cpu(cpu);
-        spin_unlock(&microcode_mutex);
-        return err;
-    }
-
-    if ( uci->mc.mc_valid )
-    {
-        err = microcode_ops->microcode_resume_match(cpu, uci->mc.mc_valid);
-        if ( err >= 0 )
-        {
-            if ( err )
-                err = microcode_ops->apply_microcode(cpu);
-            spin_unlock(&microcode_mutex);
-            return err;
-        }
-    }
-
-    nsig = uci->cpu_sig;
-    __microcode_fini_cpu(cpu);
-    uci->cpu_sig = nsig;
-
-    err = -EIO;
-    for_each_online_cpu ( cpu2 )
-    {
-        uci = &per_cpu(ucode_cpu_info, cpu2);
-        if ( uci->mc.mc_valid &&
-             microcode_ops->microcode_resume_match(cpu, uci->mc.mc_valid) > 0 )
Quoted 8 lines

xen/arch/x86/microcode.c struct microcode_info { · line 260

Quoted 5 lines
xen/arch/x86/microcode.c +0 -5
-        {
-            err = microcode_ops->apply_microcode(cpu);
-            break;
-        }
-    }

... in particular the removal of this loop, the more that both the loop and the code ahead of it also call ->apply_microcode().

Ok. Will split it out from this patch and refine the patch description.

Basically, this function tries best to find a suitable patch from the per-cpu cache and loads it. Currently, the per-cpu cache is replaced by the global cache, and ->apply_microcode() loads the global cache rather then the per-cpu cache. Hence, a simple invocation of ->apply_microcode() is enough to apply the global cache during CPU hotplug or resuming from hibernation.

Quoted 11 lines

xen/arch/x86/microcode_intel.c static enum microcode_match_result compare_patch( · line 284

Quoted 6 lines
xen/arch/x86/microcode_intel.c +0 -1
@@ -281,7 +281,6 @@ static enum microcode_match_result compare_patch(
  */
 static int get_matching_microcode(const void *mc, unsigned int cpu)
 {
-    struct ucode_cpu_info *uci = &per_cpu(ucode_cpu_info, cpu);

Note how this was using "cpu".

Quoted 2 lines
(quoted diff) +0 -0
@@ -308,17 +307,7 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)
 
Quoted 16 lines
Quoted 4 lines
xen/arch/x86/microcode_intel.c +0 -2
     pr_debug("microcode: CPU%d found a matching microcode update with"
              " version %#x (current=%#x)\n",
-             cpu, mc_header->rev, uci->cpu_sig.rev);
-    new_mc = xmalloc_bytes(total_size);

xen/arch/x86/microcode_intel.c static int get_matching_microcode(const void *mc, unsigned int cpu) · line 310

Quoted 10 lines
xen/arch/x86/microcode_intel.c +1 -9
-    if ( new_mc == NULL )
-    {
-        printk(KERN_ERR "microcode: error! Can not allocate memory\n");
-        return -ENOMEM;
-    }
-
-    memcpy(new_mc, mc, total_size);
-    xfree(uci->mc.mc_intel);
-    uci->mc.mc_intel = new_mc;
+             cpu, mc_header->rev, this_cpu(cpu_sig).rev);

Why "this_cpu()" here?

It should be a part of next patch.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#18 lore raw

PATCH v7 5/10 microcode: remove pointless 'cpu' parameter

In reply to Chao Gao

Some callbacks in microcode_ops or related functions take a cpu id parameter. But at current call sites, the cpu id parameter is always equal to current cpu id. Some of them even use an assertion to guarantee this. Remove this redundent 'cpu' parameter.

---
 xen/arch/x86/acpi/power.c       |  2 +-
 xen/arch/x86/microcode.c        | 12 ++++++------
 xen/arch/x86/microcode_amd.c    | 35 +++++++++++++----------------------
 xen/arch/x86/microcode_intel.c  | 25 +++++++++----------------
 xen/arch/x86/smpboot.c          |  2 +-
 xen/include/asm-x86/microcode.h |  7 +++----
 xen/include/asm-x86/processor.h |  2 +-
 7 files changed, 34 insertions(+), 51 deletions(-)
xen/arch/x86/acpi/power.c +1 -1
index aecc754..4f21903 100644
@@ -253,7 +253,7 @@ static int enter_state(u32 state)
 
     console_end_sync();
 
-    microcode_resume_cpu(0);
+    microcode_resume_cpu();
 
     if ( !recheck_cpu_features(0) )
         panic("Missing previously available feature(s)\n");
xen/arch/x86/microcode.c +6 -6
index 0c01dfa..16a6d50 100644
@@ -196,19 +196,19 @@ struct microcode_info {
     char buffer[1];
 };
 
-int microcode_resume_cpu(unsigned int cpu)
+int microcode_resume_cpu(void)
 {
     int err;
-    struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
+    struct cpu_signature *sig = &this_cpu(cpu_sig);
 
     if ( !microcode_ops )
         return 0;
 
     spin_lock(&microcode_mutex);
 
-    err = microcode_ops->collect_cpu_info(cpu, sig);
+    err = microcode_ops->collect_cpu_info(sig);
     if ( likely(!err) )
-        err = microcode_ops->apply_microcode(cpu);
+        err = microcode_ops->apply_microcode();
     spin_unlock(&microcode_mutex);
 
     return err;
@@ -255,9 +255,9 @@ static int microcode_update_cpu(const void *buf, size_t size)
 
     spin_lock(&microcode_mutex);
 
-    err = microcode_ops->collect_cpu_info(cpu, sig);
+    err = microcode_ops->collect_cpu_info(sig);
     if ( likely(!err) )
-        err = microcode_ops->cpu_request_microcode(cpu, buf, size);
+        err = microcode_ops->cpu_request_microcode(buf, size);
     spin_unlock(&microcode_mutex);
 
     return err;
xen/arch/x86/microcode_amd.c +13 -22
index 93af2c9..0144df1 100644
@@ -78,8 +78,9 @@ struct mpbhdr {
 static DEFINE_SPINLOCK(microcode_update_lock);
 
 /* See comment in start_update() for cases when this routine fails */
-static int collect_cpu_info(unsigned int cpu, struct cpu_signature *csig)
+static int collect_cpu_info(struct cpu_signature *csig)
 {
+    unsigned int cpu = smp_processor_id();
     struct cpuinfo_x86 *c = &cpu_data[cpu];
 
     memset(csig, 0, sizeof(*csig));
@@ -152,18 +153,15 @@ static bool_t find_equiv_cpu_id(const struct equiv_cpu_entry *equiv_cpu_table,
     return 0;
 }
 
-static bool_t microcode_fits(const struct microcode_amd *mc_amd,
-                             unsigned int cpu)
+static bool microcode_fits(const struct microcode_amd *mc_amd)
 {
+    unsigned int cpu = smp_processor_id();
     const struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
     const struct microcode_header_amd *mc_header = mc_amd->mpb;
     const struct equiv_cpu_entry *equiv_cpu_table = mc_amd->equiv_cpu_table;
     unsigned int current_cpu_id;
     unsigned int equiv_cpu_id;
 
-    /* We should bind the task to the CPU */
-    BUG_ON(cpu != raw_smp_processor_id());
-
     current_cpu_id = cpuid_eax(0x00000001);
 
     if ( !find_equiv_cpu_id(equiv_cpu_table, current_cpu_id, &equiv_cpu_id) )
@@ -192,9 +190,7 @@ static bool_t microcode_fits(const struct microcode_amd *mc_amd,
 
 static bool match_cpu(const struct microcode_patch *patch)
 {
-    if ( !patch )
-        return false;
-    return microcode_fits(patch->mc_amd, smp_processor_id());
+    return patch ? microcode_fits(patch->mc_amd) : false;
 }
 
 static struct microcode_patch *alloc_microcode_patch(
@@ -253,18 +249,16 @@ static enum microcode_match_result compare_patch(
     return MIS_UCODE;
 }
 
-static int apply_microcode(unsigned int cpu)
+static int apply_microcode(void)
 {
     unsigned long flags;
     uint32_t rev;
     int hw_err;
+    unsigned int cpu = smp_processor_id();
     struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
     const struct microcode_header_amd *hdr;
     const struct microcode_patch *patch = microcode_get_cache();
 
-    /* We should bind the task to the CPU */
-    BUG_ON(raw_smp_processor_id() != cpu);
-
     if ( !match_cpu(patch) )
         return -EINVAL;
 
@@ -435,14 +429,14 @@ static const unsigned int final_levels[] = {
     0x010000af
 };
 
-static bool_t check_final_patch_levels(unsigned int cpu)
+static bool check_final_patch_levels(void)
 {
     /*
      * Check the current patch levels on the cpu. If they are equal to
      * any of the 'final_levels', then we should not update the microcode
      * patch on the cpu as system will hang otherwise.
      */
-    const struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
+    const struct cpu_signature *sig = &this_cpu(cpu_sig);
     unsigned int i;
 
     if ( boot_cpu_data.x86 != 0x10 )
@@ -455,19 +449,16 @@ static bool_t check_final_patch_levels(unsigned int cpu)
     return 0;
 }
 
-static int cpu_request_microcode(unsigned int cpu, const void *buf,
-                                 size_t bufsize)
+static int cpu_request_microcode(const void *buf, size_t bufsize)
 {
     struct microcode_amd *mc_amd;
     size_t offset = 0;
     int error = 0;
     unsigned int current_cpu_id;
     unsigned int equiv_cpu_id;
+    unsigned int cpu = smp_processor_id();
     const struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
 
-    /* We should bind the task to the CPU */
-    BUG_ON(cpu != raw_smp_processor_id());
-
     current_cpu_id = cpuid_eax(0x00000001);
 
     if ( *(const uint32_t *)buf != UCODE_MAGIC )
@@ -477,7 +468,7 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,
         goto out;
     }
 
-    if ( check_final_patch_levels(cpu) )
+    if ( check_final_patch_levels() )
     {
         printk(XENLOG_INFO
                "microcode: Cannot update microcode patch on the cpu as we hit a final level\n");
@@ -567,7 +558,7 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,
 
         if ( match_cpu(microcode_get_cache()) )
         {
-            error = apply_microcode(cpu);
+            error = apply_microcode();
             if ( error )
                 break;
         }
xen/arch/x86/microcode_intel.c +9 -16
index bf6497f..b66844d 100644
@@ -96,13 +96,12 @@ struct extended_sigtable {
 /* serialize access to the physical write to MSR 0x79 */
 static DEFINE_SPINLOCK(microcode_update_lock);
 
-static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig)
+static int collect_cpu_info(struct cpu_signature *csig)
 {
+    unsigned int cpu_num = smp_processor_id();
     struct cpuinfo_x86 *c = &cpu_data[cpu_num];
     uint64_t msr_content;
 
-    BUG_ON(cpu_num != smp_processor_id());
-
     memset(csig, 0, sizeof(*csig));
 
     if ( (c->x86_vendor != X86_VENDOR_INTEL) || (c->x86 < 6) )
@@ -279,12 +278,13 @@ static enum microcode_match_result compare_patch(
  * return 1 - found update
  * return < 0 - error
  */
-static int get_matching_microcode(const void *mc, unsigned int cpu)
+static int get_matching_microcode(const void *mc)
 {
     const struct microcode_header_intel *mc_header = mc;
     unsigned long total_size = get_totalsize(mc_header);
     void *new_mc = xmalloc_bytes(total_size);
     struct microcode_patch *new_patch = xmalloc(struct microcode_patch);
+    unsigned int __maybe_unused cpu = smp_processor_id();
 
     if ( !new_patch || !new_mc )
     {
@@ -312,19 +312,16 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)
     return 1;
 }
 
-static int apply_microcode(unsigned int cpu)
+static int apply_microcode(void)
 {
     unsigned long flags;
     uint64_t msr_content;
     unsigned int val[2];
     unsigned int cpu_num = raw_smp_processor_id();
-    struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
+    struct cpu_signature *sig = &this_cpu(cpu_sig);
     const struct microcode_intel *mc_intel;
     const struct microcode_patch *patch = microcode_get_cache();
 
-    /* We should bind the task to the CPU */
-    BUG_ON(cpu_num != cpu);
-
     if ( !match_cpu(patch) )
         return -EINVAL;
 
@@ -391,22 +388,18 @@ static long get_next_ucode_from_buffer(void **mc, const u8 *buf,
     return offset + total_size;
 }
 
-static int cpu_request_microcode(unsigned int cpu, const void *buf,
-                                 size_t size)
+static int cpu_request_microcode(const void *buf, size_t size)
 {
     long offset = 0;
     int error = 0;
     void *mc;
 
-    /* We should bind the task to the CPU */
-    BUG_ON(cpu != raw_smp_processor_id());
-
     while ( (offset = get_next_ucode_from_buffer(&mc, buf, size, offset)) > 0 )
     {
         error = microcode_sanity_check(mc);
         if ( error )
             break;
-        error = get_matching_microcode(mc, cpu);
+        error = get_matching_microcode(mc);
         if ( error < 0 )
             break;
         /*
@@ -424,7 +417,7 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,
         error = offset;
 
     if ( !error && match_cpu(microcode_get_cache()) )
-        error = apply_microcode(cpu);
+        error = apply_microcode();
 
     return error;
 }
xen/arch/x86/smpboot.c +1 -1
index 274865a..de19b67 100644
@@ -366,7 +366,7 @@ void start_secondary(void *unused)
     if ( system_state <= SYS_STATE_smp_boot )
         early_microcode_update_cpu(false);
     else
-        microcode_resume_cpu(cpu);
+        microcode_resume_cpu();
 
     /*
      * If MSR_SPEC_CTRL is available, apply Xen's default setting and discard
xen/include/asm-x86/microcode.h +3 -4
index f2ac509..e6842d4 100644
@@ -19,10 +19,9 @@ struct microcode_patch {
 };
 
 struct microcode_ops {
-    int (*cpu_request_microcode)(unsigned int cpu, const void *buf,
-                                 size_t size);
-    int (*collect_cpu_info)(unsigned int cpu, struct cpu_signature *csig);
-    int (*apply_microcode)(unsigned int cpu);
+    int (*cpu_request_microcode)(const void *buf, size_t size);
+    int (*collect_cpu_info)(struct cpu_signature *csig);
+    int (*apply_microcode)(void);
     int (*start_update)(void);
     void (*free_patch)(struct microcode_patch *patch);
     bool (*match_cpu)(const struct microcode_patch *patch);
xen/include/asm-x86/processor.h +1 -1
index cef3ffb..8b7e484 100644
@@ -573,7 +573,7 @@ int guest_wrmsr_xen(struct vcpu *v, uint32_t idx, uint64_t val);
 
 void microcode_set_module(unsigned int);
 int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void), unsigned long len);
-int microcode_resume_cpu(unsigned int cpu);
+int microcode_resume_cpu(void);
 int early_microcode_update_cpu(bool start_update);
 int early_microcode_init(void);
 int microcode_init_intel(void);
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#19 lore raw

Re: [Xen-devel] [PATCH v7 05/10] microcode: remove pointless 'cpu' parameter

xen/arch/x86/microcode_amd.c struct mpbhdr { · line 84

Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode_amd.c +2 -1
@@ -78,8 +78,9 @@ struct mpbhdr {
 static DEFINE_SPINLOCK(microcode_update_lock);
 
 /* See comment in start_update() for cases when this routine fails */
-static int collect_cpu_info(unsigned int cpu, struct cpu_signature *csig)
+static int collect_cpu_info(struct cpu_signature *csig)
 {
+    unsigned int cpu = smp_processor_id();
     struct cpuinfo_x86 *c = &cpu_data[cpu];

I think it would be more clear if you used current_cpu_data here. The only other use of "cpu" is in a pr_debug(), which by default expands to nothing anyway, and hence is cheap to change to use smp_processor_id() instead.

Quoted 5 lines
xen/arch/x86/microcode_amd.c +0 -1
@@ -435,14 +429,14 @@ static const unsigned int final_levels[] = {
     0x010000af
 };
 
-static bool_t check_final_patch_levels(unsigned int cpu)

xen/arch/x86/microcode_amd.c static const unsigned int final_levels[] = { · line 440

Quoted 10 lines
xen/arch/x86/microcode_amd.c +2 -1
+static bool check_final_patch_levels(void)
 {
     /*
      * Check the current patch levels on the cpu. If they are equal to
      * any of the 'final_levels', then we should not update the microcode
      * patch on the cpu as system will hang otherwise.
      */
-    const struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
+    const struct cpu_signature *sig = &this_cpu(cpu_sig);
     unsigned int i;

I don't see any dependency of this function upon running on the subject CPU.

xen/arch/x86/microcode_amd.c static enum microcode_match_result compare_patch( · line 287

Quoted 12 lines
xen/arch/x86/microcode_amd.c +2 -1
@@ -279,12 +278,13 @@ static enum microcode_match_result compare_patch(
  * return 1 - found update
  * return < 0 - error
  */
-static int get_matching_microcode(const void *mc, unsigned int cpu)
+static int get_matching_microcode(const void *mc)
 {
     const struct microcode_header_intel *mc_header = mc;
     unsigned long total_size = get_totalsize(mc_header);
     void *new_mc = xmalloc_bytes(total_size);
     struct microcode_patch *new_patch = xmalloc(struct microcode_patch);
+    unsigned int __maybe_unused cpu = smp_processor_id();

The __maybe_unused is for the sole use in pr_debug()? Please instead use smp_processor_id() there, if so.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#20 lore raw

Re: [Xen-devel] [PATCH v7 05/10] microcode: remove pointless 'cpu' parameter

Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode_amd.c +0 -0
@@ -78,8 +78,9 @@ struct mpbhdr {
 static DEFINE_SPINLOCK(microcode_update_lock);
 
Quoted 11 lines

xen/arch/x86/microcode_amd.c struct mpbhdr { · line 84

Quoted 6 lines
xen/arch/x86/microcode_amd.c +2 -1
 /* See comment in start_update() for cases when this routine fails */
-static int collect_cpu_info(unsigned int cpu, struct cpu_signature *csig)
+static int collect_cpu_info(struct cpu_signature *csig)
 {
+    unsigned int cpu = smp_processor_id();
     struct cpuinfo_x86 *c = &cpu_data[cpu];

I think it would be more clear if you used current_cpu_data here. The only other use of "cpu" is in a pr_debug(), which by default expands to nothing anyway, and hence is cheap to change to use smp_processor_id() instead.

Will do.

Quoted 5 lines
Quoted 5 lines
xen/arch/x86/microcode_amd.c +0 -0
@@ -435,14 +429,14 @@ static const unsigned int final_levels[] = {
     0x010000af
 };
 
Quoted 14 lines

xen/arch/x86/microcode_amd.c static const unsigned int final_levels[] = { · line 440

Quoted 11 lines
xen/arch/x86/microcode_amd.c +2 -2
-static bool_t check_final_patch_levels(unsigned int cpu)
+static bool check_final_patch_levels(void)
 {
     /*
      * Check the current patch levels on the cpu. If they are equal to
      * any of the 'final_levels', then we should not update the microcode
      * patch on the cpu as system will hang otherwise.
      */
-    const struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
+    const struct cpu_signature *sig = &this_cpu(cpu_sig);
     unsigned int i;

I don't see any dependency of this function upon running on the subject CPU.

Ok. I will drop this change.

Quoted 16 lines

xen/arch/x86/microcode_amd.c static enum microcode_match_result compare_patch( · line 287

Quoted 13 lines
xen/arch/x86/microcode_amd.c +2 -1
@@ -279,12 +278,13 @@ static enum microcode_match_result compare_patch(
  * return 1 - found update
  * return < 0 - error
  */
-static int get_matching_microcode(const void *mc, unsigned int cpu)
+static int get_matching_microcode(const void *mc)
 {
     const struct microcode_header_intel *mc_header = mc;
     unsigned long total_size = get_totalsize(mc_header);
     void *new_mc = xmalloc_bytes(total_size);
     struct microcode_patch *new_patch = xmalloc(struct microcode_patch);
+    unsigned int __maybe_unused cpu = smp_processor_id();

The __maybe_unused is for the sole use in pr_debug()? Please instead use smp_processor_id() there, if so.

Will do.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#21 lore raw

PATCH v7 6/10 microcode: split out apply_microcode() from cpu_request_microcode()

In reply to Chao Gao

During late microcode update, apply_microcode() is invoked in cpu_request_microcode(). To make late microcode update more reliable, we want to put the apply_microcode() into stop_machine context. So we split out it from cpu_request_microcode(). As a consequence, apply_microcode() should be invoked explicitly in the common code.

Previously, apply_microcode() gets the microcode patch to be applied from the microcode cache. Now, the patch is passed as a function argument and a patch is cached for cpu-hotplug and cpu resuming, only after it has been loaded to a cpu without any error. As a consequence, the 'match_cpu' check in microcode_update_cache is removed, which otherwise would fail.

Assuming that all CPUs have the same signature, one patch matching with current CPU should match with others. Then parsing microcode only needs to be done once; cpu_request_microcode() is also moved out of microcode_update_cpu().

On AMD side, svm_host_osvw_init() is supposed to be called after microcode update. As apply_micrcode() won't be called by cpu_request_microcode() now, svm_host_osvw_init() is moved to the end of apply_microcode().

---
Changes in v7:
- to handle load failure, unvalidated patches won't be cached. They are passed as function arguments. So if update failed, we needn't any cleanup to microcode cache.
- microcode_info which passes microcode blob to be parsed to each CPU is replaced by microcode_patch.

Changes in v6:
- during early microcode update, BSP and APs call different functions. Thus AP can bypass parsing microcode blob.

---
 xen/arch/x86/acpi/power.c       |   2 +-
 xen/arch/x86/microcode.c        | 209 ++++++++++++++++++++++++++--------------
 xen/arch/x86/microcode_amd.c    |  41 ++++----
 xen/arch/x86/microcode_intel.c  |  69 ++++++-------
 xen/arch/x86/smpboot.c          |   5 +-
 xen/include/asm-x86/microcode.h |   8 +-
 xen/include/asm-x86/processor.h |   3 +-
 7 files changed, 193 insertions(+), 144 deletions(-)
xen/arch/x86/acpi/power.c +1 -1
index 4f21903..9583172 100644
@@ -253,7 +253,7 @@ static int enter_state(u32 state)
 
     console_end_sync();
 
-    microcode_resume_cpu();
+    early_microcode_update_cpu();
 
     if ( !recheck_cpu_features(0) )
         panic("Missing previously available feature(s)\n");
xen/arch/x86/microcode.c +137 -72

288 lines not on this page · raw message

xen/arch/x86/microcode_amd.c +19 -22

82 lines not on this page · raw message

xen/arch/x86/microcode_intel.c +31 -38

120 lines not on this page · raw message

xen/arch/x86/smpboot.c +1 -4
index de19b67..cd5f9cc 100644
@@ -363,10 +363,7 @@ void start_secondary(void *unused)
 
     initialize_cpu_data(cpu);
 
-    if ( system_state <= SYS_STATE_smp_boot )
-        early_microcode_update_cpu(false);
-    else
-        microcode_resume_cpu();
+    early_microcode_update_cpu();
 
     /*
      * If MSR_SPEC_CTRL is available, apply Xen's default setting and discard
xen/include/asm-x86/microcode.h +3 -5
index e6842d4..3fa3acd 100644
@@ -19,9 +19,10 @@ struct microcode_patch {
 };
 
 struct microcode_ops {
-    int (*cpu_request_microcode)(const void *buf, size_t size);
+    struct microcode_patch *(*cpu_request_microcode)(const void *buf,
+                                                     size_t size);
     int (*collect_cpu_info)(struct cpu_signature *csig);
-    int (*apply_microcode)(void);
+    int (*apply_microcode)(const struct microcode_patch *patch);
     int (*start_update)(void);
     void (*free_patch)(struct microcode_patch *patch);
     bool (*match_cpu)(const struct microcode_patch *patch);
@@ -39,7 +40,4 @@ struct cpu_signature {
 DECLARE_PER_CPU(struct cpu_signature, cpu_sig);
 extern const struct microcode_ops *microcode_ops;
 
-const struct microcode_patch *microcode_get_cache(void);
-bool microcode_update_cache(struct microcode_patch *patch);
-
 #endif /* ASM_X86__MICROCODE_H */
xen/include/asm-x86/processor.h +1 -2
index 8b7e484..d8668e6 100644
@@ -573,8 +573,7 @@ int guest_wrmsr_xen(struct vcpu *v, uint32_t idx, uint64_t val);
 
 void microcode_set_module(unsigned int);
 int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void), unsigned long len);
-int microcode_resume_cpu(void);
-int early_microcode_update_cpu(bool start_update);
+int early_microcode_update_cpu(void);
 int early_microcode_init(void);
 int microcode_init_intel(void);
 int microcode_init_amd(void);
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#22 lore raw

Re: [Xen-devel] [PATCH v7 06/10] microcode: split out apply_microcode() from cpu_request_microcode()

Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

During late microcode update, apply_microcode() is invoked in cpu_request_microcode(). To make late microcode update more reliable, we want to put the apply_microcode() into stop_machine context. So we split out it from cpu_request_microcode(). As a consequence, apply_microcode() should be invoked explicitly in the common code.

Quoted 6 lines

Previously, apply_microcode() gets the microcode patch to be applied from the microcode cache. Now, the patch is passed as a function argument and a patch is cached for cpu-hotplug and cpu resuming, only after it has been loaded to a cpu without any error. As a consequence, the 'match_cpu' check in microcode_update_cache is removed, which otherwise would fail.

The "only after it has been loaded to a cpu without any error" is a problem, precisely for the case where ucode on the different cores is not in sync initially. I would actually like to put up this question:
When a core has no ucode loaded at all yet and only strictly older (than loaded on some other cores) ucode is found to be available, whether then it wouldn't still be better to apply that ucode to _at least_ the cores that have none loaded yet.

To get the system into "sane" state it may even be necessary to downgrade ucode on the cores which did have it loaded already, in such a situation.

Quoted 4 lines

On AMD side, svm_host_osvw_init() is supposed to be called after microcode update. As apply_micrcode() won't be called by cpu_request_microcode() now, svm_host_osvw_init() is moved to the end of apply_microcode().

I guess this really ought to become a vendor hook as well, but I wouldn't insist on you doing so here.

xen/arch/x86/acpi/power.c static int enter_state(u32 state) · line 256

Quoted 8 lines
xen/arch/x86/acpi/power.c +1 -1
@@ -253,7 +253,7 @@ static int enter_state(u32 state)
 
     console_end_sync();
 
-    microcode_resume_cpu();
+    early_microcode_update_cpu();

The use here, the (changed) use in start_secondary(), and the dropping of its __init suggest to make an attempt to find a better name for the function. Maybe microcode_update_one()?

xen/arch/x86/acpi/power.c static int enter_state(u32 state)

Quoted 7 lines
xen/arch/x86/acpi/power.c +7 -0
+/*
+ * Load a microcode update to current CPU.
+ *
+ * If no patch is provided, the cached patch will be loaded. Microcode update
+ * during APs bringup and CPU resuming falls into this case.
+ */
+static int microcode_update_cpu(struct microcode_patch *patch)

const?

Quoted 22 lines

{

xen/arch/x86/acpi/power.c +9 -8
-    int err;
-    struct cpu_signature *sig = &this_cpu(cpu_sig);
+    int ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
 
-    if ( !microcode_ops )
-        return 0;
+    if ( unlikely(ret) )
+        return ret;
 
     spin_lock(&microcode_mutex);
 
-    err = microcode_ops->collect_cpu_info(sig);
-    if ( likely(!err) )
-        err = microcode_ops->apply_microcode();
-    spin_unlock(&microcode_mutex);
+    if ( patch )
+    {
+        /*
+         * If a patch is specified, it should has newer revision than
+         * that of the patch cached.
+         */

xen/arch/x86/acpi/power.c

Quoted 10 lines
xen/arch/x86/acpi/power.c +7 -2
+        if ( microcode_cache &&
+             microcode_ops->compare_patch(patch, microcode_cache) != NEW_UCODE )
+        {
+            spin_unlock(&microcode_mutex);
+            return -EINVAL;
+        }
 
-    return err;
-}
+        ret = microcode_ops->apply_microcode(patch);

There's no printk() here but ...

xen/arch/x86/acpi/power.c

Quoted 6 lines
xen/arch/x86/acpi/power.c +6 -0
+    }
+    else if ( microcode_cache )
+    {
+        ret = microcode_ops->apply_microcode(microcode_cache);
+        if ( ret == -EIO )
+            printk("Update failed. Reboot needed\n");

... you emit a log message here. Why the difference? And wouldn't it be better to have just a single call to ->apply anyway, by simply assigning "microcode_cache" to "patch" and moving the call a little further down?

xen/arch/x86/acpi/power.c

Quoted 4 lines
xen/arch/x86/acpi/power.c +4 -0
+    }
+    else
+        /* No patch to update */
+        ret = -EINVAL;

-ENOENT?

xen/arch/x86/acpi/power.c bool microcode_update_cache(struct microcode_patch *patch) · line 275

Quoted 13 lines
xen/arch/x86/acpi/power.c +2 -6
@@ -247,46 +270,32 @@ bool microcode_update_cache(struct microcode_patch *patch)
     return true;
 }
 
-static int microcode_update_cpu(const void *buf, size_t size)
+static long do_microcode_update(void *patch)
 {
-    int err;
-    unsigned int cpu = smp_processor_id();
-    struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
-
-    spin_lock(&microcode_mutex);
+    int error, cpu;

While "int" is fine for error codes, it almost certainly wants to be "unsigned int" for "cpu". The more that it had been that was before. I also don't see why you need to switch from "err" to "error" - oh,
...

xen/arch/x86/acpi/power.c bool microcode_update_cache(struct microcode_patch *patch) · line 276

Quoted 13 lines
xen/arch/x86/acpi/power.c +1 -12
-    err = microcode_ops->collect_cpu_info(sig);
-    if ( likely(!err) )
-        err = microcode_ops->cpu_request_microcode(buf, size);
-    spin_unlock(&microcode_mutex);
-
-    return err;
-}
-
-static long do_microcode_update(void *_info)
-{
-    struct microcode_info *info = _info;
-    int error;
+    error = microcode_update_cpu(patch);

... there was a pre-existing variable of that name here.

xen/arch/x86/acpi/power.c bool microcode_update_cache(struct microcode_patch *patch) · line 279

Quoted 3 lines
xen/arch/x86/acpi/power.c +3 -0
+    if ( error )
+    {
+        microcode_ops->free_patch(microcode_cache);

Does this also set "microcode_cache" to NULL? I didn't think so. It's anyway not really clear why _all_ forms of errors should lead to clearing of the cache. However - looking at the code further down, don't you rather mean to free "patch" here anyway?

Quoted 6 lines
xen/arch/x86/acpi/power.c +2 -2
+        return error;
+    }
 
-    BUG_ON(info->cpu != smp_processor_id());
 
-    error = microcode_update_cpu(info->buffer, info->buffer_size);

xen/arch/x86/acpi/power.c bool microcode_update_cache(struct microcode_patch *patch) · line 288

Quoted 10 lines
xen/arch/x86/acpi/power.c +4 -5
-    if ( error )
-        info->error = error;
+    cpu = cpumask_next(smp_processor_id(), &cpu_online_map);
+    if ( cpu < nr_cpu_ids )
+        return continue_hypercall_on_cpu(cpu, do_microcode_update, patch);
 
-    info->cpu = cpumask_next(info->cpu, &cpu_online_map);
-    if ( info->cpu < nr_cpu_ids )
-        return continue_hypercall_on_cpu(info->cpu, do_microcode_update, info);
+    microcode_update_cache(patch);

Independent of my remarks at the top I would think that updating of the cache should happen after the first successful loading on a CPU, not after all CPUs have been updated successfully. There would then also not be any need to pass "patch" on to continue_hypercall_on_cpu() a few lines up from here (albeit from a general logic perspective it may indeed be easier to keep it that way).

Quoted 35 lines
xen/arch/x86/acpi/power.c +9 -14
@@ -294,32 +303,49 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)
     if ( microcode_ops == NULL )
         return -EINVAL;
 
-    info = xmalloc_bytes(sizeof(*info) + len);
-    if ( info == NULL )
-        return -ENOMEM;
-
-    ret = copy_from_guest(info->buffer, buf, len);
-    if ( ret != 0 )
+    buffer = xmalloc_bytes(len);
+    if ( !buffer )
     {
-        xfree(info);
-        return ret;
+        ret = -ENOMEM;
+        goto free;
     }
 
-    info->buffer_size = len;
-    info->error = 0;
-    info->cpu = cpumask_first(&cpu_online_map);
+    if ( copy_from_guest(buffer, buf, len) )
+    {
+        ret = -EFAULT;
+        goto free;
+    }
 
     if ( microcode_ops->start_update )
     {
         ret = microcode_ops->start_update();
         if ( ret != 0 )
-        {
-            xfree(info);
-            return ret;

xen/arch/x86/acpi/power.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 330

Quoted 10 lines
xen/arch/x86/acpi/power.c +6 -2
-        }
+            goto free;
     }
 
-    return continue_hypercall_on_cpu(info->cpu, do_microcode_update, info);
+    patch = microcode_parse_blob(buffer, len);
+    if ( IS_ERR(patch) )
+    {
+        printk(XENLOG_ERR "Parsing microcode blob error %ld\n", PTR_ERR(patch));
+        ret = PTR_ERR(patch);

So I assume we would get here when the system already has the newest (or even newer) ucode loaded. That's not an error, and imo no log message suggesting so should be issued. Perhaps the parsing code could return NULL to indicate so? Although, judging by the code further down you already expect NULL potentially coming back, but I can't seem to be able to figure the condition(s).

Also please switch the two lines around and use "ret" in the printk() invocation.

xen/arch/x86/acpi/power.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 336

Quoted 6 lines
xen/arch/x86/acpi/power.c +6 -0
+        goto free;
+    }
+
+    if ( !microcode_ops->match_cpu(patch) )
+    {
+        printk(XENLOG_ERR "No matching or newer ucode found. Update aborted!\n");

I assume the "matching" here is meant to cover the CPU signature? The wording is ambiguous this way, because it could also mean there was no ucode found matching that which is already loaded (which, as per above, may end up being relevant).

Furthermore - why this check, when microcode_parse_blob() already looks for something that's newer than what is currently loaded (and matches the CPU signature)?

Quoted 19 lines
xen/arch/x86/acpi/power.c +8 -2
@@ -362,13 +397,41 @@ int __init early_microcode_update_cpu(bool start_update)
     }
     if ( data )
     {
-        if ( start_update && microcode_ops->start_update )
+        struct microcode_patch *patch;
+
+        if ( microcode_ops->start_update )
             rc = microcode_ops->start_update();
 
         if ( rc )
             return rc;
 
-        return microcode_update_cpu(data, len);
+        patch = microcode_parse_blob(data, len);
+        if ( IS_ERR(patch) )
+        {
+            printk(XENLOG_ERR "Parsing microcode blob error %ld\n",
+                   PTR_ERR(patch));

xen/arch/x86/acpi/power.c int __init early_microcode_update_cpu(bool start_update) · line 422

Quoted 10 lines
xen/arch/x86/acpi/power.c +10 -0
+            return PTR_ERR(patch);
+        }
+
+        if ( !microcode_ops->match_cpu(patch) )
+        {
+            printk(XENLOG_ERR "No matching or newer ucode found. Update aborted!\n");
+            if ( patch )
+                microcode_ops->free_patch(patch);
+            return -EINVAL;
+        }

Same remarks here then.

xen/arch/x86/acpi/power.c static int apply_microcode(void) · line 299

Quoted 10 lines
xen/arch/x86/acpi/power.c +4 -0
@@ -292,6 +291,10 @@ static int apply_microcode(void)
 
     sig->rev = rev;
 
+#ifdef CONFIG_HVM
+    svm_host_osvw_init();
+#endif
+
     return 0;
 }

While this now sits on the success path only, ...

Quoted 9 lines
xen/arch/x86/acpi/power.c +2 -3
@@ -592,17 +596,10 @@ static int cpu_request_microcode(const void *buf, size_t bufsize)
     }
 
   out:
-#if CONFIG_HVM
-    svm_host_osvw_init();
-#endif
+    if ( error && !patch )
+        patch = ERR_PTR(error);

xen/arch/x86/acpi/power.c static int cpu_request_microcode(const void *buf, size_t bufsize) · line 603

Quoted 10 lines
xen/arch/x86/acpi/power.c +1 -7
 
-    /*
-     * In some cases we may return an error even if processor's microcode has
-     * been updated. For example, the first patch in a container file is loaded
-     * successfully but subsequent container file processing encounters a
-     * failure.
-     */
-    return error;
+    return patch;
 }

... previously it has also been invoked in the error case. See the comment in start_update().

xen/arch/x86/microcode_intel.c static enum microcode_match_result compare_patch( · line 276

Quoted 13 lines
xen/arch/x86/microcode_intel.c +1 -6
@@ -273,46 +273,27 @@ static enum microcode_match_result compare_patch(
                                   old_header->pf, old_header->rev);
 }
 
-/*
- * return 0 - no update found
- * return 1 - found update
- * return < 0 - error
- */
-static int get_matching_microcode(const void *mc)
+static struct microcode_patch *allow_microcode_patch(

Did you perhaps mean this to be alloc_microcode_patch()?

Quoted 30 lines
xen/arch/x86/microcode_intel.c +10 -5
@@ -388,26 +368,39 @@ static long get_next_ucode_from_buffer(void **mc, const u8 *buf,
     return offset + total_size;
 }
 
-static int cpu_request_microcode(const void *buf, size_t size)
+static struct microcode_patch *cpu_request_microcode(const void *buf,
+                                                     size_t size)
 {
     long offset = 0;
     int error = 0;
     void *mc;
+    struct microcode_patch *patch = NULL;
 
     while ( (offset = get_next_ucode_from_buffer(&mc, buf, size, offset)) > 0 )
     {
+        struct microcode_patch *new_patch;
+
         error = microcode_sanity_check(mc);
         if ( error )
             break;
-        error = get_matching_microcode(mc);
-        if ( error < 0 )
+
+        new_patch = allow_microcode_patch(mc);
+        if ( IS_ERR(new_patch) )
+        {
+            error = PTR_ERR(new_patch);
             break;
-        /*
-         * It's possible the data file has multiple matching ucode,

xen/arch/x86/microcode_intel.c static long get_next_ucode_from_buffer(void **mc, const u8 *buf, · line 397

Quoted 10 lines
xen/arch/x86/microcode_intel.c +6 -4
-         * lets keep searching till the latest version
-         */
-        if ( error == 1 )
-            error = 0;
+        }
+
+        /* Compare patches and store the one with higher revision */
+        if ( !patch && match_cpu(new_patch) )
+            patch = new_patch;
+        else if ( patch && (compare_patch(new_patch, patch) == NEW_UCODE) )

At least the appearance of this is misleading, but unless I'm missing something it's actually wrong: You seem to imply that from match_cpu(patch1) returning true and compare_patch(patch2, patch1) returning true it follows that also match_cpu(patch2) would return true. But I don't think that's the case, because of how the "pf" field works.

Even in case I've overlooked an implicit match_cpu(new_patch) I'd like to ask for this to be clarified, either by changing the code structure, or by attaching a suitable comment.

xen/arch/x86/smpboot.c void start_secondary(void *unused) · line 366

Quoted 11 lines
xen/arch/x86/smpboot.c +1 -4
@@ -363,10 +363,7 @@ void start_secondary(void *unused)
 
     initialize_cpu_data(cpu);
 
-    if ( system_state <= SYS_STATE_smp_boot )
-        early_microcode_update_cpu(false);
-    else
-        microcode_resume_cpu();
+    early_microcode_update_cpu();

I'm struggling to understand how you get away without the "false" argument that was passed here before. You look to now be calling ->start_update() unconditionally (so long as the hook is not NULL), which I don't think is correct. This should be called only once by the CPU _leading_ an update (the BSP during boot, and the CPU the hypercall gets invoked on (or the first CPU an update gets issued one) for a late update. Am I missing something?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#23 lore raw

Re: [Xen-devel] [PATCH v7 06/10] microcode: split out apply_microcode() from cpu_request_microcode()

Quoted 14 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

During late microcode update, apply_microcode() is invoked in cpu_request_microcode(). To make late microcode update more reliable, we want to put the apply_microcode() into stop_machine context. So we split out it from cpu_request_microcode(). As a consequence, apply_microcode() should be invoked explicitly in the common code.

Quoted 6 lines

Previously, apply_microcode() gets the microcode patch to be applied from the microcode cache. Now, the patch is passed as a function argument and a patch is cached for cpu-hotplug and cpu resuming, only after it has been loaded to a cpu without any error. As a consequence, the 'match_cpu' check in microcode_update_cache is removed, which otherwise would fail.

Quoted 7 lines

The "only after it has been loaded to a cpu without any error" is a problem, precisely for the case where ucode on the different cores is not in sync initially. I would actually like to put up this question:
When a core has no ucode loaded at all yet and only strictly older (than loaded on some other cores) ucode is found to be available, whether then it wouldn't still be better to apply that ucode to _at least_ the cores that have none loaded yet.

Yes, it is better for this special case. And I agree to support this case.

This in v7, a patch is loaded only if its revision is newer than that loaded to current CPU. And it is stored only if it has been loaded successfully. But, as you described, a broken bios might puts the system in an inconsistent state (multiple microcode revision in the system) and furthermore in this case, if no or an older microcode update is provided, early loading cannot get the system into sane state. So for both early and late microcode loading, we could face a situation that the patch to be loaded has equal or old revision than microcode of some CPUs.

Changes I plan to make in next version are:
1. For early microcode, a patch would be stored if it covers current CPU's signature. All CPUs would try to load from the cache.
2. For late microcode, a patch is loaded only if its revision is newer than *the patch cached*. And it is stored only if has been loaded without an "EIO" error.
3. Cache replacement remains the same.

But it is a temperary solution, especially for CSPs. A better way might be getting the newest ucode or upgrading to the newest bios, even downgrading the bios to an older version which wouldn't put the system into "insane" state.

Quoted 17 lines

To get the system into "sane" state it may even be necessary to downgrade ucode on the cores which did have it loaded already, in such a situation.

Quoted 4 lines

On AMD side, svm_host_osvw_init() is supposed to be called after microcode update. As apply_micrcode() won't be called by cpu_request_microcode() now, svm_host_osvw_init() is moved to the end of apply_microcode().

I guess this really ought to become a vendor hook as well, but I wouldn't insist on you doing so here.

Quoted 4 lines
xen/arch/x86/acpi/power.c +0 -0
@@ -253,7 +253,7 @@ static int enter_state(u32 state)
 
Quoted 8 lines

xen/arch/x86/acpi/power.c static int enter_state(u32 state) · line 256

Quoted 4 lines
xen/arch/x86/acpi/power.c +1 -1
     console_end_sync();
 
-    microcode_resume_cpu();
+    early_microcode_update_cpu();

The use here, the (changed) use in start_secondary(), and the dropping of its __init suggest to make an attempt to find a better name for the function. Maybe microcode_update_one()?

Will do.

Quoted 52 lines

xen/arch/x86/acpi/power.c static int enter_state(u32 state)

Quoted 7 lines
xen/arch/x86/acpi/power.c +7 -0
+/*
+ * Load a microcode update to current CPU.
+ *
+ * If no patch is provided, the cached patch will be loaded. Microcode update
+ * during APs bringup and CPU resuming falls into this case.
+ */
+static int microcode_update_cpu(struct microcode_patch *patch)

const?

Quoted 22 lines

{

xen/arch/x86/acpi/power.c +9 -8
-    int err;
-    struct cpu_signature *sig = &this_cpu(cpu_sig);
+    int ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
 
-    if ( !microcode_ops )
-        return 0;
+    if ( unlikely(ret) )
+        return ret;
 
     spin_lock(&microcode_mutex);
 
-    err = microcode_ops->collect_cpu_info(sig);
-    if ( likely(!err) )
-        err = microcode_ops->apply_microcode();
-    spin_unlock(&microcode_mutex);
+    if ( patch )
+    {
+        /*
+         * If a patch is specified, it should has newer revision than
+         * that of the patch cached.
+         */

xen/arch/x86/acpi/power.c

Quoted 10 lines
xen/arch/x86/acpi/power.c +7 -2
+        if ( microcode_cache &&
+             microcode_ops->compare_patch(patch, microcode_cache) != NEW_UCODE )
+        {
+            spin_unlock(&microcode_mutex);
+            return -EINVAL;
+        }
 
-    return err;
-}
+        ret = microcode_ops->apply_microcode(patch);

There's no printk() here but ...

xen/arch/x86/acpi/power.c

Quoted 6 lines
xen/arch/x86/acpi/power.c +6 -0
+    }
+    else if ( microcode_cache )
+    {
+        ret = microcode_ops->apply_microcode(microcode_cache);
+        if ( ret == -EIO )
+            printk("Update failed. Reboot needed\n");
Quoted 4 lines

... you emit a log message here. Why the difference? And wouldn't it be better to have just a single call to ->apply anyway, by simply assigning "microcode_cache" to "patch" and moving the call a little further down?

-EIO means we loaded the patch but the revision didn't change. This error code indicates an error in the patch. It is unlikely to happen. And in this v7, a patch is stored after being loading successfully on a CPU. To simplify handling of load failure and avoid cleanup to the global cache (if a patch has a success rate, we need to consider which one to save when we have a new patch with 95% rate and an old one with 100% rate, it would be really complex), we assume that loading the cache (being loaded successfully on a CPU) shouldn't return EIO. Otherwise, an error message is prompted.

Quoted 43 lines

xen/arch/x86/acpi/power.c

Quoted 5 lines
xen/arch/x86/acpi/power.c +4 -0
+    }
+    else
+        /* No patch to update */
+        ret = -EINVAL;

-ENOENT?

xen/arch/x86/acpi/power.c bool microcode_update_cache(struct microcode_patch *patch) · line 275

Quoted 13 lines
xen/arch/x86/acpi/power.c +2 -6
@@ -247,46 +270,32 @@ bool microcode_update_cache(struct microcode_patch *patch)
     return true;
 }
 
-static int microcode_update_cpu(const void *buf, size_t size)
+static long do_microcode_update(void *patch)
 {
-    int err;
-    unsigned int cpu = smp_processor_id();
-    struct cpu_signature *sig = &per_cpu(cpu_sig, cpu);
-
-    spin_lock(&microcode_mutex);
+    int error, cpu;

While "int" is fine for error codes, it almost certainly wants to be "unsigned int" for "cpu". The more that it had been that was before. I also don't see why you need to switch from "err" to "error" - oh,
...

xen/arch/x86/acpi/power.c bool microcode_update_cache(struct microcode_patch *patch) · line 276

Quoted 13 lines
xen/arch/x86/acpi/power.c +1 -12
-    err = microcode_ops->collect_cpu_info(sig);
-    if ( likely(!err) )
-        err = microcode_ops->cpu_request_microcode(buf, size);
-    spin_unlock(&microcode_mutex);
-
-    return err;
-}
-
-static long do_microcode_update(void *_info)
-{
-    struct microcode_info *info = _info;
-    int error;
+    error = microcode_update_cpu(patch);

... there was a pre-existing variable of that name here.

Quoted 8 lines

xen/arch/x86/acpi/power.c bool microcode_update_cache(struct microcode_patch *patch) · line 279

Quoted 3 lines
xen/arch/x86/acpi/power.c +3 -0
+    if ( error )
+    {
+        microcode_ops->free_patch(microcode_cache);

Does this also set "microcode_cache" to NULL? I didn't think so. It's anyway not really clear why _all_ forms of errors should lead to clearing of the cache. However - looking at the code further down, don't you rather mean to free "patch" here anyway?

Sorry, it should be "patch".

Quoted 18 lines
Quoted 7 lines
xen/arch/x86/acpi/power.c +2 -2
+        return error;
+    }
 
-    BUG_ON(info->cpu != smp_processor_id());
 
-    error = microcode_update_cpu(info->buffer, info->buffer_size);

xen/arch/x86/acpi/power.c bool microcode_update_cache(struct microcode_patch *patch) · line 288

Quoted 10 lines
xen/arch/x86/acpi/power.c +4 -5
-    if ( error )
-        info->error = error;
+    cpu = cpumask_next(smp_processor_id(), &cpu_online_map);
+    if ( cpu < nr_cpu_ids )
+        return continue_hypercall_on_cpu(cpu, do_microcode_update, patch);
 
-    info->cpu = cpumask_next(info->cpu, &cpu_online_map);
-    if ( info->cpu < nr_cpu_ids )
-        return continue_hypercall_on_cpu(info->cpu, do_microcode_update, info);
+    microcode_update_cache(patch);
Quoted 6 lines

Independent of my remarks at the top I would think that updating of the cache should happen after the first successful loading on a CPU, not after all CPUs have been updated successfully. There would then also not be any need to pass "patch" on to continue_hypercall_on_cpu() a few lines up from here (albeit from a general logic perspective it may indeed be easier to keep it that way).

The logic is removed in the next patch. So I prefer to keep it the same.

Quoted 40 lines
Quoted 40 lines
xen/arch/x86/acpi/power.c +10 -15
@@ -294,32 +303,49 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)
     if ( microcode_ops == NULL )
         return -EINVAL;
 
-    info = xmalloc_bytes(sizeof(*info) + len);
-    if ( info == NULL )
-        return -ENOMEM;
-
-    ret = copy_from_guest(info->buffer, buf, len);
-    if ( ret != 0 )
+    buffer = xmalloc_bytes(len);
+    if ( !buffer )
     {
-        xfree(info);
-        return ret;
+        ret = -ENOMEM;
+        goto free;
     }
 
-    info->buffer_size = len;
-    info->error = 0;
-    info->cpu = cpumask_first(&cpu_online_map);
+    if ( copy_from_guest(buffer, buf, len) )
+    {
+        ret = -EFAULT;
+        goto free;
+    }
 
     if ( microcode_ops->start_update )
     {
         ret = microcode_ops->start_update();
         if ( ret != 0 )
-        {
-            xfree(info);
-            return ret;
-        }
+            goto free;
     }
 
Quoted 11 lines

xen/arch/x86/acpi/power.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 330

Quoted 6 lines
xen/arch/x86/acpi/power.c +5 -1
-    return continue_hypercall_on_cpu(info->cpu, do_microcode_update, info);
+    patch = microcode_parse_blob(buffer, len);
+    if ( IS_ERR(patch) )
+    {
+        printk(XENLOG_ERR "Parsing microcode blob error %ld\n", PTR_ERR(patch));
+        ret = PTR_ERR(patch);

So I assume we would get here when the system already has the newest (or even newer) ucode loaded. That's not an error, and imo no log message suggesting so should be issued. Perhaps the parsing code could return NULL to indicate so?

Yes. 'patch' is set to indicate an error only if no matching patch has been found and an error (for example, -ENOMEM) happens during parsing. If no patch in the blob covers the current cpu signature, the parsing code will return NULL.

Quoted 7 lines

Although, judging
by the code further down you already expect NULL potentially coming back, but I can't seem to be able to figure the condition(s).

Also please switch the two lines around and use "ret" in the printk() invocation.

Quoted 8 lines

xen/arch/x86/acpi/power.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 336

Quoted 6 lines
xen/arch/x86/acpi/power.c +6 -0
+        goto free;
+    }
+
+    if ( !microcode_ops->match_cpu(patch) )
+    {
+        printk(XENLOG_ERR "No matching or newer ucode found. Update aborted!\n");

I assume the "matching" here is meant to cover the CPU signature?

Yes.

Quoted 3 lines

The wording is ambiguous this way, because it could also mean there was no ucode found matching that which is already loaded (which, as per above, may end up being relevant).

Then I suppose it is fine to remove the "matching or".

Quoted 4 lines

Furthermore - why this check, when microcode_parse_blob() already looks for something that's newer than what is currently loaded (and matches the CPU signature)?

Yes. It can be replaced with a null check.

Quoted 56 lines
Quoted 20 lines
xen/arch/x86/acpi/power.c +8 -2
@@ -362,13 +397,41 @@ int __init early_microcode_update_cpu(bool start_update)
     }
     if ( data )
     {
-        if ( start_update && microcode_ops->start_update )
+        struct microcode_patch *patch;
+
+        if ( microcode_ops->start_update )
             rc = microcode_ops->start_update();
 
         if ( rc )
             return rc;
 
-        return microcode_update_cpu(data, len);
+        patch = microcode_parse_blob(data, len);
+        if ( IS_ERR(patch) )
+        {
+            printk(XENLOG_ERR "Parsing microcode blob error %ld\n",
+                   PTR_ERR(patch));

xen/arch/x86/acpi/power.c int __init early_microcode_update_cpu(bool start_update) · line 422

Quoted 10 lines
xen/arch/x86/acpi/power.c +10 -0
+            return PTR_ERR(patch);
+        }
+
+        if ( !microcode_ops->match_cpu(patch) )
+        {
+            printk(XENLOG_ERR "No matching or newer ucode found. Update aborted!\n");
+            if ( patch )
+                microcode_ops->free_patch(patch);
+            return -EINVAL;
+        }

Same remarks here then.

xen/arch/x86/acpi/power.c static int apply_microcode(void) · line 299

Quoted 10 lines
xen/arch/x86/acpi/power.c +4 -0
@@ -292,6 +291,10 @@ static int apply_microcode(void)
 
     sig->rev = rev;
 
+#ifdef CONFIG_HVM
+    svm_host_osvw_init();
+#endif
+
     return 0;
 }

While this now sits on the success path only, ...

Quoted 10 lines
xen/arch/x86/acpi/power.c +2 -3
@@ -592,17 +596,10 @@ static int cpu_request_microcode(const void *buf, size_t bufsize)
     }
 
   out:
-#if CONFIG_HVM
-    svm_host_osvw_init();
-#endif
+    if ( error && !patch )
+        patch = ERR_PTR(error);
 
Quoted 12 lines

xen/arch/x86/acpi/power.c static int cpu_request_microcode(const void *buf, size_t bufsize) · line 603

Quoted 9 lines
xen/arch/x86/acpi/power.c +1 -7
-    /*
-     * In some cases we may return an error even if processor's microcode has
-     * been updated. For example, the first patch in a container file is loaded
-     * successfully but subsequent container file processing encounters a
-     * failure.
-     */
-    return error;
+    return patch;
 }

... previously it has also been invoked in the error case. See the comment in start_update().

It depends on the scope of related MSRs. But I failed to find such information in AMD SDM. So I will find a better place for svm_host_osvw_init(). The last resort is to introduce another callback, probably ->end_update().

Quoted 65 lines
Quoted 4 lines

xen/arch/x86/microcode_intel.c static enum microcode_match_result compare_patch( · line 276

Quoted 10 lines
xen/arch/x86/microcode_intel.c +1 -6
@@ -273,46 +273,27 @@ static enum microcode_match_result compare_patch(
                                   old_header->pf, old_header->rev);
 }
 
-/*
- * return 0 - no update found
- * return 1 - found update
- * return < 0 - error
- */
-static int get_matching_microcode(const void *mc)
+static struct microcode_patch *allow_microcode_patch(

Did you perhaps mean this to be alloc_microcode_patch()?

Quoted 30 lines
xen/arch/x86/microcode_intel.c +10 -5
@@ -388,26 +368,39 @@ static long get_next_ucode_from_buffer(void **mc, const u8 *buf,
     return offset + total_size;
 }
 
-static int cpu_request_microcode(const void *buf, size_t size)
+static struct microcode_patch *cpu_request_microcode(const void *buf,
+                                                     size_t size)
 {
     long offset = 0;
     int error = 0;
     void *mc;
+    struct microcode_patch *patch = NULL;
 
     while ( (offset = get_next_ucode_from_buffer(&mc, buf, size, offset)) > 0 )
     {
+        struct microcode_patch *new_patch;
+
         error = microcode_sanity_check(mc);
         if ( error )
             break;
-        error = get_matching_microcode(mc);
-        if ( error < 0 )
+
+        new_patch = allow_microcode_patch(mc);
+        if ( IS_ERR(new_patch) )
+        {
+            error = PTR_ERR(new_patch);
             break;
-        /*
-         * It's possible the data file has multiple matching ucode,

xen/arch/x86/microcode_intel.c static long get_next_ucode_from_buffer(void **mc, const u8 *buf, · line 397

Quoted 10 lines
xen/arch/x86/microcode_intel.c +6 -4
-         * lets keep searching till the latest version
-         */
-        if ( error == 1 )
-            error = 0;
+        }
+
+        /* Compare patches and store the one with higher revision */
+        if ( !patch && match_cpu(new_patch) )
+            patch = new_patch;
+        else if ( patch && (compare_patch(new_patch, patch) == NEW_UCODE) )

At least the appearance of this is misleading, but unless I'm missing something it's actually wrong: You seem to imply that from match_cpu(patch1) returning true and compare_patch(patch2, patch1) returning true it follows that also match_cpu(patch2) would return true. But I don't think that's the case, because of how the "pf" field works.

Quoted 3 lines

Even in case I've overlooked an implicit match_cpu(new_patch) I'd like to ask for this to be clarified, either by changing the code structure, or by attaching a suitable comment.

I did take it for granted. Checking whether each patch covers current signature before the comparison can solve this problem.

Quoted 7 lines
Quoted 7 lines
xen/arch/x86/smpboot.c +0 -0
@@ -363,10 +363,7 @@ void start_secondary(void *unused)
 
     initialize_cpu_data(cpu);
 
Quoted 13 lines

xen/arch/x86/smpboot.c void start_secondary(void *unused) · line 366

Quoted 5 lines
xen/arch/x86/smpboot.c +1 -4
-    if ( system_state <= SYS_STATE_smp_boot )
-        early_microcode_update_cpu(false);
-    else
-        microcode_resume_cpu();
+    early_microcode_update_cpu();

I'm struggling to understand how you get away without the "false" argument that was passed here before. You look to now be calling ->start_update() unconditionally (so long as the hook is not NULL), which I don't think is correct. This should be called only once by the CPU _leading_ an update (the BSP during boot, and the CPU the hypercall gets invoked on (or the first CPU an update gets issued one) for a late update. Am I missing something?

BSP and APs call different functions, early_microcode_parse_and_update_cpu() and early_microcode_update_cpu() respectively. The latter won't call ->start_update().

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#24 lore raw

Re: [Xen-devel] [PATCH v7 06/10] microcode: split out apply_microcode() from cpu_request_microcode()

Quoted 36 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 05:32, <[email protected]> wrote:
Quoted 14 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

During late microcode update, apply_microcode() is invoked in cpu_request_microcode(). To make late microcode update more reliable, we want to put the apply_microcode() into stop_machine context. So we split out it from cpu_request_microcode(). As a consequence, apply_microcode() should be invoked explicitly in the common code.

Quoted 6 lines

Previously, apply_microcode() gets the microcode patch to be applied from the microcode cache. Now, the patch is passed as a function argument and a patch is cached for cpu-hotplug and cpu resuming, only after it has been loaded to a cpu without any error. As a consequence, the 'match_cpu' check in microcode_update_cache is removed, which otherwise would fail.

Quoted 7 lines

The "only after it has been loaded to a cpu without any error" is a problem, precisely for the case where ucode on the different cores is not in sync initially. I would actually like to put up this question:
When a core has no ucode loaded at all yet and only strictly older (than loaded on some other cores) ucode is found to be available, whether then it wouldn't still be better to apply that ucode to _at least_ the cores that have none loaded yet.

Yes, it is better for this special case. And I agree to support this case.

This in v7, a patch is loaded only if its revision is newer than that loaded to current CPU. And it is stored only if it has been loaded successfully. But, as you described, a broken bios might puts the system in an inconsistent state (multiple microcode revision in the system) and furthermore in this case, if no or an older microcode update is provided, early loading cannot get the system into sane state. So for both early and late microcode loading, we could face a situation that the patch to be loaded has equal or old revision than microcode of some CPUs.

Quoted 7 lines

Changes I plan to make in next version are:
1. For early microcode, a patch would be stored if it covers current CPU's signature. All CPUs would try to load from the cache.
2. For late microcode, a patch is loaded only if its revision is newer than *the patch cached*. And it is stored only if has been loaded without an "EIO" error.
3. Cache replacement remains the same.

Why the difference between early and late loading?

Quoted 4 lines

But it is a temperary solution, especially for CSPs. A better way might be getting the newest ucode or upgrading to the newest bios, even downgrading the bios to an older version which wouldn't put the system into "insane" state.

On the quoted system, all BIOS versions I've ever been provided had the same odd behavior.

Quoted 48 lines
Quoted 43 lines
Quoted 23 lines
xen/arch/x86/microcode.c +10 -8
+static int microcode_update_cpu(struct microcode_patch *patch)
 {
-    int err;
-    struct cpu_signature *sig = &this_cpu(cpu_sig);
+    int ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
 
-    if ( !microcode_ops )
-        return 0;
+    if ( unlikely(ret) )
+        return ret;
 
     spin_lock(&microcode_mutex);
 
-    err = microcode_ops->collect_cpu_info(sig);
-    if ( likely(!err) )
-        err = microcode_ops->apply_microcode();
-    spin_unlock(&microcode_mutex);
+    if ( patch )
+    {
+        /*
+         * If a patch is specified, it should has newer revision than
+         * that of the patch cached.
+         */

xen/arch/x86/microcode.c static DEFINE_SPINLOCK(microcode_mutex); · line 233

Quoted 10 lines
xen/arch/x86/microcode.c +7 -2
+        if ( microcode_cache &&
+             microcode_ops->compare_patch(patch, microcode_cache) != NEW_UCODE )
+        {
+            spin_unlock(&microcode_mutex);
+            return -EINVAL;
+        }
 
-    return err;
-}
+        ret = microcode_ops->apply_microcode(patch);

There's no printk() here but ...

xen/arch/x86/microcode.c static DEFINE_SPINLOCK(microcode_mutex); · line 239

Quoted 6 lines
xen/arch/x86/microcode.c +6 -0
+    }
+    else if ( microcode_cache )
+    {
+        ret = microcode_ops->apply_microcode(microcode_cache);
+        if ( ret == -EIO )
+            printk("Update failed. Reboot needed\n");
Quoted 4 lines

... you emit a log message here. Why the difference? And wouldn't it be better to have just a single call to ->apply anyway, by simply assigning "microcode_cache" to "patch" and moving the call a little further down?

Quoted 9 lines

-EIO means we loaded the patch but the revision didn't change. This error code indicates an error in the patch. It is unlikely to happen. And in this v7, a patch is stored after being loading successfully on a CPU. To simplify handling of load failure and avoid cleanup to the global cache (if a patch has a success rate, we need to consider which one to save when we have a new patch with 95% rate and an old one with 100% rate, it would be really complex), we assume that loading the cache (being loaded successfully on a CPU) shouldn't return EIO. Otherwise, an error message is prompted.

But then the log message itself needs to be more specific. That'll then either allow to understand why one is wanted here but not elsewhere, or additionally a comment should be attached.

Quoted 20 lines
Quoted 6 lines
Quoted 6 lines
xen/arch/x86/smpboot.c +0 -0
@@ -363,10 +363,7 @@ void start_secondary(void *unused)
 
     initialize_cpu_data(cpu);
 
Quoted 13 lines

xen/arch/x86/smpboot.c void start_secondary(void *unused) · line 366

Quoted 5 lines
xen/arch/x86/smpboot.c +1 -4
-    if ( system_state <= SYS_STATE_smp_boot )
-        early_microcode_update_cpu(false);
-    else
-        microcode_resume_cpu();
+    early_microcode_update_cpu();

I'm struggling to understand how you get away without the "false" argument that was passed here before. You look to now be calling ->start_update() unconditionally (so long as the hook is not NULL), which I don't think is correct. This should be called only once by the CPU _leading_ an update (the BSP during boot, and the CPU the hypercall gets invoked on (or the first CPU an update gets issued one) for a late update. Am I missing something?

Quoted 3 lines

BSP and APs call different functions, early_microcode_parse_and_update_cpu() and early_microcode_update_cpu() respectively. The latter won't call ->start_update().

Oh, I see - I'm sorry. I've been mislead by the patch context indicators. But still, by their names there's not supposed to be such a difference. I wonder whether we wouldn't better stick to just one function (early_microcode_update_cpu()), having it take a boolean as is the case now. That would control both parsing and ->start_update() invocation.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#25 lore raw

Re: [Xen-devel] [PATCH v7 06/10] microcode: split out apply_microcode() from cpu_request_microcode()

Quoted 44 lines · Jan Beulich
Quoted 36 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 05:32, <[email protected]> wrote:
Quoted 14 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

During late microcode update, apply_microcode() is invoked in cpu_request_microcode(). To make late microcode update more reliable, we want to put the apply_microcode() into stop_machine context. So we split out it from cpu_request_microcode(). As a consequence, apply_microcode() should be invoked explicitly in the common code.

Quoted 6 lines

Previously, apply_microcode() gets the microcode patch to be applied from the microcode cache. Now, the patch is passed as a function argument and a patch is cached for cpu-hotplug and cpu resuming, only after it has been loaded to a cpu without any error. As a consequence, the 'match_cpu' check in microcode_update_cache is removed, which otherwise would fail.

Quoted 7 lines

The "only after it has been loaded to a cpu without any error" is a problem, precisely for the case where ucode on the different cores is not in sync initially. I would actually like to put up this question:
When a core has no ucode loaded at all yet and only strictly older (than loaded on some other cores) ucode is found to be available, whether then it wouldn't still be better to apply that ucode to _at least_ the cores that have none loaded yet.

Yes, it is better for this special case. And I agree to support this case.

This in v7, a patch is loaded only if its revision is newer than that loaded to current CPU. And it is stored only if it has been loaded successfully. But, as you described, a broken bios might puts the system in an inconsistent state (multiple microcode revision in the system) and furthermore in this case, if no or an older microcode update is provided, early loading cannot get the system into sane state. So for both early and late microcode loading, we could face a situation that the patch to be loaded has equal or old revision than microcode of some CPUs.

Quoted 7 lines

Changes I plan to make in next version are:
1. For early microcode, a patch would be stored if it covers current CPU's signature. All CPUs would try to load from the cache.
2. For late microcode, a patch is loaded only if its revision is newer than *the patch cached*. And it is stored only if has been loaded without an "EIO" error.
3. Cache replacement remains the same.

Quoted 1 line

Why the difference between early and late loading?

Storing a patch without loading it is problematic. We need complex logics to restore the old patch if the current patch is proved to be broken. I really want to avoid going this way. So for late microcode, we still stick to the rule: storing a patch only after it has been loaded. For late loading, we can try to load a patch as long as the patch covers current cpu signature to avoid missing any possible update. But thanks to early loading, the oldest microcode revision on all online CPUs shouldn't be older than the cache. So as an optimization, we initiate an update system-wide only if the patch's revision is newer than the cache.

For early loading, to avoid discarding a potential useful patch, an exception is made to store the newest matching patch without loading it and all CPus try to load the patch. One problem is if a broken patch with very high revision is provided, any subsequent attempt of late loading would fail. It is unlikely to happen, so I plan to leave it aside. Otherwise, we can clean up the cache in microcode_init() if no cpu has loaded this patch (we need a global variable to track the status).

Quoted 95 lines
Quoted 5 lines

But it is a temperary solution, especially for CSPs. A better way might be getting the newest ucode or upgrading to the newest bios, even downgrading the bios to an older version which wouldn't put the system into "insane" state.

On the quoted system, all BIOS versions I've ever been provided had the same odd behavior.

Quoted 48 lines
Quoted 43 lines
Quoted 23 lines
xen/arch/x86/microcode.c +10 -8
+static int microcode_update_cpu(struct microcode_patch *patch)
 {
-    int err;
-    struct cpu_signature *sig = &this_cpu(cpu_sig);
+    int ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
 
-    if ( !microcode_ops )
-        return 0;
+    if ( unlikely(ret) )
+        return ret;
 
     spin_lock(&microcode_mutex);
 
-    err = microcode_ops->collect_cpu_info(sig);
-    if ( likely(!err) )
-        err = microcode_ops->apply_microcode();
-    spin_unlock(&microcode_mutex);
+    if ( patch )
+    {
+        /*
+         * If a patch is specified, it should has newer revision than
+         * that of the patch cached.
+         */

xen/arch/x86/microcode.c static DEFINE_SPINLOCK(microcode_mutex); · line 233

Quoted 10 lines
xen/arch/x86/microcode.c +7 -2
+        if ( microcode_cache &&
+             microcode_ops->compare_patch(patch, microcode_cache) != NEW_UCODE )
+        {
+            spin_unlock(&microcode_mutex);
+            return -EINVAL;
+        }
 
-    return err;
-}
+        ret = microcode_ops->apply_microcode(patch);

There's no printk() here but ...

xen/arch/x86/microcode.c static DEFINE_SPINLOCK(microcode_mutex); · line 239

Quoted 6 lines
xen/arch/x86/microcode.c +6 -0
+    }
+    else if ( microcode_cache )
+    {
+        ret = microcode_ops->apply_microcode(microcode_cache);
+        if ( ret == -EIO )
+            printk("Update failed. Reboot needed\n");
Quoted 4 lines

... you emit a log message here. Why the difference? And wouldn't it be better to have just a single call to ->apply anyway, by simply assigning "microcode_cache" to "patch" and moving the call a little further down?

Quoted 9 lines

-EIO means we loaded the patch but the revision didn't change. This error code indicates an error in the patch. It is unlikely to happen. And in this v7, a patch is stored after being loading successfully on a CPU. To simplify handling of load failure and avoid cleanup to the global cache (if a patch has a success rate, we need to consider which one to save when we have a new patch with 95% rate and an old one with 100% rate, it would be really complex), we assume that loading the cache (being loaded successfully on a CPU) shouldn't return EIO. Otherwise, an error message is prompted.

But then the log message itself needs to be more specific. That'll then either allow to understand why one is wanted here but not elsewhere, or additionally a comment should be attached.

Quoted 20 lines
Quoted 6 lines
Quoted 6 lines
xen/arch/x86/smpboot.c +0 -0
@@ -363,10 +363,7 @@ void start_secondary(void *unused)
 
     initialize_cpu_data(cpu);
 
Quoted 13 lines

xen/arch/x86/smpboot.c void start_secondary(void *unused) · line 366

Quoted 5 lines
xen/arch/x86/smpboot.c +1 -4
-    if ( system_state <= SYS_STATE_smp_boot )
-        early_microcode_update_cpu(false);
-    else
-        microcode_resume_cpu();
+    early_microcode_update_cpu();

I'm struggling to understand how you get away without the "false" argument that was passed here before. You look to now be calling ->start_update() unconditionally (so long as the hook is not NULL), which I don't think is correct. This should be called only once by the CPU _leading_ an update (the BSP during boot, and the CPU the hypercall gets invoked on (or the first CPU an update gets issued one) for a late update. Am I missing something?

Quoted 3 lines

BSP and APs call different functions, early_microcode_parse_and_update_cpu() and early_microcode_update_cpu() respectively. The latter won't call ->start_update().

Quoted 6 lines

Oh, I see - I'm sorry. I've been mislead by the patch context indicators. But still, by their names there's not supposed to be such a difference. I wonder whether we wouldn't better stick to just one function (early_microcode_update_cpu()), having it take a boolean as is the case now. That would control both parsing and ->start_update() invocation.

Will do

Thanks
Chao

Quoted 1 line

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#26 lore raw

Re: [Xen-devel] [PATCH v7 06/10] microcode: split out apply_microcode() from cpu_request_microcode()

Quoted 48 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 10:53, <[email protected]> wrote:
Quoted 44 lines · Jan Beulich
Quoted 36 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 05:32, <[email protected]> wrote:
Quoted 14 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

During late microcode update, apply_microcode() is invoked in cpu_request_microcode(). To make late microcode update more reliable, we want to put the apply_microcode() into stop_machine context. So we split out it from cpu_request_microcode(). As a consequence, apply_microcode() should be invoked explicitly in the common code.

Quoted 6 lines

Previously, apply_microcode() gets the microcode patch to be applied from the microcode cache. Now, the patch is passed as a function argument and a patch is cached for cpu-hotplug and cpu resuming, only after it has been loaded to a cpu without any error. As a consequence, the 'match_cpu' check in microcode_update_cache is removed, which otherwise would fail.

Quoted 7 lines

The "only after it has been loaded to a cpu without any error" is a problem, precisely for the case where ucode on the different cores is not in sync initially. I would actually like to put up this question:
When a core has no ucode loaded at all yet and only strictly older (than loaded on some other cores) ucode is found to be available, whether then it wouldn't still be better to apply that ucode to _at least_ the cores that have none loaded yet.

Yes, it is better for this special case. And I agree to support this case.

This in v7, a patch is loaded only if its revision is newer than that loaded to current CPU. And it is stored only if it has been loaded successfully. But, as you described, a broken bios might puts the system in an inconsistent state (multiple microcode revision in the system) and furthermore in this case, if no or an older microcode update is provided, early loading cannot get the system into sane state. So for both early and late microcode loading, we could face a situation that the patch to be loaded has equal or old revision than microcode of some CPUs.

Quoted 7 lines

Changes I plan to make in next version are:
1. For early microcode, a patch would be stored if it covers current CPU's signature. All CPUs would try to load from the cache.
2. For late microcode, a patch is loaded only if its revision is newer than *the patch cached*. And it is stored only if has been loaded without an "EIO" error.
3. Cache replacement remains the same.

Quoted 1 line

Why the difference between early and late loading?

Quoted 9 lines

Storing a patch without loading it is problematic. We need complex logics to restore the old patch if the current patch is proved to be broken. I really want to avoid going this way. So for late microcode, we still stick to the rule: storing a patch only after it has been loaded. For late loading, we can try to load a patch as long as the patch covers current cpu signature to avoid missing any possible update. But thanks to early loading, the oldest microcode revision on all online CPUs shouldn't be older than the cache. So as an optimization, we initiate an update system-wide only if the patch's revision is newer than the cache.

Well, you seem to be considering only one out of two possible cases. What if no ucode was loaded early at all? There's no "thanks to early loading" then.

Quoted 7 lines

For early loading, to avoid discarding a potential useful patch, an exception is made to store the newest matching patch without loading it and all CPus try to load the patch. One problem is if a broken patch with very high revision is provided, any subsequent attempt of late loading would fail. It is unlikely to happen, so I plan to leave it aside. Otherwise, we can clean up the cache in microcode_init() if no cpu has loaded this patch (we need a global variable to track the status).

Yes, if a patch that claims to be newer than what any CPU has already loaded fails to apply on every CPU, then yes, it surely should be purged from the cache.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#27 lore raw

PATCH v7 7/10 microcode/intel: Writeback and invalidate caches before updating microcode

In reply to Chao Gao

Updating microcode is less error prone when caches have been flushed and depending on what exactly the microcode is updating. For example, some of the issues around certain Broadwell parts can be addressed by doing a full cache flush.

With parallel microcode update, the cost of this patch is hardly noticable. Although only BDX with an old microcode needs this fix, we would like to avoid future issues in case they come by later due to other reasons.

[linux commit: 91df9fdf51492aec9fed6b4cbd33160886740f47]
Cc Ashok Raj <[email protected]>

---
Changes in v7:
- explain why we do 'wbinvd' unconditionally rather than only for BDX in commit message

Changes in v6:
- new

---
 xen/arch/x86/microcode_intel.c | 6 ++++++
 1 file changed, 6 insertions(+)
xen/arch/x86/microcode_intel.c +6 -0
index 650495d..bfb48ce 100644
@@ -310,6 +310,12 @@ static int apply_microcode(const struct microcode_patch *patch)
     /* serialize access to the physical write to MSR 0x79 */
     spin_lock_irqsave(&microcode_update_lock, flags);
 
+    /*
+     * Writeback and invalidate caches before updating microcode to avoid
+     * internal issues depending on what the microcode is updating.
+     */
+    wbinvd();
+
     /* write microcode via MSR 0x79 */
     wrmsrl(MSR_IA32_UCODE_WRITE, (unsigned long)mc_intel->bits);
     wrmsrl(MSR_IA32_UCODE_REV, 0x0ULL);
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#28 lore raw

Re: [Xen-devel] [PATCH v7 07/10] microcode/intel: Writeback and invalidate caches before updating microcode

Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

Updating microcode is less error prone when caches have been flushed and depending on what exactly the microcode is updating.

Up to the "and" I understand this sentence, but the rest doesn't really seem to fit. Taking out the good part it seems to me you're saying "Updating microcode is less error prone depending on what exactly the
microcode is updating," which - to me at least - doesn't make a hole lot
of sense. Should it perhaps be

"Updating microcode, depending on what exactly the microcode is
updating, may be less error prone when caches have been flushed."
(The same could perhaps also be achieved by replacing the "and" by a comma.)

Quoted 8 lines

For example, some
of the issues around certain Broadwell parts can be addressed by doing a full cache flush.

With parallel microcode update, the cost of this patch is hardly noticable. Although only BDX with an old microcode needs this fix, we would like to avoid future issues in case they come by later due to other reasons.

I doubt the "hardly noticable" part, and I'm sure you're also aware of the patch (going on top of your series) to make selecting between serial or parallel application a runtime option. But I'm not going to stand in the way if everyone else thinks this is the way to go; it's just that from previous discussions I didn't get such an impression.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#29 lore raw

PATCH v7 8/10 x86/microcode: Synchronize late microcode loading

In reply to Chao Gao

This patch ports microcode improvement patches from linux kernel.

Before you read any further: the early loading method is still the preferred one and you should always do that. The following patch is improving the late loading mechanism for long running jobs and cloud use cases.

Gather all cores and serialize the microcode update on them by doing it one-by-one to make the late update process as reliable as possible and avoid potential issues caused by the microcode update.

[linux commit: a5321aec6412b20b5ad15db2d6b916c05349dbff]
[linux commit: bb8c13d61a629276a162c1d2b1a20a815cbcfbb7]
Cc Kevin Tian <[email protected]>
Cc Jun Nakajima <[email protected]>
Cc Ashok Raj <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Andrew Cooper <[email protected]>
Cc Jan Beulich <[email protected]>

---
Changes in v7:
- Check whether 'timeout' is 0 rather than "<=0" since it is unsigned int.
- reword the comment above microcode_update_cpu() to clearly state that one thread per core should do the update.

Changes in v6:
- Use one timeout period for rendezvous stage and another for update stage.
- scale time to wait by the number of remaining cpus to respond. It helps to find something wrong earlier and thus we can reboot the system earlier.

---
 xen/arch/x86/microcode.c | 171 ++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 155 insertions(+), 16 deletions(-)
xen/arch/x86/microcode.c +155 -16
index 23cf550..f4a417e 100644
@@ -22,6 +22,7 @@
  */
 
 #include <xen/cpu.h>
+#include <xen/cpumask.h>
 #include <xen/lib.h>
 #include <xen/kernel.h>
 #include <xen/init.h>
@@ -30,15 +31,34 @@
 #include <xen/smp.h>
 #include <xen/softirq.h>
 #include <xen/spinlock.h>
+#include <xen/stop_machine.h>
 #include <xen/tasklet.h>
 #include <xen/guest_access.h>
 #include <xen/earlycpio.h>
+#include <xen/watchdog.h>
 
+#include <asm/delay.h>
 #include <asm/msr.h>
 #include <asm/processor.h>
 #include <asm/setup.h>
 #include <asm/microcode.h>
 
+/*
+ * Before performing a late microcode update on any thread, we
+ * rendezvous all cpus in stop_machine context. The timeout for
+ * waiting for cpu rendezvous is 30ms. It is the timeout used by
+ * live patching
+ */
+#define MICROCODE_CALLIN_TIMEOUT_US 30000
+
+/*
+ * Timeout for each thread to complete update is set to 1s. It is a
+ * conservative choice considering all possible interference (for
+ * instance, sometimes wbinvd takes relative long time). And a perfect
+ * timeout doesn't help a lot except an early shutdown.
+ */
+#define MICROCODE_UPDATE_TIMEOUT_US 1000000
+
 static module_t __initdata ucode_mod;
 static signed int __initdata ucode_mod_idx;
 static bool_t __initdata ucode_mod_forced;
@@ -190,6 +210,12 @@ static DEFINE_SPINLOCK(microcode_mutex);
 DEFINE_PER_CPU(struct cpu_signature, cpu_sig);
 
 /*
+ * Count the CPUs that have entered, exited the rendezvous and succeeded in
+ * microcode update during late microcode update respectively.
+ */
+static atomic_t cpu_in, cpu_out, cpu_updated;
+
+/*
  * Return the patch with the highest revision id among all matching
  * patches in the blob. Return NULL if no suitable patch.
  */
@@ -270,31 +296,90 @@ bool microcode_update_cache(struct microcode_patch *patch)
     return true;
 }
 
-static long do_microcode_update(void *patch)
+/* Wait for CPUs to rendezvous with a timeout (us) */
+static int wait_for_cpus(atomic_t *cnt, unsigned int expect,
+                         unsigned int timeout)
 {
-    int error, cpu;
-
-    error = microcode_update_cpu(patch);
-    if ( error )
+    while ( atomic_read(cnt) < expect )
     {
-        microcode_ops->free_patch(microcode_cache);
-        return error;
+        if ( !timeout )
+        {
+            printk("CPU%d: Timeout when waiting for CPUs calling in\n",
+                   smp_processor_id());
+            return -EBUSY;
+        }
+        udelay(1);
+        timeout--;
     }
 
+    return 0;
+}
 
-    cpu = cpumask_next(smp_processor_id(), &cpu_online_map);
-    if ( cpu < nr_cpu_ids )
-        return continue_hypercall_on_cpu(cpu, do_microcode_update, patch);
+static int do_microcode_update(void *patch)
+{
+    unsigned int cpu = smp_processor_id();
+    unsigned int cpu_nr = num_online_cpus();
+    unsigned int finished;
+    int ret;
+    static bool error;
 
-    microcode_update_cache(patch);
+    atomic_inc(&cpu_in);
+    ret = wait_for_cpus(&cpu_in, cpu_nr, MICROCODE_CALLIN_TIMEOUT_US);
+    if ( ret )
+        return ret;
 
-    return error;
+    ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
+    /*
+     * Load microcode update on only one logical processor per core.
+     * Here, among logical processors of a core, the one with the
+     * lowest thread id is chosen to perform the loading.
+     */
+    if ( !ret && (cpu == cpumask_first(per_cpu(cpu_sibling_mask, cpu))) )
+    {
+        ret = microcode_ops->apply_microcode(patch);
+        if ( !ret )
+            atomic_inc(&cpu_updated);
+    }
+    /*
+     * Increase the wait timeout to a safe value here since we're serializing
+     * the microcode update and that could take a while on a large number of
+     * CPUs. And that is fine as the *actual* timeout will be determined by
+     * the last CPU finished updating and thus cut short
+     */
+    atomic_inc(&cpu_out);
+    finished = atomic_read(&cpu_out);
+    while ( !error && finished != cpu_nr )
+    {
+        /*
+         * During each timeout interval, at least a CPU is expected to
+         * finish its update. Otherwise, something goes wrong.
+         */
+        if ( wait_for_cpus(&cpu_out, finished + 1,
+                           MICROCODE_UPDATE_TIMEOUT_US) && !error )
+        {
+            error = true;
+            panic("Timeout when finishing updating microcode (finished %d/%d)",
+                  finished, cpu_nr);
+        }
+
+        finished = atomic_read(&cpu_out);
+    }
+
+    /*
+     * Refresh CPU signature (revision) on threads which didn't call
+     * apply_microcode().
+     */
+    if ( cpu != cpumask_first(per_cpu(cpu_sibling_mask, cpu)) )
+        ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
+
+    return ret;
 }
 
 int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)
 {
     int ret;
     void *buffer;
+    unsigned int cpu, nr_cores;
     struct microcode_patch *patch;
 
     if ( len != (uint32_t)len )
@@ -316,11 +401,18 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)
         goto free;
     }
 
+    /* cpu_online_map must not change during update */
+    if ( !get_cpu_maps() )
+    {
+        ret = -EBUSY;
+        goto free;
+    }
+
     if ( microcode_ops->start_update )
     {
         ret = microcode_ops->start_update();
         if ( ret != 0 )
-            goto free;
+            goto put;
     }
 
     patch = microcode_parse_blob(buffer, len);
@@ -337,12 +429,59 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)
         if ( patch )
             microcode_ops->free_patch(patch);
         ret = -EINVAL;
-        goto free;
+        goto put;
     }
 
-    ret = continue_hypercall_on_cpu(cpumask_first(&cpu_online_map),
-                                    do_microcode_update, patch);
+    atomic_set(&cpu_in, 0);
+    atomic_set(&cpu_out, 0);
+    atomic_set(&cpu_updated, 0);
+
+    /* Calculate the number of online CPU core */
+    nr_cores = 0;
+    for_each_online_cpu(cpu)
+        if ( cpu == cpumask_first(per_cpu(cpu_sibling_mask, cpu)) )
+            nr_cores++;
+
+    printk(XENLOG_INFO "%d cores are to update their microcode\n", nr_cores);
+
+    /*
+     * We intend to disable interrupt for long time, which may lead to
+     * watchdog timeout.
+     */
+    watchdog_disable();
+    /*
+     * Late loading dance. Why the heavy-handed stop_machine effort?
+     *
+     * - HT siblings must be idle and not execute other code while the other
+     *   sibling is loading microcode in order to avoid any negative
+     *   interactions cause by the loading.
+     *
+     * - In addition, microcode update on the cores must be serialized until
+     *   this requirement can be relaxed in the future. Right now, this is
+     *   conservative and good.
+     */
+    ret = stop_machine_run(do_microcode_update, patch, NR_CPUS);
+    watchdog_enable();
+
+    if ( atomic_read(&cpu_updated) == nr_cores )
+    {
+        spin_lock(&microcode_mutex);
+        microcode_update_cache(patch);
+        spin_unlock(&microcode_mutex);
+    }
+    else if ( atomic_read(&cpu_updated) == 0 )
+        microcode_ops->free_patch(patch);
+    else
+    {
+        printk("Updating microcode succeeded on part of CPUs and failed on\n"
+               "others due to an unknown reason. A system with different\n"
+               "microcode revisions is considered unstable. Please reboot and\n"
+               "do not load the microcode that triggers this warning\n");
+        microcode_ops->free_patch(patch);
+    }
 
+ put:
+    put_cpu_maps();
  free:
     xfree(buffer);
     return ret;
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#30 lore raw

Re: [Xen-devel] [PATCH v7 08/10] x86/microcode: Synchronize late microcode loading

Quoted 43 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

This patch ports microcode improvement patches from linux kernel.

Before you read any further: the early loading method is still the preferred one and you should always do that. The following patch is improving the late loading mechanism for long running jobs and cloud use cases.

Gather all cores and serialize the microcode update on them by doing it one-by-one to make the late update process as reliable as possible and avoid potential issues caused by the microcode update.

[linux commit: a5321aec6412b20b5ad15db2d6b916c05349dbff]
[linux commit: bb8c13d61a629276a162c1d2b1a20a815cbcfbb7]
Cc Kevin Tian <[email protected]>
Cc Jun Nakajima <[email protected]>
Cc Ashok Raj <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Andrew Cooper <[email protected]>
Cc Jan Beulich <[email protected]>

---
Changes in v7:
- Check whether 'timeout' is 0 rather than "<=0" since it is unsigned int.
- reword the comment above microcode_update_cpu() to clearly state that one thread per core should do the update.

Changes in v6:
- Use one timeout period for rendezvous stage and another for update stage.
- scale time to wait by the number of remaining cpus to respond. It helps to find something wrong earlier and thus we can reboot the system earlier.

---
 xen/arch/x86/microcode.c | 171 ++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 155 insertions(+), 16 deletions(-)
xen/arch/x86/microcode.c +0 -0
index 23cf550..f4a417e 100644

xen/arch/x86/microcode.c line 25

Quoted 4 lines
xen/arch/x86/microcode.c +1 -0
@@ -22,6 +22,7 @@
  */
 
 #include <xen/cpu.h>
+#include <xen/cpumask.h>

It seems vanishingly unlikely that you would need this explicit #include here, but it certainly isn't wrong.

Quoted 16 lines
xen/arch/x86/microcode.c +4 -6
@@ -270,31 +296,90 @@ bool microcode_update_cache(struct microcode_patch *patch)
     return true;
 }
 
-static long do_microcode_update(void *patch)
+/* Wait for CPUs to rendezvous with a timeout (us) */
+static int wait_for_cpus(atomic_t *cnt, unsigned int expect,
+                         unsigned int timeout)
 {
-    int error, cpu;
-
-    error = microcode_update_cpu(patch);
-    if ( error )
+    while ( atomic_read(cnt) < expect )
     {
-        microcode_ops->free_patch(microcode_cache);

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 313

Quoted 10 lines
xen/arch/x86/microcode.c +8 -1
-        return error;
+        if ( !timeout )
+        {
+            printk("CPU%d: Timeout when waiting for CPUs calling in\n",
+                   smp_processor_id());
+            return -EBUSY;
+        }
+        udelay(1);
+        timeout--;
     }

There's no comment here and nothing in the description: I don't recall clarification as to whether RDTSC is fine to be issued by a thread when ucode is being updated by another thread on the same core.

Quoted 12 lines
xen/arch/x86/microcode.c +10 -1
+static int do_microcode_update(void *patch)
+{
+    unsigned int cpu = smp_processor_id();
+    unsigned int cpu_nr = num_online_cpus();
+    unsigned int finished;
+    int ret;
+    static bool error;
 
-    microcode_update_cache(patch);
+    atomic_inc(&cpu_in);
+    ret = wait_for_cpus(&cpu_in, cpu_nr, MICROCODE_CALLIN_TIMEOUT_US);
+    if ( ret )

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 337

Quoted 10 lines
xen/arch/x86/microcode.c +8 -1
+        return ret;
 
-    return error;
+    ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
+    /*
+     * Load microcode update on only one logical processor per core.
+     * Here, among logical processors of a core, the one with the
+     * lowest thread id is chosen to perform the loading.
+     */
+    if ( !ret && (cpu == cpumask_first(per_cpu(cpu_sibling_mask, cpu))) )

At the very least it's not obvious whether this hyper-threading-centric view ("logical processor") also applies to AMD's compute unit model (which reuses cpu_sibling_mask). It does, as the respective MSRs are per-compute-unit rather than per-core, but I'd appreciate if the wording could be adjusted to explicitly name both cases (multiple threads per core and multiple cores per CU).

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 344

Quoted 7 lines
xen/arch/x86/microcode.c +7 -0
+    {
+        ret = microcode_ops->apply_microcode(patch);
+        if ( !ret )
+            atomic_inc(&cpu_updated);
+    }
+    /*
+     * Increase the wait timeout to a safe value here since we're serializing

I'm struggling with the "increase": I don't see anything being increased here. You simply use a larger timeout than above.

Quoted 8 lines
xen/arch/x86/microcode.c +8 -0
+     * the microcode update and that could take a while on a large number of
+     * CPUs. And that is fine as the *actual* timeout will be determined by
+     * the last CPU finished updating and thus cut short
+     */
+    atomic_inc(&cpu_out);
+    finished = atomic_read(&cpu_out);
+    while ( !error && finished != cpu_nr )
+    {

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 362

Quoted 10 lines
xen/arch/x86/microcode.c +10 -0
+        /*
+         * During each timeout interval, at least a CPU is expected to
+         * finish its update. Otherwise, something goes wrong.
+         */
+        if ( wait_for_cpus(&cpu_out, finished + 1,
+                           MICROCODE_UPDATE_TIMEOUT_US) && !error )
+        {
+            error = true;
+            panic("Timeout when finishing updating microcode (finished %d/%d)",
+                  finished, cpu_nr);

Why the setting of "error" when you panic anyway?

And please use format specifiers matching the types of the further arguments (i.e. twice %u here, but please check other code as well).

Furthermore (and I'm sure I've given this comment before) if you really hit the limit, how many panic() invocations are there going to be? You run this function on all CPUs after all.

On the whole, taking a 256-thread system as example, you allow the whole process to take over 4 min without calling panic(). Leaving aside guests, I don't think Xen itself would survive this in all cases. We've found the need to process softirqs with far smaller delays, in particular from key handlers producing lots of output. At the very least there should be a bold warning logged if the system had been in stop-machine state for, say, longer than 100ms (value subject to discussion).

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 373

Quoted 11 lines
xen/arch/x86/microcode.c +11 -0
+        }
+
+        finished = atomic_read(&cpu_out);
+    }
+
+    /*
+     * Refresh CPU signature (revision) on threads which didn't call
+     * apply_microcode().
+     */
+    if ( cpu != cpumask_first(per_cpu(cpu_sibling_mask, cpu)) )
+        ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));

Another option would be for the CPU doing the update to simply propagate the new value to all its siblings' cpu_sig values.

Quoted 47 lines
xen/arch/x86/microcode.c +38 -3
@@ -337,12 +429,59 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)
         if ( patch )
             microcode_ops->free_patch(patch);
         ret = -EINVAL;
-        goto free;
+        goto put;
     }
 
-    ret = continue_hypercall_on_cpu(cpumask_first(&cpu_online_map),
-                                    do_microcode_update, patch);
+    atomic_set(&cpu_in, 0);
+    atomic_set(&cpu_out, 0);
+    atomic_set(&cpu_updated, 0);
+
+    /* Calculate the number of online CPU core */
+    nr_cores = 0;
+    for_each_online_cpu(cpu)
+        if ( cpu == cpumask_first(per_cpu(cpu_sibling_mask, cpu)) )
+            nr_cores++;
+
+    printk(XENLOG_INFO "%d cores are to update their microcode\n", nr_cores);
+
+    /*
+     * We intend to disable interrupt for long time, which may lead to
+     * watchdog timeout.
+     */
+    watchdog_disable();
+    /*
+     * Late loading dance. Why the heavy-handed stop_machine effort?
+     *
+     * - HT siblings must be idle and not execute other code while the other
+     *   sibling is loading microcode in order to avoid any negative
+     *   interactions cause by the loading.
+     *
+     * - In addition, microcode update on the cores must be serialized until
+     *   this requirement can be relaxed in the future. Right now, this is
+     *   conservative and good.
+     */
+    ret = stop_machine_run(do_microcode_update, patch, NR_CPUS);
+    watchdog_enable();
+
+    if ( atomic_read(&cpu_updated) == nr_cores )
+    {
+        spin_lock(&microcode_mutex);
+        microcode_update_cache(patch);
+        spin_unlock(&microcode_mutex);
+    }

xen/arch/x86/microcode.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 481

Quoted 10 lines
xen/arch/x86/microcode.c +10 -0
+    else if ( atomic_read(&cpu_updated) == 0 )
+        microcode_ops->free_patch(patch);
+    else
+    {
+        printk("Updating microcode succeeded on part of CPUs and failed on\n"
+               "others due to an unknown reason. A system with different\n"
+               "microcode revisions is considered unstable. Please reboot and\n"
+               "do not load the microcode that triggers this warning\n");
+        microcode_ops->free_patch(patch);
+    }

As said on an earlier patch, I think the cache can be updated if at least one CPU loaded the blob successfully. Additionally I'd like to ask that you log the number of successfully updated cores. And finally perhaps "differing" instead of "different" and omit "due to an unknown reason"?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#31 lore raw

Re: [Xen-devel] [PATCH v7 08/10] x86/microcode: Synchronize late microcode loading

Quoted 74 lines · Jan Beulich
Quoted 43 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

This patch ports microcode improvement patches from linux kernel.

Before you read any further: the early loading method is still the preferred one and you should always do that. The following patch is improving the late loading mechanism for long running jobs and cloud use cases.

Gather all cores and serialize the microcode update on them by doing it one-by-one to make the late update process as reliable as possible and avoid potential issues caused by the microcode update.

[linux commit: a5321aec6412b20b5ad15db2d6b916c05349dbff]
[linux commit: bb8c13d61a629276a162c1d2b1a20a815cbcfbb7]
Cc Kevin Tian <[email protected]>
Cc Jun Nakajima <[email protected]>
Cc Ashok Raj <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Andrew Cooper <[email protected]>
Cc Jan Beulich <[email protected]>

---
Changes in v7:
- Check whether 'timeout' is 0 rather than "<=0" since it is unsigned int.
- reword the comment above microcode_update_cpu() to clearly state that one thread per core should do the update.

Changes in v6:
- Use one timeout period for rendezvous stage and another for update stage.
- scale time to wait by the number of remaining cpus to respond. It helps to find something wrong earlier and thus we can reboot the system earlier.

---
 xen/arch/x86/microcode.c | 171 ++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 155 insertions(+), 16 deletions(-)
xen/arch/x86/microcode.c +0 -0
index 23cf550..f4a417e 100644

xen/arch/x86/microcode.c line 25

Quoted 4 lines
xen/arch/x86/microcode.c +1 -0
@@ -22,6 +22,7 @@
  */
 
 #include <xen/cpu.h>
+#include <xen/cpumask.h>

It seems vanishingly unlikely that you would need this explicit #include here, but it certainly isn't wrong.

Quoted 23 lines
xen/arch/x86/microcode.c +10 -7
@@ -270,31 +296,90 @@ bool microcode_update_cache(struct microcode_patch *patch)
     return true;
 }
 
-static long do_microcode_update(void *patch)
+/* Wait for CPUs to rendezvous with a timeout (us) */
+static int wait_for_cpus(atomic_t *cnt, unsigned int expect,
+                         unsigned int timeout)
 {
-    int error, cpu;
-
-    error = microcode_update_cpu(patch);
-    if ( error )
+    while ( atomic_read(cnt) < expect )
     {
-        microcode_ops->free_patch(microcode_cache);
-        return error;
+        if ( !timeout )
+        {
+            printk("CPU%d: Timeout when waiting for CPUs calling in\n",
+                   smp_processor_id());
+            return -EBUSY;
+        }
Quoted 8 lines

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 313

Quoted 3 lines
xen/arch/x86/microcode.c +2 -0
+        udelay(1);
+        timeout--;
     }

There's no comment here and nothing in the description: I don't recall clarification as to whether RDTSC is fine to be issued by a thread when ucode is being updated by another thread on the same core.

Yes. I think it is fine.

Ashok, could you share your opinion on this question?

Quoted 15 lines
Quoted 15 lines
xen/arch/x86/microcode.c +11 -1
+static int do_microcode_update(void *patch)
+{
+    unsigned int cpu = smp_processor_id();
+    unsigned int cpu_nr = num_online_cpus();
+    unsigned int finished;
+    int ret;
+    static bool error;
 
-    microcode_update_cache(patch);
+    atomic_inc(&cpu_in);
+    ret = wait_for_cpus(&cpu_in, cpu_nr, MICROCODE_CALLIN_TIMEOUT_US);
+    if ( ret )
+        return ret;
 
Quoted 15 lines

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 337

Quoted 8 lines
xen/arch/x86/microcode.c +7 -1
-    return error;
+    ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
+    /*
+     * Load microcode update on only one logical processor per core.
+     * Here, among logical processors of a core, the one with the
+     * lowest thread id is chosen to perform the loading.
+     */
+    if ( !ret && (cpu == cpumask_first(per_cpu(cpu_sibling_mask, cpu))) )

At the very least it's not obvious whether this hyper-threading-centric view ("logical processor") also applies to AMD's compute unit model (which reuses cpu_sibling_mask). It does, as the respective MSRs are per-compute-unit rather than per-core, but I'd appreciate if the wording could be adjusted to explicitly name both cases (multiple threads per core and multiple cores per CU).

OK. Will do

Quoted 33 lines

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 344

Quoted 8 lines
xen/arch/x86/microcode.c +7 -0
+    {
+        ret = microcode_ops->apply_microcode(patch);
+        if ( !ret )
+            atomic_inc(&cpu_updated);
+    }
+    /*
+     * Increase the wait timeout to a safe value here since we're serializing

I'm struggling with the "increase": I don't see anything being increased here. You simply use a larger timeout than above.

Quoted 8 lines
xen/arch/x86/microcode.c +8 -0
+     * the microcode update and that could take a while on a large number of
+     * CPUs. And that is fine as the *actual* timeout will be determined by
+     * the last CPU finished updating and thus cut short
+     */
+    atomic_inc(&cpu_out);
+    finished = atomic_read(&cpu_out);
+    while ( !error && finished != cpu_nr )
+    {

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 362

Quoted 10 lines
xen/arch/x86/microcode.c +10 -0
+        /*
+         * During each timeout interval, at least a CPU is expected to
+         * finish its update. Otherwise, something goes wrong.
+         */
+        if ( wait_for_cpus(&cpu_out, finished + 1,
+                           MICROCODE_UPDATE_TIMEOUT_US) && !error )
+        {
+            error = true;
+            panic("Timeout when finishing updating microcode (finished %d/%d)",
+                  finished, cpu_nr);

Why the setting of "error" when you panic anyway?

Quoted 7 lines

And please use format specifiers matching the types of the further arguments (i.e. twice %u here, but please check other code as well).

Furthermore (and I'm sure I've given this comment before) if you really hit the limit, how many panic() invocations are there going to be? You run this function on all CPUs after all.

"error" is to avoid calling of panic() on multiple CPUs simultaneously. Roger is right: atomic primitives should be used here.

Quoted 11 lines

On the whole, taking a 256-thread system as example, you allow the whole process to take over 4 min without calling panic().
Leaving aside guests, I don't think Xen itself would survive this in all cases. We've found the need to process softirqs with far smaller delays, in particular from key handlers producing lots of output. At the very least there should be a bold warning logged if the system had been in stop-machine state for, say, longer than 100ms (value subject to discussion).

In theory, if you mean 256 cores, yes. Do you think a configurable and run-time changeable upper bound for the whole process can address your concern? The default value for this upper bound can be set to a large value (for example, 1s * the number of online core) and the admin can ajust/lower the upper bound according to the way (serial or parallel) to perform the update and other requirements. Once the upper bound is reached, we would call panic().

Quoted 6 lines
Quoted 6 lines
xen/arch/x86/microcode.c +6 -0
+        }
+
+        finished = atomic_read(&cpu_out);
+    }
+
+    /*
Quoted 8 lines

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 373

Quoted 5 lines
xen/arch/x86/microcode.c +5 -0
+     * Refresh CPU signature (revision) on threads which didn't call
+     * apply_microcode().
+     */
+    if ( cpu != cpumask_first(per_cpu(cpu_sibling_mask, cpu)) )
+        ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));

Another option would be for the CPU doing the update to simply propagate the new value to all its siblings' cpu_sig values.

Will do.

Quoted 56 lines
Quoted 56 lines
xen/arch/x86/microcode.c +46 -3
@@ -337,12 +429,59 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)
         if ( patch )
             microcode_ops->free_patch(patch);
         ret = -EINVAL;
-        goto free;
+        goto put;
     }
 
-    ret = continue_hypercall_on_cpu(cpumask_first(&cpu_online_map),
-                                    do_microcode_update, patch);
+    atomic_set(&cpu_in, 0);
+    atomic_set(&cpu_out, 0);
+    atomic_set(&cpu_updated, 0);
+
+    /* Calculate the number of online CPU core */
+    nr_cores = 0;
+    for_each_online_cpu(cpu)
+        if ( cpu == cpumask_first(per_cpu(cpu_sibling_mask, cpu)) )
+            nr_cores++;
+
+    printk(XENLOG_INFO "%d cores are to update their microcode\n", nr_cores);
+
+    /*
+     * We intend to disable interrupt for long time, which may lead to
+     * watchdog timeout.
+     */
+    watchdog_disable();
+    /*
+     * Late loading dance. Why the heavy-handed stop_machine effort?
+     *
+     * - HT siblings must be idle and not execute other code while the other
+     *   sibling is loading microcode in order to avoid any negative
+     *   interactions cause by the loading.
+     *
+     * - In addition, microcode update on the cores must be serialized until
+     *   this requirement can be relaxed in the future. Right now, this is
+     *   conservative and good.
+     */
+    ret = stop_machine_run(do_microcode_update, patch, NR_CPUS);
+    watchdog_enable();
+
+    if ( atomic_read(&cpu_updated) == nr_cores )
+    {
+        spin_lock(&microcode_mutex);
+        microcode_update_cache(patch);
+        spin_unlock(&microcode_mutex);
+    }
+    else if ( atomic_read(&cpu_updated) == 0 )
+        microcode_ops->free_patch(patch);
+    else
+    {
+        printk("Updating microcode succeeded on part of CPUs and failed on\n"
+               "others due to an unknown reason. A system with different\n"
+               "microcode revisions is considered unstable. Please reboot and\n"
+               "do not load the microcode that triggers this warning\n");
Quoted 8 lines

xen/arch/x86/microcode.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 481

Quoted 2 lines
xen/arch/x86/microcode.c +2 -0
+        microcode_ops->free_patch(patch);
+    }

As said on an earlier patch, I think the cache can be updated if at least one CPU loaded the blob successfully. Additionally I'd like to ask that you log the number of successfully updated cores. And finally perhaps "differing" instead of "different" and omit "due to an unknown reason"?

Will do.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#32 lore raw

Re: [Xen-devel] [PATCH v7 08/10] x86/microcode: Synchronize late microcode loading

Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 14:36, <[email protected]> wrote:
Quoted 10 lines · Jan Beulich
Quoted 1 line
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

On the whole, taking a 256-thread system as example, you allow the whole process to take over 4 min without calling panic().
Leaving aside guests, I don't think Xen itself would survive this in all cases. We've found the need to process softirqs with far smaller delays, in particular from key handlers producing lots of output. At the very least there should be a bold warning logged if the system had been in stop-machine state for, say, longer than 100ms (value subject to discussion).

Quoted 7 lines

In theory, if you mean 256 cores, yes. Do you think a configurable and run-time changeable upper bound for the whole process can address your concern? The default value for this upper bound can be set to a large value (for example, 1s * the number of online core) and the admin can ajust/lower the upper bound according to the way (serial or parallel) to perform the update and other requirements. Once the upper bound is reached, we would call panic().

Well, a command line option to control the total time until calling panic() may help, but as you've said in the past: If we panic anyway, it doesn't matter much what the timeout is. My point was rather to make explicit that the process may have completed after a (too) long time. Remember you mean this late loading to happen with guests running. We should avoid making the system unstable as much as we can. This includes this taking long and the completing successfully _as well as_ calling panic().

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#33 lore raw

Re: [Xen-devel] [PATCH v7 08/10] x86/microcode: Synchronize late microcode loading

In reply to Chao Gao

Hi Gao, Jan

Quoted 11 lines · Chao Gao
Quoted 5 lines · Jan Beulich

There's no comment here and nothing in the description: I don't recall clarification as to whether RDTSC is fine to be issued by a thread when ucode is being updated by another thread on the same core.

Yes. I think it is fine.

Ashok, could you share your opinion on this question?

Yes, rdtsc should be fine for other threads to execute while waiting for the microcode update to complete on others. We do the same in Linux as well.

Cheers,
Ashok

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#34 lore raw

Re: [Xen-devel] [PATCH v7 08/10] x86/microcode: Synchronize late microcode loading

In reply to Chao Gao
Quoted 68 lines · Chao Gao

This patch ports microcode improvement patches from linux kernel.

Before you read any further: the early loading method is still the preferred one and you should always do that. The following patch is improving the late loading mechanism for long running jobs and cloud use cases.

Gather all cores and serialize the microcode update on them by doing it one-by-one to make the late update process as reliable as possible and avoid potential issues caused by the microcode update.

[linux commit: a5321aec6412b20b5ad15db2d6b916c05349dbff]
[linux commit: bb8c13d61a629276a162c1d2b1a20a815cbcfbb7]
Cc Kevin Tian <[email protected]>
Cc Jun Nakajima <[email protected]>
Cc Ashok Raj <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Andrew Cooper <[email protected]>
Cc Jan Beulich <[email protected]>

---
Changes in v7:
- Check whether 'timeout' is 0 rather than "<=0" since it is unsigned int.
- reword the comment above microcode_update_cpu() to clearly state that one thread per core should do the update.

Changes in v6:
- Use one timeout period for rendezvous stage and another for update stage.
- scale time to wait by the number of remaining cpus to respond. It helps to find something wrong earlier and thus we can reboot the system earlier.

---
 xen/arch/x86/microcode.c | 171 ++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 155 insertions(+), 16 deletions(-)
xen/arch/x86/microcode.c +7 -0
index 23cf550..f4a417e 100644
@@ -22,6 +22,7 @@
  */
 
 #include <xen/cpu.h>
+#include <xen/cpumask.h>
 #include <xen/lib.h>
 #include <xen/kernel.h>
 #include <xen/init.h>
@@ -30,15 +31,34 @@
 #include <xen/smp.h>
 #include <xen/softirq.h>
 #include <xen/spinlock.h>
+#include <xen/stop_machine.h>
 #include <xen/tasklet.h>
 #include <xen/guest_access.h>
 #include <xen/earlycpio.h>
+#include <xen/watchdog.h>
 
+#include <asm/delay.h>
 #include <asm/msr.h>
 #include <asm/processor.h>
 #include <asm/setup.h>
 #include <asm/microcode.h>
 
+/*
+ * Before performing a late microcode update on any thread, we
+ * rendezvous all cpus in stop_machine context. The timeout for

xen/arch/x86/microcode.c line 58

Quoted 10 lines
xen/arch/x86/microcode.c +10 -0
+ * waiting for cpu rendezvous is 30ms. It is the timeout used by
+ * live patching
+ */
+#define MICROCODE_CALLIN_TIMEOUT_US 30000
+
+/*
+ * Timeout for each thread to complete update is set to 1s. It is a
+ * conservative choice considering all possible interference (for
+ * instance, sometimes wbinvd takes relative long time). And a perfect
+ * timeout doesn't help a lot except an early shutdown.

I would remove the "And a perfect..." sentence. I don't think it makes much sense to speak about "perfect timeouts".

Quoted 34 lines
xen/arch/x86/microcode.c +13 -5
+ */
+#define MICROCODE_UPDATE_TIMEOUT_US 1000000
+
 static module_t __initdata ucode_mod;
 static signed int __initdata ucode_mod_idx;
 static bool_t __initdata ucode_mod_forced;
@@ -190,6 +210,12 @@ static DEFINE_SPINLOCK(microcode_mutex);
 DEFINE_PER_CPU(struct cpu_signature, cpu_sig);
 
 /*
+ * Count the CPUs that have entered, exited the rendezvous and succeeded in
+ * microcode update during late microcode update respectively.
+ */
+static atomic_t cpu_in, cpu_out, cpu_updated;
+
+/*
  * Return the patch with the highest revision id among all matching
  * patches in the blob. Return NULL if no suitable patch.
  */
@@ -270,31 +296,90 @@ bool microcode_update_cache(struct microcode_patch *patch)
     return true;
 }
 
-static long do_microcode_update(void *patch)
+/* Wait for CPUs to rendezvous with a timeout (us) */
+static int wait_for_cpus(atomic_t *cnt, unsigned int expect,
+                         unsigned int timeout)
 {
-    int error, cpu;
-
-    error = microcode_update_cpu(patch);
-    if ( error )
+    while ( atomic_read(cnt) < expect )
     {

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 312

Quoted 10 lines
xen/arch/x86/microcode.c +8 -2
-        microcode_ops->free_patch(microcode_cache);
-        return error;
+        if ( !timeout )
+        {
+            printk("CPU%d: Timeout when waiting for CPUs calling in\n",
+                   smp_processor_id());
+            return -EBUSY;
+        }
+        udelay(1);
+        timeout--;

Nit: you could do the decrement inside the if condition.

Quoted 30 lines
xen/arch/x86/microcode.c +20 -5
     }
 
+    return 0;
+}
 
-    cpu = cpumask_next(smp_processor_id(), &cpu_online_map);
-    if ( cpu < nr_cpu_ids )
-        return continue_hypercall_on_cpu(cpu, do_microcode_update, patch);
+static int do_microcode_update(void *patch)
+{
+    unsigned int cpu = smp_processor_id();
+    unsigned int cpu_nr = num_online_cpus();
+    unsigned int finished;
+    int ret;
+    static bool error;
 
-    microcode_update_cache(patch);
+    atomic_inc(&cpu_in);
+    ret = wait_for_cpus(&cpu_in, cpu_nr, MICROCODE_CALLIN_TIMEOUT_US);
+    if ( ret )
+        return ret;
 
-    return error;
+    ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
+    /*
+     * Load microcode update on only one logical processor per core.
+     * Here, among logical processors of a core, the one with the
+     * lowest thread id is chosen to perform the loading.
+     */
+    if ( !ret && (cpu == cpumask_first(per_cpu(cpu_sibling_mask, cpu))) )

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 347

Quoted 10 lines
xen/arch/x86/microcode.c +10 -0
+    {
+        ret = microcode_ops->apply_microcode(patch);
+        if ( !ret )
+            atomic_inc(&cpu_updated);
+    }
+    /*
+     * Increase the wait timeout to a safe value here since we're serializing
+     * the microcode update and that could take a while on a large number of
+     * CPUs. And that is fine as the *actual* timeout will be determined by
+     * the last CPU finished updating and thus cut short

It's likely me missing something, but where is this serialization being done?

I assume it's done by apply_microcode because do_microcode_update doesn't do any serialization of microcode loading.

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 360

Quoted 13 lines
xen/arch/x86/microcode.c +13 -0
+     */
+    atomic_inc(&cpu_out);
+    finished = atomic_read(&cpu_out);
+    while ( !error && finished != cpu_nr )
+    {
+        /*
+         * During each timeout interval, at least a CPU is expected to
+         * finish its update. Otherwise, something goes wrong.
+         */
+        if ( wait_for_cpus(&cpu_out, finished + 1,
+                           MICROCODE_UPDATE_TIMEOUT_US) && !error )
+        {
+            error = true;

I'm not sure I see the point of the error variable, you already bring the system down with panic. If the intention is to prevent multiple panics from different threads then you need to use some kind of atomic fetch and set or else the code is racy.

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 361

Quoted 1 line
xen/arch/x86/microcode.c +1 -0
+            panic("Timeout when finishing updating microcode (finished %d/%d)",

Both finished and cpu_nr are unsigned ints, hence you should use %u instead of %d.

xen/arch/x86/microcode.c bool microcode_update_cache(struct microcode_patch *patch) · line 363

Quoted 2 lines
xen/arch/x86/microcode.c +2 -0
+                  finished, cpu_nr);
+        }

This whole loop seems to be designed for serialized microcode application, which is not the case with the current implementation where microcode is updated in parallel on all the cores?

IMO you should just wait for MICROCODE_UPDATE_TIMEOUT_US a single time.

Quoted 52 lines
xen/arch/x86/microcode.c +22 -4
+        finished = atomic_read(&cpu_out);
+    }
+
+    /*
+     * Refresh CPU signature (revision) on threads which didn't call
+     * apply_microcode().
+     */
+    if ( cpu != cpumask_first(per_cpu(cpu_sibling_mask, cpu)) )
+        ret = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
+
+    return ret;
 }
 
 int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)
 {
     int ret;
     void *buffer;
+    unsigned int cpu, nr_cores;
     struct microcode_patch *patch;
 
     if ( len != (uint32_t)len )
@@ -316,11 +401,18 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)
         goto free;
     }
 
+    /* cpu_online_map must not change during update */
+    if ( !get_cpu_maps() )
+    {
+        ret = -EBUSY;
+        goto free;
+    }
+
     if ( microcode_ops->start_update )
     {
         ret = microcode_ops->start_update();
         if ( ret != 0 )
-            goto free;
+            goto put;
     }
 
     patch = microcode_parse_blob(buffer, len);
@@ -337,12 +429,59 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)
         if ( patch )
             microcode_ops->free_patch(patch);
         ret = -EINVAL;
-        goto free;
+        goto put;
     }
 
-    ret = continue_hypercall_on_cpu(cpumask_first(&cpu_online_map),
-                                    do_microcode_update, patch);
+    atomic_set(&cpu_in, 0);

xen/arch/x86/microcode.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 445

Quoted 10 lines
xen/arch/x86/microcode.c +10 -0
+    atomic_set(&cpu_out, 0);
+    atomic_set(&cpu_updated, 0);
+
+    /* Calculate the number of online CPU core */
+    nr_cores = 0;
+    for_each_online_cpu(cpu)
+        if ( cpu == cpumask_first(per_cpu(cpu_sibling_mask, cpu)) )
+            nr_cores++;
+
+    printk(XENLOG_INFO "%d cores are to update their microcode\n", nr_cores);

Same here, nr_cores is unsigned.

Quoted 21 lines
xen/arch/x86/microcode.c +21 -0
+
+    /*
+     * We intend to disable interrupt for long time, which may lead to
+     * watchdog timeout.
+     */
+    watchdog_disable();
+    /*
+     * Late loading dance. Why the heavy-handed stop_machine effort?
+     *
+     * - HT siblings must be idle and not execute other code while the other
+     *   sibling is loading microcode in order to avoid any negative
+     *   interactions cause by the loading.
+     *
+     * - In addition, microcode update on the cores must be serialized until
+     *   this requirement can be relaxed in the future. Right now, this is
+     *   conservative and good.
+     */
+    ret = stop_machine_run(do_microcode_update, patch, NR_CPUS);
+    watchdog_enable();
+
+    if ( atomic_read(&cpu_updated) == nr_cores )

xen/arch/x86/microcode.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 476

Quoted 10 lines
xen/arch/x86/microcode.c +10 -0
+    {
+        spin_lock(&microcode_mutex);
+        microcode_update_cache(patch);
+        spin_unlock(&microcode_mutex);
+    }
+    else if ( atomic_read(&cpu_updated) == 0 )
+        microcode_ops->free_patch(patch);
+    else
+    {
+        printk("Updating microcode succeeded on part of CPUs and failed on\n"

I would prefix this with XENLOG_ERR and an explicit "ERROR: " prefix in the format string.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#35 lore raw

PATCH v7 9/10 microcode: remove microcode_update_lock

In reply to Chao Gao

microcode_update_lock is to prevent logic threads of a same core from updating microcode at the same time. But due to using a global lock, it also prevented parallel microcode updating on different cores.

Remove this lock in order to update microcode in parallel. It is safe because we have already ensured serialization of sibling threads at the caller side.
1.For late microcode update, do_microcode_update() ensures that only one
sibiling thread of a core can update microcode.
2.For microcode update during system startup or CPU-hotplug,
microcode_mutex() guarantees update serialization of logical threads.
3.get/put_cpu_bitmaps() prevents the concurrency of CPU-hotplug and
late microcode update.

Note that printk in apply_microcode() and svm_host_osvm_init() (for AMD only) are still processed sequentially.

---
Changes in v7:
- reworked. Remove complex lock logics introduced in v5 and v6. The microcode patch to be applied is passed as an argument without any global variable. Thus no lock is added to serialize potential readers/writers. Callers of apply_microcode() will guarantee the correctness: the patch poninted by the arguments won't be changed by others.

Changes in v6:
- introduce early_ucode_update_lock to serialize early ucode update.

Changes in v5:
- newly add

---
 xen/arch/x86/microcode_amd.c   | 8 +-------
 xen/arch/x86/microcode_intel.c | 8 +-------
 2 files changed, 2 insertions(+), 14 deletions(-)
xen/arch/x86/microcode_amd.c +1 -7
index c819028..b64a58d 100644
@@ -74,9 +74,6 @@ struct mpbhdr {
     uint8_t data[];
 };
 
-/* serialize access to the physical write */
-static DEFINE_SPINLOCK(microcode_update_lock);
-
 /* See comment in start_update() for cases when this routine fails */
 static int collect_cpu_info(struct cpu_signature *csig)
 {
@@ -251,7 +248,6 @@ static enum microcode_match_result compare_patch(
 
 static int apply_microcode(const struct microcode_patch *patch)
 {
-    unsigned long flags;
     uint32_t rev;
     int hw_err;
     unsigned int cpu = smp_processor_id();
@@ -263,15 +259,13 @@ static int apply_microcode(const struct microcode_patch *patch)
 
     hdr = patch->mc_amd->mpb;
 
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
     hw_err = wrmsr_safe(MSR_AMD_PATCHLOADER, (unsigned long)hdr);
 
     /* get patch id after patching */
     rdmsrl(MSR_AMD_PATCHLEVEL, rev);
 
-    spin_unlock_irqrestore(&microcode_update_lock, flags);
-
     /*
      * Some processors leave the ucode blob mapping as UC after the update.
      * Flush the mapping to regain normal cacheability.
xen/arch/x86/microcode_intel.c +1 -7
index bfb48ce..94a1561 100644
@@ -93,9 +93,6 @@ struct extended_sigtable {
 
 #define exttable_size(et) ((et)->count * EXT_SIGNATURE_SIZE + EXT_HEADER_SIZE)
 
-/* serialize access to the physical write to MSR 0x79 */
-static DEFINE_SPINLOCK(microcode_update_lock);
-
 static int collect_cpu_info(struct cpu_signature *csig)
 {
     unsigned int cpu_num = smp_processor_id();
@@ -295,7 +292,6 @@ static struct microcode_patch *allow_microcode_patch(
 
 static int apply_microcode(const struct microcode_patch *patch)
 {
-    unsigned long flags;
     uint64_t msr_content;
     unsigned int val[2];
     unsigned int cpu_num = raw_smp_processor_id();
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch *patch)
 
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
     /*
      * Writeback and invalidate caches before updating microcode to avoid
@@ -327,7 +322,6 @@ static int apply_microcode(const struct microcode_patch *patch)
     rdmsrl(MSR_IA32_UCODE_REV, msr_content);
     val[1] = (uint32_t)(msr_content >> 32);
 
-    spin_unlock_irqrestore(&microcode_update_lock, flags);
     if ( val[1] != mc_intel->hdr.rev )
     {
         printk(KERN_ERR "microcode: CPU%d update from revision "
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#36 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

Quoted 3 lines · Chao Gao

microcode_update_lock is to prevent logic threads of a same core from updating microcode at the same time. But due to using a global lock, it also prevented parallel microcode updating on different cores.

Oh, OK, so that's what I was missing from patch 8 and what serializes the updating.

Quoted 3 lines

Remove this lock in order to update microcode in parallel. It is safe because we have already ensured serialization of sibling threads at the caller side.

Then you certainly need to fix the wait loop in do_microcode_update to only wait for MICROCODE_UPDATE_TIMEOUT_US regardless of the number of CPUs in the system?

Quoted 11 lines

1.For late microcode update, do_microcode_update() ensures that only one
sibiling thread of a core can update microcode.
2.For microcode update during system startup or CPU-hotplug,
microcode_mutex() guarantees update serialization of logical threads.
3.get/put_cpu_bitmaps() prevents the concurrency of CPU-hotplug and
late microcode update.

Note that printk in apply_microcode() and svm_host_osvm_init() (for AMD only) are still processed sequentially.

Patch LGTM, but it needs to fix the wait loop in do_microcode_update.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#37 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 05.06.19 at 16:52, <[email protected]> wrote:
Quoted 3 lines · Chao Gao

microcode_update_lock is to prevent logic threads of a same core from updating microcode at the same time. But due to using a global lock, it also prevented parallel microcode updating on different cores.

Oh, OK, so that's what I was missing from patch 8 and what serializes the updating.

Quoted 7 lines
Quoted 3 lines

Remove this lock in order to update microcode in parallel. It is safe because we have already ensured serialization of sibling threads at the caller side.

Then you certainly need to fix the wait loop in do_microcode_update to only wait for MICROCODE_UPDATE_TIMEOUT_US regardless of the number of CPUs in the system?

Well, no, not exactly. On huge systems it may indeed still take longer than on smaller ones. The way the waiting is coded now (expecting forward progress in every MICROCODE_UPDATE_TIMEOUT_US period, is, I think, acceptable. Plus leaving that logic alone will avoid touching it yet again when introducing serial application policy as an option.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Reviewed-by on v7 9/10
#38 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

In reply to Chao Gao
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

microcode_update_lock is to prevent logic threads of a same core from updating microcode at the same time. But due to using a global lock, it also prevented parallel microcode updating on different cores.

Remove this lock in order to update microcode in parallel. It is safe because we have already ensured serialization of sibling threads at the caller side.
1.For late microcode update, do_microcode_update() ensures that only one
sibiling thread of a core can update microcode.
2.For microcode update during system startup or CPU-hotplug,
microcode_mutex() guarantees update serialization of logical threads.
3.get/put_cpu_bitmaps() prevents the concurrency of CPU-hotplug and
late microcode update.

Quoted 4 lines

Note that printk in apply_microcode() and svm_host_osvm_init() (for AMD only) are still processed sequentially.

Quoted 7 lines

---
Changes in v7:
- reworked. Remove complex lock logics introduced in v5 and v6. The microcode patch to be applied is passed as an argument without any global variable. Thus no lock is added to serialize potential readers/writers. Callers of apply_microcode() will guarantee the correctness: the patch poninted by the arguments won't be changed by others.

Much better this way indeed.

xen/arch/x86/microcode_intel.c static int apply_microcode(const struct microcode_patch *patch) · line 309

Quoted 10 lines
xen/arch/x86/microcode_intel.c +1 -2
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch *patch)
 
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
     /*
      * Writeback and invalidate caches before updating microcode to avoid

Thinking about it - what happens if we hit an NMI or #MC here? watchdog_disable(), a call to which you add in an earlier patch, doesn't really suppress the generation of NMIs, it only tells the handler not to look at the accumulated statistics.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#39 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

Quoted 15 lines · Jan Beulich
Quoted 15 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

microcode_update_lock is to prevent logic threads of a same core from updating microcode at the same time. But due to using a global lock, it also prevented parallel microcode updating on different cores.

Remove this lock in order to update microcode in parallel. It is safe because we have already ensured serialization of sibling threads at the caller side.
1.For late microcode update, do_microcode_update() ensures that only one
sibiling thread of a core can update microcode.
2.For microcode update during system startup or CPU-hotplug,
microcode_mutex() guarantees update serialization of logical threads.
3.get/put_cpu_bitmaps() prevents the concurrency of CPU-hotplug and
late microcode update.

Quoted 6 lines
Quoted 4 lines

Note that printk in apply_microcode() and svm_host_osvm_init() (for AMD only) are still processed sequentially.

Thanks.

Quoted 19 lines
Quoted 8 lines

---
Changes in v7:
- reworked. Remove complex lock logics introduced in v5 and v6. The microcode patch to be applied is passed as an argument without any global variable. Thus no lock is added to serialize potential readers/writers. Callers of apply_microcode() will guarantee the correctness: the patch poninted by the arguments won't be changed by others.

Much better this way indeed.

Quoted 8 lines
xen/arch/x86/microcode_intel.c +1 -2
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch *patch)
 
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
Quoted 7 lines

xen/arch/x86/microcode_intel.c static int apply_microcode(const struct microcode_patch *patch) · line 309

Quoted 2 lines
xen/arch/x86/microcode_intel.c +0 -0
     /*
      * Writeback and invalidate caches before updating microcode to avoid

Thinking about it - what happens if we hit an NMI or #MC here? watchdog_disable(), a call to which you add in an earlier patch, doesn't really suppress the generation of NMIs, it only tells the handler not to look at the accumulated statistics.

I think they should be suppressed. Ashok, could you confirm it?

I will figure out how to suppress them in Xen.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#40 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 14:46, <[email protected]> wrote:
Quoted 9 lines · Jan Beulich
Quoted 9 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +1 -2
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch *patch)
 
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
Quoted 7 lines

xen/arch/x86/microcode_intel.c static int apply_microcode(const struct microcode_patch *patch) · line 309

Quoted 2 lines
xen/arch/x86/microcode_intel.c +0 -0
     /*
      * Writeback and invalidate caches before updating microcode to avoid

Thinking about it - what happens if we hit an NMI or #MC here? watchdog_disable(), a call to which you add in an earlier patch, doesn't really suppress the generation of NMIs, it only tells the handler not to look at the accumulated statistics.

Quoted 3 lines

I think they should be suppressed. Ashok, could you confirm it?

I will figure out how to suppress them in Xen.

Well, afaik suppressing #MC is impossible. CR4.MCE clear leads to immediate shutdown in case of a machine check, unless I'm mistaken. The watchdog NMI, otoh, could of course be "properly" disabled.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#41 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

In reply to Chao Gao
Quoted 18 lines · Chao Gao
Quoted 9 lines · Jan Beulich
Quoted 9 lines
xen/arch/x86/microcode_intel.c +1 -2
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch *patch)
 
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
Quoted 7 lines

xen/arch/x86/microcode_intel.c static int apply_microcode(const struct microcode_patch *patch) · line 309

Quoted 2 lines
xen/arch/x86/microcode_intel.c +0 -0
     /*
      * Writeback and invalidate caches before updating microcode to avoid

Thinking about it - what happens if we hit an NMI or #MC here? watchdog_disable(), a call to which you add in an earlier patch, doesn't really suppress the generation of NMIs, it only tells the handler not to look at the accumulated statistics.

Quoted 1 line

I think they should be suppressed. Ashok, could you confirm it?

I think the only sources would be the watchdog as you pointed out which we already touch to keep it from expiring. The perf counters i'm not an expert in, but i'll check. When we are in stop_machine() type flow, its not clear if any of those would fire. (I might be wrong, but let me check).

#MC shouldn't fire once you entered the rendezvous, if it does its usually fatal like a 3strike or something. Machine is going down at that time so nothing we could do to handle.

Cheers,
Ashok

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#42 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 18:04, <[email protected]> wrote:
Quoted 3 lines · Chao Gao
Quoted 2 lines · Jan Beulich

Review static int apply_microcode(const struct microcode_patch

Quoted 2 lines
(quoted diff) +0 -0
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch 

*patch)

Quoted 15 lines
Quoted 14 lines

xen/arch/x86/microcode_intel.c static int apply_microcode(const struct microcode_patch *patch) · line 309

Quoted 9 lines
xen/arch/x86/microcode_intel.c +1 -2
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
     /*
      * Writeback and invalidate caches before updating microcode to avoid

Thinking about it - what happens if we hit an NMI or #MC here? watchdog_disable(), a call to which you add in an earlier patch, doesn't really suppress the generation of NMIs, it only tells the handler not to look at the accumulated statistics.

Quoted 7 lines
Quoted 1 line

I think they should be suppressed. Ashok, could you confirm it?

I think the only sources would be the watchdog as you pointed out which we already touch to keep it from expiring. The perf counters i'm not an expert in, but i'll check. When we are in stop_machine() type flow, its not clear if any of those would fire. (I might be wrong, but let me check).

Well, without disarming the watchdog NMI at the LAPIC / IO-APIC, how would it _not_ potentially fire?

Quoted 3 lines

#MC shouldn't fire once you entered the rendezvous, if it does its usually fatal like a 3strike or something. Machine is going down at that time so nothing we could do to handle.

Right - as long as we assume that #MC would be fatal anyway, there's no point in thinking about ways to suppress it. I guess it is fatal (almost) always right now, but I don't think it ought to be. It's just that no-one has the time and environment to make it actually behave better.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#43 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

Quoted 23 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 18:04, <[email protected]> wrote:
Quoted 3 lines · Chao Gao
Quoted 2 lines · Jan Beulich

Review static int apply_microcode(const struct microcode_patch

Quoted 2 lines
(quoted diff) +0 -0
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch 

*patch)

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
xen/arch/x86/microcode_intel.c +1 -2
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
Quoted 7 lines

xen/arch/x86/microcode_intel.c static int apply_microcode(const struct microcode_patch *patch) · line 309

Quoted 2 lines
xen/arch/x86/microcode_intel.c +0 -0
     /*
      * Writeback and invalidate caches before updating microcode to avoid

Thinking about it - what happens if we hit an NMI or #MC here? watchdog_disable(), a call to which you add in an earlier patch, doesn't really suppress the generation of NMIs, it only tells the handler not to look at the accumulated statistics.

I think they should be suppressed. Ashok, could you confirm it?

Quoted 8 lines
Quoted 5 lines

I think the only sources would be the watchdog as you pointed out which we already touch to keep it from expiring. The perf counters i'm not an expert in, but i'll check. When we are in stop_machine() type flow, its not clear if any of those would fire. (I might be wrong, but let me check).

Well, without disarming the watchdog NMI at the LAPIC / IO-APIC, how would it _not_ potentially fire?

We plan not to prevent NMI being fired. Instead, if one thread of a core is updating microcode, other threads of this core would stop in the handler of NMI until the update completion. Is this approach acceptable?

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#44 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

Quoted 31 lines
Quoted 1 line
Quoted 1 line · On 13.06.19 at 16:05, <[email protected]> wrote:
Quoted 29 lines · Jan Beulich
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 18:04, <[email protected]> wrote:
Quoted 3 lines · Chao Gao
Quoted 2 lines · Jan Beulich

Review static int apply_microcode(const struct microcode_patch

Quoted 2 lines
(quoted diff) +0 -0
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch 

*patch)

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
xen/arch/x86/microcode_intel.c +1 -2
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
     /*
      * Writeback and invalidate caches before updating microcode to avoid
Quoted 6 lines
Quoted 4 lines

Thinking about it - what happens if we hit an NMI or #MC here? watchdog_disable(), a call to which you add in an earlier patch, doesn't really suppress the generation of NMIs, it only tells the handler not to look at the accumulated statistics.

I think they should be suppressed. Ashok, could you confirm it?

I think the only sources would be the watchdog as you pointed out which we already touch to keep it from expiring. The perf counters i'm not an expert in, but i'll check. When we are in stop_machine() type flow, its not clear if any of those would fire. (I might be wrong, but let me check).

Quoted 6 lines
Quoted 2 lines

Well, without disarming the watchdog NMI at the LAPIC / IO-APIC, how would it _not_ potentially fire?

We plan not to prevent NMI being fired. Instead, if one thread of a core is updating microcode, other threads of this core would stop in the handler of NMI until the update completion. Is this approach acceptable?

Well, I have to return the question: It is you who knows what is or is not acceptable while an ucode update is in progress. In particular it obviously matters how much ucode is involved in the delivery of an NMI (and in allowing the handler to get to the point where you'd "stop" it).

If the approach you suggest is fine for the NMI case, I'd then wonder if it couldn't also be used for the #MC one.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#45 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

Quoted 38 lines · Jan Beulich
Quoted 31 lines
Quoted 1 line
Quoted 1 line · On 13.06.19 at 16:05, <[email protected]> wrote:
Quoted 29 lines · Jan Beulich
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 18:04, <[email protected]> wrote:
Quoted 3 lines · Chao Gao
Quoted 2 lines · Jan Beulich

Review static int apply_microcode(const struct microcode_patch

Quoted 2 lines
(quoted diff) +0 -0
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch 

*patch)

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
(quoted diff) +1 -2
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
     /*
      * Writeback and invalidate caches before updating microcode to avoid
Quoted 6 lines
Quoted 4 lines

Thinking about it - what happens if we hit an NMI or #MC here? watchdog_disable(), a call to which you add in an earlier patch, doesn't really suppress the generation of NMIs, it only tells the handler not to look at the accumulated statistics.

I think they should be suppressed. Ashok, could you confirm it?

I think the only sources would be the watchdog as you pointed out which we already touch to keep it from expiring. The perf counters i'm not an expert in, but i'll check. When we are in stop_machine() type flow, its not clear if any of those would fire. (I might be wrong, but let me check).

Quoted 6 lines
Quoted 2 lines

Well, without disarming the watchdog NMI at the LAPIC / IO-APIC, how would it _not_ potentially fire?

We plan not to prevent NMI being fired. Instead, if one thread of a core is updating microcode, other threads of this core would stop in the handler of NMI until the update completion. Is this approach acceptable?

Quoted 7 lines

Well, I have to return the question: It is you who knows what is or is not acceptable while an ucode update is in progress. In particular it obviously matters how much ucode is involved in the delivery of an NMI (and in allowing the handler to get to the point where you'd "stop" it).

If the approach you suggest is fine for the NMI case,

Yes. It is fine. It is a suggestion from Ashok and what he is working on in linux kernel. I just wanted to make sure you didn't oppose this approach in Xen (considering disarming watchdog NMI might be an alternative).

Quoted 1 line

I'd then wonder if it couldn't also be used for the #MC one.

I think no much pratical value for #MC because we still need to wait for the callin of all threads. But as you and Ashok said, #MC is usually fatal and machine goes down anyway.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#46 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

In reply to Jan Beulich
Quoted 38 lines · Jan Beulich
Quoted 31 lines
Quoted 1 line
Quoted 1 line · On 13.06.19 at 16:05, <[email protected]> wrote:
Quoted 29 lines · Jan Beulich
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 18:04, <[email protected]> wrote:
Quoted 3 lines · Chao Gao
Quoted 2 lines · Jan Beulich

Review static int apply_microcode(const struct microcode_patch

Quoted 2 lines
(quoted diff) +0 -0
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch 

*patch)

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
(quoted diff) +1 -2
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
     /*
      * Writeback and invalidate caches before updating microcode to avoid
Quoted 6 lines
Quoted 4 lines

Thinking about it - what happens if we hit an NMI or #MC here? watchdog_disable(), a call to which you add in an earlier patch, doesn't really suppress the generation of NMIs, it only tells the handler not to look at the accumulated statistics.

I think they should be suppressed. Ashok, could you confirm it?

I think the only sources would be the watchdog as you pointed out which we already touch to keep it from expiring. The perf counters i'm not an expert in, but i'll check. When we are in stop_machine() type flow, its not clear if any of those would fire. (I might be wrong, but let me check).

Quoted 6 lines
Quoted 2 lines

Well, without disarming the watchdog NMI at the LAPIC / IO-APIC, how would it _not_ potentially fire?

We plan not to prevent NMI being fired. Instead, if one thread of a core is updating microcode, other threads of this core would stop in the handler of NMI until the update completion. Is this approach acceptable?

Quoted 8 lines

Well, I have to return the question: It is you who knows what is or is not acceptable while an ucode update is in progress. In particular it obviously matters how much ucode is involved in the delivery of an NMI (and in allowing the handler to get to the point where you'd "stop" it).

If the approach you suggest is fine for the NMI case, I'd then wonder if it couldn't also be used for the #MC one.

Architecturally only one #MC can be active in the system. If a new #MC condition happens when MCG_STATUS.MCIP is already set, that would cause spontaneous shutdown.

If another NMI arrives on the CPU doing the wrmsr, it will be pended in the lapic and delivered after the wrmsr returns. wrmsr flow can't be interrupted.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#47 lore raw

Re: [Xen-devel] [PATCH v7 09/10] microcode: remove microcode_update_lock

Quoted 47 lines
Quoted 1 line
Quoted 1 line · On 13.06.19 at 19:47, <[email protected]> wrote:
Quoted 45 lines · Jan Beulich
Quoted 32 lines
Quoted 1 line
Quoted 1 line · On 13.06.19 at 16:05, <[email protected]> wrote:
Quoted 30 lines · Jan Beulich
Quoted 29 lines
Quoted 1 line
Quoted 1 line · On 11.06.19 at 18:04, <[email protected]> wrote:
Quoted 3 lines · Chao Gao
Quoted 2 lines · Jan Beulich

Review static int apply_microcode(const struct microcode_patch

Quoted 2 lines
(quoted diff) +0 -0
@@ -307,8 +303,7 @@ static int apply_microcode(const struct microcode_patch 

*patch)

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
(quoted diff) +1 -2
     mc_intel = patch->mc_intel;
 
-    /* serialize access to the physical write to MSR 0x79 */
-    spin_lock_irqsave(&microcode_update_lock, flags);
+    BUG_ON(local_irq_is_enabled());
 
     /*
      * Writeback and invalidate caches before updating microcode to avoid
Quoted 6 lines
Quoted 4 lines

Thinking about it - what happens if we hit an NMI or #MC here? watchdog_disable(), a call to which you add in an earlier patch, doesn't really suppress the generation of NMIs, it only tells the handler not to look at the accumulated statistics.

I think they should be suppressed. Ashok, could you confirm it?

I think the only sources would be the watchdog as you pointed out which we already touch to keep it from expiring. The perf counters i'm not an expert in, but i'll check. When we are in stop_machine() type flow, its not clear if any of those would fire. (I might be wrong, but let me check).

Quoted 6 lines
Quoted 2 lines

Well, without disarming the watchdog NMI at the LAPIC / IO-APIC, how would it _not_ potentially fire?

We plan not to prevent NMI being fired. Instead, if one thread of a core is updating microcode, other threads of this core would stop in the handler of NMI until the update completion. Is this approach acceptable?

Well, I have to return the question: It is you who knows what is or is not acceptable while an ucode update is in progress. In particular it obviously matters how much ucode is involved in the delivery of an NMI (and in allowing the handler to get to the point where you'd "stop" it).

Quoted 7 lines
Quoted 2 lines

If the approach you suggest is fine for the NMI case, I'd then wonder if it couldn't also be used for the #MC one.

Architecturally only one #MC can be active in the system. If a new #MC condition happens when MCG_STATUS.MCIP is already set, that would cause spontaneous
shutdown.

That's understood.

Quoted 3 lines

If another NMI arrives on the CPU doing the wrmsr, it will be pended in the lapic and delivered after the wrmsr returns. wrmsr flow can't be interrupted.

Of course.

Neither part of your response is an argument against adding the same "defense" to the #MC handler, though. While likely #MC will be fatal to the system anyway, we should try to avoid making things worse when we can.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#48 lore raw

PATCH v7 10/10 x86/microcode: always collect_cpu_info() during boot

In reply to Chao Gao

From: Sergey Dyasli <[email protected]>

Currently cpu_sig struct is not updated during boot when either:

1. ucode_scan is set to false (e.g. no "ucode=scan" in cmdline)
2. initrd does not contain a microcode blob

These will result in cpu_sig.rev being 0 which affects APIC's check_deadline_errata() and retpoline_safe() functions.

Fix this by getting ucode revision early during boot and SMP bring up. While at it.

---
changes in v7:
- rebase on patch 1~9

---
 xen/arch/x86/microcode.c | 4 ++++
 1 file changed, 4 insertions(+)
xen/arch/x86/microcode.c +4 -0
index f4a417e..8aeb152 100644
@@ -590,6 +590,10 @@ int __init early_microcode_init(void)
 
     if ( microcode_ops )
     {
+        rc = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
+        if ( rc )
+            return rc;
+
         if ( ucode_mod.mod_end || ucode_blob.size )
             rc = early_microcode_parse_and_update_cpu();
     }
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#49 lore raw

Re: [Xen-devel] [PATCH v7 10/10] x86/microcode: always collect_cpu_info() during boot

Quoted 4 lines · Chao Gao

From: Sergey Dyasli <[email protected]>

Currently cpu_sig struct is not updated during boot when either:

Quoted 8 lines

1. ucode_scan is set to false (e.g. no "ucode=scan" in cmdline)
2. initrd does not contain a microcode blob

These will result in cpu_sig.rev being 0 which affects APIC's check_deadline_errata() and retpoline_safe() functions.

Fix this by getting ucode revision early during boot and SMP bring up. While at it.

I don't understand the last "While at it" sentence. Can it be removed?

Is this an issue with current code? If so this could be merged ahead of the rest of the series, and should likely be patch 1.

OTOH if the issue this patch is fixing is introduced by this series please merge the fix with the respective patch that introduced the bug.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#50 lore raw

Re: [Xen-devel] [PATCH v7 10/10] x86/microcode: always collect_cpu_info() during boot

Quoted 8 lines · Roger Pau Monné
Quoted 7 lines · Chao Gao

From: Sergey Dyasli <[email protected]>

Currently cpu_sig struct is not updated during boot when either:

1. ucode_scan is set to false (e.g. no "ucode=scan" in cmdline)
2. initrd does not contain a microcode blob

Quoted 8 lines
Quoted 5 lines

These will result in cpu_sig.rev being 0 which affects APIC's check_deadline_errata() and retpoline_safe() functions.

Fix this by getting ucode revision early during boot and SMP bring up. While at it.

I don't understand the last "While at it" sentence. Can it be removed?

Yes.

Quoted 7 lines

Is this an issue with current code? If so this could be merged ahead of the rest of the series, and should likely be patch 1.

OTOH if the issue this patch is fixing is introduced by this series please merge the fix with the respective patch that introduced the bug.

It is the former. Will send it separately.
Really appreciate your other comments.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#51 lore raw

Re: [Xen-devel] [PATCH v7 10/10] x86/microcode: always collect_cpu_info() during boot

In reply to Chao Gao
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

From: Sergey Dyasli <[email protected]>

Currently cpu_sig struct is not updated during boot when either:

1. ucode_scan is set to false (e.g. no "ucode=scan" in cmdline)
2. initrd does not contain a microcode blob

I thought we'd already discussed this - "ucode=<number>" is not covered by this.

Quoted 5 lines

These will result in cpu_sig.rev being 0 which affects APIC's check_deadline_errata() and retpoline_safe() functions.

Fix this by getting ucode revision early during boot and SMP bring up. While at it.

While at it?

Quoted 5 lines

---
changes in v7:
- rebase on patch 1~9

From the looks of it this doesn't depend on any of the earlier changes (except the ucode_cpu_info -> cpu_sig change), and hence could go in right away. Am I overlooking something? If not, all that's needed would be clarifications of the description as per above.

xen/arch/x86/microcode.c int __init early_microcode_init(void) · line 599

Quoted 13 lines
xen/arch/x86/microcode.c +4 -0
@@ -590,6 +590,10 @@ int __init early_microcode_init(void)
 
     if ( microcode_ops )
     {
+        rc = microcode_ops->collect_cpu_info(&this_cpu(cpu_sig));
+        if ( rc )
+            return rc;
+
         if ( ucode_mod.mod_end || ucode_blob.size )
             rc = early_microcode_parse_and_update_cpu();
     }

Do we really need to bail on error here? I don't see anything wrong with simply continuing. The caller doesn't care about the return value anyway, so best effort would seem to be good enough.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#52 lore raw

Re: [Xen-devel] [PATCH v7 10/10] x86/microcode: always collect_cpu_info() during boot

Quoted 25 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 27.05.19 at 10:31, <[email protected]> wrote:

From: Sergey Dyasli <[email protected]>

Currently cpu_sig struct is not updated during boot when either:

1. ucode_scan is set to false (e.g. no "ucode=scan" in cmdline)
2. initrd does not contain a microcode blob

I thought we'd already discussed this - "ucode=<number>" is not covered by this.

Quoted 5 lines

These will result in cpu_sig.rev being 0 which affects APIC's check_deadline_errata() and retpoline_safe() functions.

Fix this by getting ucode revision early during boot and SMP bring up. While at it.

While at it?

Quoted 5 lines

---
changes in v7:
- rebase on patch 1~9

Quoted 4 lines

From the looks of it this doesn't depend on any of the earlier changes (except the ucode_cpu_info -> cpu_sig change), and hence could go in right away. Am I overlooking something? If not, all that's needed would be clarifications of the description as per above.

I think no. Will send this patch separately.

Thanks
Chao

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel