improve late microcode loading
v5 cover improve late microcode loading
Changes in this version:
- support parallel microcode updates for all cores (see patch 8)
- Address Roger's comments on the last version.
The intention of this series is to make the late microcode loading more reliable by rendezvousing all cpus in stop_machine context. This idea comes from Ashok. I am porting his linux patch to Xen (see patch 7 for more details).
This series makes three changes:
1. Patch 1-6: introduce a global microcode cache
2. Patch 7: synchronize late microcode loading
3. Patch 8: support parallel microcodes update on different cores
Currently, late microcode loading does a lot of things including parsing microcode blob, checking the signature/revision and performing update. Putting all of them into stop_machine context is a bad idea because of complexity (One issue I observed is memory allocation triggered one assertion in stop_machine context). In order to simplify the load process, I move parsing microcode out of the load process. The microcode blob is parsed and a global microcode cache is built on a single CPU before rendezvousing all cpus to update microcode. Other CPUs just get and load a suitable microcode from the global cache. With this global cache, it is safe to put simplified load process to stop_machine context.
Regarding changes to AMD side, I didn't do any test for them due to lack of hardware. Could you help to test this series on an AMD machine? At least, two basic tests are needed:
* do a microcode update after system bootup
* don't bring all pCPUs up at bootup by specifying maxcpus option in xen command line and then do a microcode update and online all offlined CPUs via 'xen-hptool'.
Chao Gao (8):
microcode/intel: remove redundent check against ucode size microcode/intel: extend microcode_update_match()
microcode: introduce the global microcode cache
microcode: delete 'mc' field from struct ucode_cpu_info
microcode: split out apply_microcode() from cpu_request_microcode()
microcode: delete microcode pointer and size from microcode_info x86/microcode: Synchronize late microcode loading
microcode: update microcode on cores in parallel
xen/arch/x86/microcode.c | 336 ++++++++++++++++++++++++++-------------- xen/arch/x86/microcode_amd.c | 202 ++++++++++++------------ xen/arch/x86/microcode_intel.c | 139 +++++++++-------- xen/include/asm-x86/microcode.h | 25 ++- 4 files changed, 416 insertions(+), 286 deletions(-)
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 1/8 microcode/intel: remove redundent check against ucode size
This check has been done in microcode_sanity_check(). Needn't do it again in get_matching_microcode().
--- xen/arch/x86/microcode_intel.c | 3 --- 1 file changed, 3 deletions(-)
xen/arch/x86/microcode_intel.c +0 -3
@@ -246,9 +246,6 @@ static int get_matching_microcode(const void *mc, unsigned int cpu) mc_header->sig, mc_header->pf) ) goto find; -if ( total_size <= (get_datasize (mc_header) +MC_HEADER_SIZE ) ) -return 0 ; - ext_header = mc +get_datasize (mc_header) +MC_HEADER_SIZE ; ext_sigcount = ext_header->count; ext_sig = (void *)ext_header +EXT_HEADER_SIZE ;
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 1/8] microcode/intel: remove redundent check against ucode size
Quoted 4 lines · Chao Gao
This check has been done in
microcode_sanity_check(). Needn't do it again inget_matching_microcode().Signed-off-by Chao Gao <[email protected]>
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 1/8] microcode/intel: remove redundent check against ucode size
In reply to Chao GaoQuoted 3 lines
Quoted 1 line
Quoted 1 line
Chao Gao <[email protected]> 01/28/19 8:06 AM >>>
This check has been done in
microcode_sanity_check(). Needn't do it again inget_matching_microcode().
But while there are two call sites of get_matching_microcode() only one is preceded by a call to microcode_sanity_check(). There's also no visible connection between cpu_request_microcode() (which frees the blob it has validated) and microcode_resume_match() (and in particular the blob passed into there). What am I missing?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 1/8] microcode/intel: remove redundent check against ucode size
Quoted 9 lines · Jan Beulich
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Chao Gao <[email protected]> 01/28/19 8:06 AM >>>
This check has been done in
microcode_sanity_check(). Needn't do it again inget_matching_microcode().But while there are two call sites of
get_matching_microcode()only one is preceded by a call tomicrocode_sanity_check(). There's also no visible connection betweencpu_request_microcode()(which frees the blob it has validated) andmicrocode_resume_match()(and in particular the blob passed into there). What am I missing?
microcode_resume_match() takes a cached blob; the blob should have passed the sanity check. I will explain it in the commit message.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 2/8 microcode/intel: extend microcode_update_match()
In reply to Chao Gaoto a more generic function. Then, this function can compare two given microcodes' signature/revision as well. Comparing two microcodes is used to update the global microcode cache (introduced by the later patches in this series) when a new microcode is given.
---
Changes in v5:
- constify the extended_signature
- use named enum type for the return value of microcode_update_match
--- xen/arch/x86/microcode_intel.c | 43 ++++++++++++++++++----------------------- xen/include/asm-x86/microcode.h | 6 ++++++ 2 files changed, 25 insertions(+), 24 deletions(-)
xen/arch/x86/microcode_intel.c +19 -24
@@ -127,14 +127,24 @@ static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig)return 0 ; } -static inline int microcode_update_match ( -unsigned int cpu_num,const struct microcode_header_intel *mc_header, -int sig,int pf) +static enum microcode_match_result microcode_update_match ( +const void *mc,unsigned int sig,unsigned int pf,unsigned int rev) { -struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu_num); +const struct microcode_header_intel *mc_header = mc; +const struct extended_sigtable *ext_header; +const struct extended_signature *ext_sig; +unsigned int i; + +if (sigmatch (sig, mc_header->sig, pf, mc_header->pf) ) +return (mc_header->rev > rev) ?NEW_UCODE :OLD_UCODE ; -return (sigmatch (sig, uci->cpu_sig.sig, pf, uci->cpu_sig.pf) && - (mc_header->rev > uci->cpu_sig.rev)); + ext_header = mc +get_datasize (mc_header) +MC_HEADER_SIZE ; + ext_sig = (const void *)ext_header +EXT_HEADER_SIZE ; +for ( i =0 ; i < ext_header->count; i++ ) +if (sigmatch (sig, ext_sig[i].sig, pf, ext_sig[i].pf) ) +return (mc_header->rev > rev) ?NEW_UCODE :OLD_UCODE ; + +return MIS_UCODE ; }static int microcode_sanity_check (void *mc)
@@ -236,28 +246,13 @@ static int get_matching_microcode(const void *mc, unsigned int cpu) {struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu);const struct microcode_header_intel *mc_header = mc; -const struct extended_sigtable *ext_header;unsigned long total_size =get_totalsize (mc_header); -int ext_sigcount, i; -struct extended_signature *ext_sig;void *new_mc; -if (microcode_update_match (cpu, mc_header, - mc_header->sig, mc_header->pf) ) - goto find; +if (microcode_update_match (mc, uci->cpu_sig.sig, uci->cpu_sig.pf, + uci->cpu_sig.rev) !=NEW_UCODE ) +return 0 ; - ext_header = mc +get_datasize (mc_header) +MC_HEADER_SIZE ; - ext_sigcount = ext_header->count; - ext_sig = (void *)ext_header +EXT_HEADER_SIZE ; -for ( i =0 ; i < ext_sigcount; i++ ) - { -if (microcode_update_match (cpu, mc_header, - ext_sig->sig, ext_sig->pf) ) - goto find; - ext_sig++; - } -return 0 ; - find:pr_debug ("microcode: CPU%d found a matching microcode update with" " version %#x (current=%#x)\n" , cpu, mc_header->rev, uci->cpu_sig.rev);
xen/include/asm-x86/microcode.h +6 -0
@@ -3,6 +3,12 @@#include <xen/percpu.h> +enum microcode_match_result { +OLD_UCODE ,/* signature matched, but revision id isn't newer */ +NEW_UCODE ,/* signature matched, but revision id is newer */ +MIS_UCODE ,/* signature mismatched */ +}; +struct cpu_signature ;struct ucode_cpu_info ;
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 2/8] microcode/intel: extend microcode_update_match()
Quoted 20 lines · Chao Gao
to a more generic function. Then, this function can compare two given microcodes' signature/revision as well. Comparing two microcodes is used to update the global microcode cache (introduced by the later patches in this series) when a new microcode is given.
Signed-off-by Chao Gao <[email protected]>---
Changes in v5:
- constify theextended_signature
- use named enum type for the return value ofmicrocode_update_match--- xen/arch/x86/microcode_intel.c | 43 ++++++++++++++++++----------------------- xen/include/asm-x86/microcode.h | 6 ++++++ 2 files changed, 25 insertions(+), 24 deletions(-)xen/arch/x86/microcode_intel.c +0
-0
xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 131
Quoted 8 lines
xen/arch/x86/microcode_intel.c +2
-3@@ -127,14 +127,24 @@ static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig)return 0 ; } -static inline int microcode_update_match ( -unsigned int cpu_num,const struct microcode_header_intel *mc_header, -int sig,int pf) +static enum microcode_match_result microcode_update_match ( +const void *mc,unsigned int sig,unsigned int pf,unsigned int rev)
Why are you passing this as a void pointer? The only caller is already passing this as a mc_header pointer.
xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 139
Quoted 9 lines
xen/arch/x86/microcode_intel.c +7
-1{ -struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu_num); +const struct microcode_header_intel *mc_header = mc; +const struct extended_sigtable *ext_header; +const struct extended_signature *ext_sig; +unsigned int i; + +if (sigmatch (sig, mc_header->sig, pf, mc_header->pf) ) +return (mc_header->rev > rev) ?NEW_UCODE :OLD_UCODE ;
The code above implies that a microcode blob can only have a single version of microcode for each model, I guess this is OK and guaranteed by Intel?
With at least the first comment fixed:
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 2/8] microcode/intel: extend microcode_update_match()
Quoted 45 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 17:55, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num,
Quoted 20 lines · Chao Gao
to a more generic function. Then, this function can compare two given microcodes' signature/revision as well. Comparing two microcodes is used to update the global microcode cache (introduced by the later patches in this series) when a new microcode is given.
Signed-off-by Chao Gao <[email protected]>---
Changes in v5:
- constify theextended_signature
- use named enum type for the return value ofmicrocode_update_match--- xen/arch/x86/microcode_intel.c | 43 ++++++++++++++++++----------------------- xen/include/asm-x86/microcode.h | 6 ++++++ 2 files changed, 25 insertions(+), 24 deletions(-)xen/arch/x86/microcode_intel.c +0
-0@@ -127,14 +127,24 @@ static int collect_cpu_info(unsigned int cpu_num,struct
cpu_signature*csig)xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 131
Quoted 8 lines
xen/arch/x86/microcode_intel.c +2
-3return 0 ; } -static inline int microcode_update_match ( -unsigned int cpu_num,const struct microcode_header_intel *mc_header, -int sig,int pf) +static enum microcode_match_result microcode_update_match ( +const void *mc,unsigned int sig,unsigned int pf,unsigned int rev)Why are you passing this as a void pointer? The only caller is already passing this as a
mc_headerpointer.xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 139
Quoted 9 lines
xen/arch/x86/microcode_intel.c +7
-1{ -struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu_num); +const struct microcode_header_intel *mc_header = mc; +const struct extended_sigtable *ext_header; +const struct extended_signature *ext_sig; +unsigned int i; + +if (sigmatch (sig, mc_header->sig, pf, mc_header->pf) ) +return (mc_header->rev > rev) ?NEW_UCODE :OLD_UCODE ;
Quoted 3 lines
The code above implies that a microcode blob can only have a single version of microcode for each model, I guess this is OK and guaranteed by Intel?
The containerization is a little different from AMD's. Multiple different versions can be present in what commonly named microcode.bin, but each individual blob holds exactly one version for one family:model:stepping tuple (plus the further restriction, named "pf" above).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 2/8] microcode/intel: extend microcode_update_match()
In reply to Chao GaoQuoted 11 lines
Quoted 1 line
Quoted 1 line
Chao Gao <[email protected]> 01/28/19 8:10 AM >>>
xen/arch/x86/microcode_intel.c +1
-3@@ -127,14 +127,24 @@ static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig)return 0 ; } -static inline int microcode_update_match ( -unsigned int cpu_num,const struct microcode_header_intel *mc_header, -int sig,int pf) +static enum microcode_match_result microcode_update_match(
xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 139
Quoted 10 lines
xen/arch/x86/microcode_intel.c +8
-1+ const void *mc, unsigned int sig, unsigned int pf, unsigned int rev) { -struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu_num); + const struct microcode_header_intel *mc_header = mc; +const struct extended_sigtable *ext_header; +const struct extended_signature *ext_sig; +unsigned int i; + +if (sigmatch (sig, mc_header->sig, pf, mc_header->pf) ) +return (mc_header->rev > rev) ?NEW_UCODE :OLD_UCODE ;
You may want a tristate return here: I know there are systems where firmware updates ucode only on core 0 of every socket, in which case we'd very much like to apply the same microcode on the other cores in case we find the blob matching what is currently installed. IOW depending how later patches actually work, you may also want a SAME_UCODE return case.
xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 141
Quoted 1 line
xen/arch/x86/microcode_intel.c +1
-0+ ext_header = mc +get_datasize (mc_header) +MC_HEADER_SIZE ;
On top of what Roger has said, isn't mc + MC_HEADER_SIZE the same as mc_header + 1?
xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 142
Quoted 1 line
xen/arch/x86/microcode_intel.c +1
-0+ ext_sig = (const void *)ext_header +EXT_HEADER_SIZE ;
And (const void *)ext_header + EXT_HEADER_SIZE the same as (const void *)(ext_header + 1)?
In both cases this would eliminate unnecessary implications of certain two sub-terms to refer to the same types, i.e. also make the casts less scary / dangerous.
xen/include/asm-x86/microcode.h line 10
Quoted 11 lines
xen/include/asm-x86/microcode.h +5
-0@@ -3,6 +3,12 @@#include <xen/percpu.h> +enum microcode_match_result { +OLD_UCODE ,/* signature matched, but revision id isn't newer */ +NEW_UCODE ,/* signature matched, but revision id is newer */ +MIS_UCODE ,/* signature mismatched */ +};
It's not clear at this point of the series or from the commit message whether this is to be used by AMD code as well. If not, it would better move into microcode_intel.c.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 2/8] microcode/intel: extend microcode_update_match()
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line
Chao Gao <[email protected]> 01/28/19 8:10 AM >>>
xen/arch/x86/microcode_intel.c +7
-4@@ -127,14 +127,24 @@ static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig)return 0 ; } -static inline int microcode_update_match ( -unsigned int cpu_num,const struct microcode_header_intel *mc_header, -int sig,int pf) +static enum microcode_match_result microcode_update_match ( +const void *mc,unsigned int sig,unsigned int pf,unsigned int rev) { -struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu_num); +const struct microcode_header_intel *mc_header = mc; +const struct extended_sigtable *ext_header; +const struct extended_signature *ext_sig; +unsigned int i; +
Quoted 8 lines
xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 139
Quoted 2 lines
xen/arch/x86/microcode_intel.c +2
-0+if (sigmatch (sig, mc_header->sig, pf, mc_header->pf) ) +return (mc_header->rev > rev) ?NEW_UCODE :OLD_UCODE ;You may want a tristate return here: I know there are systems where firmware updates ucode only on core 0 of every socket, in which case we'd very much like to apply the same microcode on the other cores in case we find the blob matching what is currently installed. IOW depending how later patches actually work, you may also want a
SAME_UCODEreturn case.
It seems we don't need it. For CPU hot-plug, we would also meet the same issue you described. To resolve it, a ucode patch is added to the global cache without checking against the revision on current CPU even if the signature is matched.
Quoted 6 lines
xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 141
Quoted 2 lines
xen/arch/x86/microcode_intel.c +1
-0+ ext_header = mc +get_datasize (mc_header) +MC_HEADER_SIZE ;On top of what Roger has said, isn't mc +
MC_HEADER_SIZEthe same asmc_header+ 1?
Quoted 9 lines
xen/arch/x86/microcode_intel.c static int collect_cpu_info(unsigned int cpu_num, struct cpu_signature *csig) · line 142
Quoted 1 line
xen/arch/x86/microcode_intel.c +1
-0+ ext_sig = (const void *)ext_header +EXT_HEADER_SIZE ;And (const void *)ext_header +
EXT_HEADER_SIZEthe same as (const void *)(ext_header + 1)?In both cases this would eliminate unnecessary implications of certain two sub-terms to refer to the same types, i.e. also make the casts less scary / dangerous.
Will do.
Quoted 7 lines
Quoted 7 lines
xen/include/asm-x86/microcode.h +0
-0@@ -3,6 +3,12 @@#include <xen/percpu.h>
Quoted 9 lines
xen/include/asm-x86/microcode.h line 10
Quoted 5 lines
xen/include/asm-x86/microcode.h +5
-0+enum microcode_match_result { +OLD_UCODE ,/* signature matched, but revision id isn't newer */ +NEW_UCODE ,/* signature matched, but revision id is newer */ +MIS_UCODE ,/* signature mismatched */ +};It's not clear at this point of the series or from the commit message whether this is to be used by AMD code as well. If not, it would better move into
microcode_intel.c.
It will be used by AMD code. Will mention this in commit message.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 3/8 microcode: introduce the global microcode cache
In reply to Chao Gaoto replace the current per-cpu cache 'uci->mc'.
Compared to the current per-cpu cache, the benefits of the global microcode cache are:
1. It reduces the work that need to be done on each CPU. Parsing ucode file can be done once on one CPU. Other CPUs needn't parse ucode file. Instead, they can find out and load a patch with newer revision from the global cache.
2. It reduces the memory consumption on a system with many CPU cores.
Two functions, save_patch() and find_patch() are introduced. The former adds one given patch to the global cache. The latter gets a newer and matched ucode patch from the global cache.
Note that I deliberately avoid touching 'uci->mc' as I am going to remove it completely in the next patch.
---
Changes in v5:
- reword the commit description
- find_patch() and save_patch() are abstracted into common functions with some hooks for AMD and Intel
--- xen/arch/x86/microcode.c | 54 +++++++++++++++++++++++ xen/arch/x86/microcode_amd.c | 94 ++++++++++++++++++++++++++++++++++++++--- xen/arch/x86/microcode_intel.c | 71 ++++++++++++++++++++++++++++--- xen/include/asm-x86/microcode.h | 13 ++++++ 4 files changed, 219 insertions(+), 13 deletions(-)
xen/arch/x86/microcode.c +54 -0
@@ -61,6 +61,8 @@ static struct ucode_mod_blob __initdata ucode_blob;*/ static bool_t __initdata ucode_scan; +static LIST_HEAD (microcode_cache); +void __init microcode_set_module (unsigned int idx) { ucode_mod_idx = idx;
@@ -208,6 +210,58 @@ static void microcode_fini_cpu(unsigned int cpu)spin_unlock (µcode_mutex); } +/* Save a ucode patch to the global cache list */ +bool save_patch (struct microcode_patch *new_patch) +{ +struct microcode_patch *microcode_patch; + +list_for_each_entry (microcode_patch, µcode_cache, list) + { +enum microcode_match_result result = + microcode_ops->replace_patch (new_patch, microcode_patch); + +switch ( result ) + { +case OLD_UCODE : + microcode_ops->free_patch (new_patch); +return false; + +case NEW_UCODE : + microcode_ops->free_patch (microcode_patch); +return true; + +case MIS_UCODE : +continue ; + +default : +ASSERT_UNREACHABLE (); +return 0 ; + } + } +list_add_tail (&new_patch->list, µcode_cache); +return true; +} + +/* Find a ucode patch who has newer revision than the one in use */ +struct microcode_patch *find_patch (unsigned int cpu) +{ +struct microcode_patch *microcode_patch; +struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); + +if ( microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig) ) + { +__microcode_fini_cpu (cpu); +return NULL ; + } + +list_for_each_entry (microcode_patch, µcode_cache, list) + { +if ( microcode_ops->match_cpu (microcode_patch, cpu) ) +return microcode_patch; + } +return NULL ; +} +int microcode_resume_cpu (unsigned int cpu) {int err;
xen/arch/x86/microcode_amd.c +88 -6
125 lines not on this page · raw message
xen/arch/x86/microcode_intel.c +64 -7
125 lines not on this page · raw message
xen/include/asm-x86/microcode.h +13 -0
@@ -1,6 +1,7 @@#ifndef ASM_X86__MICROCODE_H #define ASM_X86__MICROCODE_H +#include <xen/list.h> #include <xen/percpu.h> enum microcode_match_result {
@@ -12,6 +13,11 @@ enum microcode_match_result {struct cpu_signature ;struct ucode_cpu_info ; +struct microcode_patch { +struct list_head list; +void *data; +}; +struct microcode_ops {int (*microcode_resume_match)(unsigned int cpu,const void *mc); int (*cpu_request_microcode)(unsigned int cpu, const void *buf,
@@ -19,6 +25,10 @@ struct microcode_ops {int (*collect_cpu_info)(unsigned int cpu,struct cpu_signature *csig);int (*apply_microcode)(unsigned int cpu);int (*start_update)(void ); +enum microcode_match_result (*replace_patch)(struct microcode_patch *new, +struct microcode_patch *old); +void (*free_patch)(struct microcode_patch *patch); +bool (*match_cpu)(const struct microcode_patch *patch,unsigned int cpu); };struct cpu_signature {
@@ -39,4 +49,7 @@ struct ucode_cpu_info { DECLARE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info);extern const struct microcode_ops *microcode_ops; +bool save_patch (struct microcode_patch *new_patch); +struct microcode_patch *find_patch (unsigned int cpu); +#endif /* ASM_X86__MICROCODE_H */
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 3/8] microcode: introduce the global microcode cache
Quoted 44 lines · Chao Gao
to replace the current per-cpu cache '
uci->mc'.Compared to the current per-cpu cache, the benefits of the global microcode cache are:
1. It reduces the work that need to be done on each CPU. Parsing ucode file can be done once on one CPU. Other CPUs needn't parse ucode file. Instead, they can find out and load a patch with newer revision from the global cache.
2. It reduces the memory consumption on a system with many CPU cores.Two functions,
save_patch()andfind_patch()are introduced. The former adds one given patch to the global cache. The latter gets a newer and matched ucode patch from the global cache.Note that I deliberately avoid touching '
uci->mc' as I am going to remove it completely in the next patch.Signed-off-by Chao Gao <[email protected]>---
Changes in v5:
- reword the commit description
-find_patch()andsave_patch()are abstracted into common functions with some hooks for AMD and Intel--- xen/arch/x86/microcode.c | 54 +++++++++++++++++++++++ xen/arch/x86/microcode_amd.c | 94 ++++++++++++++++++++++++++++++++++++++--- xen/arch/x86/microcode_intel.c | 71 ++++++++++++++++++++++++++++--- xen/include/asm-x86/microcode.h | 13 ++++++ 4 files changed, 219 insertions(+), 13 deletions(-)xen/arch/x86/microcode.c +2
-0@@ -61,6 +61,8 @@ static struct ucode_mod_blob __initdata ucode_blob;*/ static bool_t __initdata ucode_scan; +static LIST_HEAD (microcode_cache); +void __init microcode_set_module (unsigned int idx) { ucode_mod_idx = idx;
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 214
Quoted 5 lines
xen/arch/x86/microcode.c +2
-0@@ -208,6 +210,58 @@ static void microcode_fini_cpu(unsigned int cpu)spin_unlock (µcode_mutex); } +/* Save a ucode patch to the global cache list */ +bool save_patch (struct microcode_patch *new_patch)
This being a global function likely requires some kind of prefix, I would suggest microcode_save_patch, the same applies to the find_patch function below.
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 218
Quoted 4 lines
xen/arch/x86/microcode.c +4
-0+{ +struct microcode_patch *microcode_patch; + +list_for_each_entry (microcode_patch, µcode_cache, list)
I think I'm missing something here, but given the conversation we had in the previous version of the series [0] I assumed there was only a single microcode patch that applies to the whole system, and that there was no need to keep a list?
Because Xen doesn't support running on such mixed systems anyway?
[0] https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg00381.html
Quoted 43 lines
xen/arch/x86/microcode_intel.c +10
-4@@ -276,18 +324,24 @@ static int apply_microcode(unsigned int cpu)unsigned int val[2 ];unsigned int cpu_num =raw_smp_processor_id ();struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu_num); +struct microcode_intel *mc_intel; +struct microcode_patch *patch;/* We should bind the task to the CPU */ BUG_ON (cpu_num != cpu); -if ( uci->mc.mc_intel ==NULL ) + patch =find_patch (cpu); +if ( !patch )return -EINVAL ; + mc_intel = patch->data; +BUG_ON (!mc_intel); +/* serialize access to the physical write to MSR 0x79 */ spin_lock_irqsave (µcode_update_lock, flags);/* write microcode via MSR 0x79 */ -wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )uci->mc.mc_intel->bits); +wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )mc_intel->bits);wrmsrl (MSR_IA32_UCODE_REV ,0x0ULL );/* As documented in the SDM: Do a CPUID 1 here */ @@ -298,19 +352,19 @@ static int apply_microcode(unsigned int cpu) val[1 ] = (uint32_t )(msr_content >>32 );spin_unlock_irqrestore (µcode_update_lock, flags); -if ( val[1 ] != uci->mc.mc_intel->hdr.rev ) +if ( val[1 ] != mc_intel->hdr.rev ) {printk (KERN_ERR "microcode: CPU%d update from revision " "%#x to %#x failed. Resulting revision is %#x.\n" , cpu_num, - uci->cpu_sig.rev, uci->mc.mc_intel->hdr.rev, val[1 ]); + uci->cpu_sig.rev, mc_intel->hdr.rev, val[1 ]);return -EIO ;
xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 367
Quoted 10 lines
xen/arch/x86/microcode_intel.c +3
-3}printk (KERN_INFO "microcode: CPU%d updated from revision " "%#x to %#x, date = %04x-%02x-%02x \n" , cpu_num, uci->cpu_sig.rev, val[1 ], - uci->mc.mc_intel->hdr.date &0xffff , - uci->mc.mc_intel->hdr.date >>24 , - (uci->mc.mc_intel->hdr.date >>16 ) &0xff ); + mc_intel->hdr.date &0xffff , + mc_intel->hdr.date >>24 , + (mc_intel->hdr.date >>16 ) &0xff );
Nit: while here could you make an union of the date field with it's format, so that you can print it without having to perform this shifting and masking?
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 3/8] microcode: introduce the global microcode cache
Quoted 48 lines · Roger Pau Monné
Quoted 47 lines · Chao Gao
to replace the current per-cpu cache '
uci->mc'.Compared to the current per-cpu cache, the benefits of the global microcode cache are:
1. It reduces the work that need to be done on each CPU. Parsing ucode file can be done once on one CPU. Other CPUs needn't parse ucode file. Instead, they can find out and load a patch with newer revision from the global cache.
2. It reduces the memory consumption on a system with many CPU cores.Two functions,
save_patch()andfind_patch()are introduced. The former adds one given patch to the global cache. The latter gets a newer and matched ucode patch from the global cache.Note that I deliberately avoid touching '
uci->mc' as I am going to remove it completely in the next patch.Signed-off-by Chao Gao <[email protected]>---
Changes in v5:
- reword the commit description
-find_patch()andsave_patch()are abstracted into common functions with some hooks for AMD and Intel--- xen/arch/x86/microcode.c | 54 +++++++++++++++++++++++ xen/arch/x86/microcode_amd.c | 94 ++++++++++++++++++++++++++++++++++++++--- xen/arch/x86/microcode_intel.c | 71 ++++++++++++++++++++++++++++--- xen/include/asm-x86/microcode.h | 13 ++++++ 4 files changed, 219 insertions(+), 13 deletions(-)xen/arch/x86/microcode.c +2
-0@@ -61,6 +61,8 @@ static struct ucode_mod_blob __initdata ucode_blob;*/ static bool_t __initdata ucode_scan; +static LIST_HEAD (microcode_cache); +void __init microcode_set_module (unsigned int idx) { ucode_mod_idx = idx;@@ -208,6 +210,58 @@ static void microcode_fini_cpu(unsigned int cpu)spin_unlock (µcode_mutex); }
Quoted 6 lines
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 214
Quoted 2 lines
xen/arch/x86/microcode.c +2
-0+/* Save a ucode patch to the global cache list */ +bool save_patch (struct microcode_patch *new_patch)This being a global function likely requires some kind of prefix, I would suggest
microcode_save_patch, the same applies to thefind_patchfunction below.
Will do.
Quoted 11 lines
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 218
Quoted 5 lines
xen/arch/x86/microcode.c +4
-0+{ +struct microcode_patch *microcode_patch; + +list_for_each_entry (microcode_patch, µcode_cache, list)I think I'm missing something here, but given the conversation we had in the previous version of the series [0] I assumed there was only a single microcode patch that applies to the whole system, and that there was no need to keep a list?
Quoted 1 line
Because Xen doesn't support running on such mixed systems anyway?
No. As Jan pointed out in [1], we still want to support mixed systems which are allowed by Intel and AMD.
[1] https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg03479.html
Quoted 52 lines
[0] https://lists.xenproject.org/archives/html/xen-devel/2018-12/msg00381.html
Quoted 49 lines
xen/arch/x86/microcode_intel.c +10
-6@@ -276,18 +324,24 @@ static int apply_microcode(unsigned int cpu)unsigned int val[2 ];unsigned int cpu_num =raw_smp_processor_id ();struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu_num); +struct microcode_intel *mc_intel; +struct microcode_patch *patch;/* We should bind the task to the CPU */ BUG_ON (cpu_num != cpu); -if ( uci->mc.mc_intel ==NULL ) + patch =find_patch (cpu); +if ( !patch )return -EINVAL ; + mc_intel = patch->data; +BUG_ON (!mc_intel); +/* serialize access to the physical write to MSR 0x79 */ spin_lock_irqsave (µcode_update_lock, flags);/* write microcode via MSR 0x79 */ -wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )uci->mc.mc_intel->bits); +wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )mc_intel->bits);wrmsrl (MSR_IA32_UCODE_REV ,0x0ULL );/* As documented in the SDM: Do a CPUID 1 here */ @@ -298,19 +352,19 @@ static int apply_microcode(unsigned int cpu) val[1 ] = (uint32_t )(msr_content >>32 );spin_unlock_irqrestore (µcode_update_lock, flags); -if ( val[1 ] != uci->mc.mc_intel->hdr.rev ) +if ( val[1 ] != mc_intel->hdr.rev ) {printk (KERN_ERR "microcode: CPU%d update from revision " "%#x to %#x failed. Resulting revision is %#x.\n" , cpu_num, - uci->cpu_sig.rev, uci->mc.mc_intel->hdr.rev, val[1 ]); + uci->cpu_sig.rev, mc_intel->hdr.rev, val[1 ]);return -EIO ; } printk(KERN_INFO "microcode: CPU%d updated from revision " "%#x to %#x, date = %04x-%02x-%02x \n", cpu_num, uci->cpu_sig.rev, val[1], - uci->mc.mc_intel->hdr.date & 0xffff, - uci->mc.mc_intel->hdr.date >> 24,
Quoted 8 lines
xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 367
Quoted 4 lines
xen/arch/x86/microcode_intel.c +3
-1- (uci->mc.mc_intel->hdr.date >> 16) & 0xff); + mc_intel->hdr.date &0xffff , + mc_intel->hdr.date >>24 , + (mc_intel->hdr.date >> 16) & 0xff);Nit: while here could you make an union of the date field with it's format, so that you can print it without having to perform this shifting and masking?
Will do.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Roger Pau Monné
Reviewed-byRe: [PATCH v5 3/8] microcode: introduce the global microcode cache
Quoted 69 lines · Chao Gao
Quoted 48 lines · Roger Pau Monné
Quoted 47 lines · Chao Gao
to replace the current per-cpu cache '
uci->mc'.Compared to the current per-cpu cache, the benefits of the global microcode cache are:
1. It reduces the work that need to be done on each CPU. Parsing ucode file can be done once on one CPU. Other CPUs needn't parse ucode file. Instead, they can find out and load a patch with newer revision from the global cache.
2. It reduces the memory consumption on a system with many CPU cores.Two functions,
save_patch()andfind_patch()are introduced. The former adds one given patch to the global cache. The latter gets a newer and matched ucode patch from the global cache.Note that I deliberately avoid touching '
uci->mc' as I am going to remove it completely in the next patch.Signed-off-by Chao Gao <[email protected]>---
Changes in v5:
- reword the commit description
-find_patch()andsave_patch()are abstracted into common functions with some hooks for AMD and Intel--- xen/arch/x86/microcode.c | 54 +++++++++++++++++++++++ xen/arch/x86/microcode_amd.c | 94 ++++++++++++++++++++++++++++++++++++++--- xen/arch/x86/microcode_intel.c | 71 ++++++++++++++++++++++++++++--- xen/include/asm-x86/microcode.h | 13 ++++++ 4 files changed, 219 insertions(+), 13 deletions(-)xen/arch/x86/microcode.c +2
-0@@ -61,6 +61,8 @@ static struct ucode_mod_blob __initdata ucode_blob;*/ static bool_t __initdata ucode_scan; +static LIST_HEAD (microcode_cache); +void __init microcode_set_module (unsigned int idx) { ucode_mod_idx = idx;@@ -208,6 +210,58 @@ static void microcode_fini_cpu(unsigned int cpu)spin_unlock (µcode_mutex); }Quoted 6 lines
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 214
Quoted 2 lines
xen/arch/x86/microcode.c +2
-0+/* Save a ucode patch to the global cache list */ +bool save_patch (struct microcode_patch *new_patch)This being a global function likely requires some kind of prefix, I would suggest
microcode_save_patch, the same applies to thefind_patchfunction below.Will do.
Quoted 11 lines
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 218
Quoted 5 lines
xen/arch/x86/microcode.c +4
-0+{ +struct microcode_patch *microcode_patch; + +list_for_each_entry (microcode_patch, µcode_cache, list)I think I'm missing something here, but given the conversation we had in the previous version of the series [0] I assumed there was only a single microcode patch that applies to the whole system, and that there was no need to keep a list?
Quoted 6 lines
Quoted 1 line
Because Xen doesn't support running on such mixed systems anyway?
No. As Jan pointed out in [1], we still want to support mixed systems which are allowed by Intel and AMD.
[1] https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg03479.html
Oh thanks, I've missed that reply while reading the previous series. Then with the comments fixed you can add:
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 3/8] microcode: introduce the global microcode cache
In reply to Chao GaoQuoted 21 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode.c +16
-0@@ -208,6 +210,58 @@ static void microcode_fini_cpu(unsigned int cpu)spin_unlock (µcode_mutex); } +/* Save a ucode patch to the global cache list */ +bool save_patch (struct microcode_patch *new_patch) +{ +struct microcode_patch *microcode_patch; + +list_for_each_entry (microcode_patch, µcode_cache, list) + { +enum microcode_match_result result = + microcode_ops->replace_patch (new_patch, microcode_patch); + +switch ( result ) + { +case OLD_UCODE : + microcode_ops->free_patch (new_patch); +return false; +
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 238
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+case NEW_UCODE : + microcode_ops->free_patch (microcode_patch); +return true; + +case MIS_UCODE : +continue ; + +default : +ASSERT_UNREACHABLE (); +return 0 ;
false (or true; either value is going to be fine/wrong here afaict)
Anyway I'm having some difficulty seeing what the intended meaning of the return value is, and without that being clear I also can't make up my mind whether I agree with the cases here.
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 241
Quoted 3 lines
xen/arch/x86/microcode.c +3
-0+ } + } +list_add_tail (&new_patch->list, µcode_cache);
Hmm, you add _every_ patch producing MIS_UCODE to the list. This is going to be a long list then - there are over 100 blobs in the latest dropping, and this number is only going to grow. Saving blobs is useful only for cases where mixed steppings are actually supported. I guess that wouldn't go much beyond the stepping and/or "pf" differing, but family and model matching up.
Additionally list management generally requires locking. This function doesn't even have a comment saying what lock(s) is (are) necessary to be held (same elsewhere).
Finally please add blank lines ahead of the line above as well as (not just here) ahead of the main return statement of the function.
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 246
Quoted 2 lines
xen/arch/x86/microcode.c +2
-0+/* Find a ucode patch who has newer revision than the one in use */ +struct microcode_patch *find_patch (unsigned int cpu)
Is the caller allowed to alter the returned object? If not, you want to return a pointer to const.
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 253
Quoted 7 lines
xen/arch/x86/microcode.c +7
-0+{ +struct microcode_patch *microcode_patch; +struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); + +if ( microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig) ) + { +__microcode_fini_cpu (cpu);
This is kind of a library function - I can't see how it could legitimately invoke "fini", the more that you do here but not ...
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 262
Quoted 9 lines
xen/arch/x86/microcode.c +9
-0+return NULL ; + } + +list_for_each_entry (microcode_patch, µcode_cache, list) + { +if ( microcode_ops->match_cpu (microcode_patch, cpu) ) +return microcode_patch; + } +return NULL ;
... here.
xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 245
Quoted 7 lines
xen/arch/x86/microcode_amd.c +7
-0+static enum microcode_match_result replace_patch (struct microcode_patch *new, +struct microcode_patch *old) +{ +struct microcode_amd *new_mc = new->data; +struct microcode_header_amd *new_header = new_mc->mpb; +struct microcode_amd *old_mc = old->data; +struct microcode_header_amd *old_header = old_mc->mpb;
const (all four of them afaict)
xen/arch/x86/microcode_intel.c static enum microcode_match_result microcode_update_match( · line 152
Quoted 9 lines
xen/arch/x86/microcode_intel.c +3
-0@@ -147,6 +147,15 @@ static enum microcode_match_result microcode_update_match(return MIS_UCODE ; } +static bool match_cpu (const struct microcode_patch *patch,unsigned int cpu) +{ +struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu);
const
xen/arch/x86/microcode_intel.c static int microcode_sanity_check(void *mc) · line 258
Quoted 4 lines
xen/arch/x86/microcode_intel.c +4
-0+static enum microcode_match_result replace_patch (struct microcode_patch *new, +struct microcode_patch *old) +{ +struct microcode_header_intel *old_header = old->data;
const
xen/arch/x86/microcode_intel.c static int microcode_sanity_check(void *mc) · line 264
Quoted 6 lines
xen/arch/x86/microcode_intel.c +6
-0+enum microcode_match_result ret = +microcode_update_match (new->data, old_header->sig, + old_header->pf, old_header->rev); + +if ( ret ==NEW_UCODE ) +list_replace (&old->list, &new->list);
I think it would be better to leave actual list maintenance to generic code. That way the function parameters can also be pointer-to-const.
xen/arch/x86/microcode_intel.c static int get_matching_microcode(const void *mc, unsigned int cpu) · line 281
Quoted 6 lines
xen/arch/x86/microcode_intel.c +2
-0@@ -248,6 +277,25 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)const struct microcode_header_intel *mc_header = mc;unsigned long total_size =get_totalsize (mc_header);void *new_mc; + struct microcode_patch *microcode_patch = xmalloc(struct microcode_patch); +void *new_mc2 =xmalloc_bytes (total_size);
Why don't you use the already existing new_mc here?
Quoted 7 lines
xen/arch/x86/microcode_intel.c +2
-0@@ -276,18 +324,24 @@ static int apply_microcode(unsigned int cpu)unsigned int val[2 ];unsigned int cpu_num =raw_smp_processor_id ();struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu_num); +struct microcode_intel *mc_intel; +struct microcode_patch *patch;
xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 338
Quoted 10 lines
xen/arch/x86/microcode_intel.c +4
-1/* We should bind the task to the CPU */ BUG_ON (cpu_num != cpu); -if ( uci->mc.mc_intel ==NULL ) + patch =find_patch (cpu); +if ( !patch )return -EINVAL ; + mc_intel = patch->data; +BUG_ON (!mc_intel);
I'm not convinced this is a useful check - you never save_patch() anything that has a NULL pointer here. And general corruption might put bad values other than NULL into the field.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 3/8] microcode: introduce the global microcode cache
Quoted 34 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode.c +16
-0@@ -208,6 +210,58 @@ static void microcode_fini_cpu(unsigned int cpu)spin_unlock (µcode_mutex); } +/* Save a ucode patch to the global cache list */ +bool save_patch (struct microcode_patch *new_patch) +{ +struct microcode_patch *microcode_patch; + +list_for_each_entry (microcode_patch, µcode_cache, list) + { +enum microcode_match_result result = + microcode_ops->replace_patch (new_patch, microcode_patch); + +switch ( result ) + { +case OLD_UCODE : + microcode_ops->free_patch (new_patch); +return false; +xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 238
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+case NEW_UCODE : + microcode_ops->free_patch (microcode_patch); +return true; + +case MIS_UCODE : +continue ; + +default : +ASSERT_UNREACHABLE (); +return 0 ;false (or true; either value is going to be fine/wrong here afaict)
Quoted 4 lines
Anyway I'm having some difficulty seeing what the intended meaning of the return value is, and without that being clear I also can't make up my mind whether I agree with the cases here.
The return value means whether saving a ucode patch succeeded or failed. In another word, it also means whether the ucode cache is updated or not. According to the return value, the caller decides not to do the expensive late ucode update for some cases (i.e. when admin provides a old ucode).
Quoted 11 lines
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 241
Quoted 4 lines
xen/arch/x86/microcode.c +3
-0+ } + } +list_add_tail (&new_patch->list, µcode_cache);Hmm, you add _every_ patch producing
MIS_UCODEto the list. This is going to be a long list then - there are over 100 blobs in the latest dropping, and this number is only going to grow. Saving blobs is useful only for cases where mixed steppings are actually supported. I guess that wouldn't go much beyond the stepping and/or "pf" differing, but family and model matching up.
We can introduce another type (i.e. MIX_UCODE) to denote the ucode patch which mismatches any saved patches but only differs from current cpu in stepping or "pf" (or model for AMD).
And save ucodes of MIX_UCODE type and discard ucodes of MIS_UCODE type.
Arch specific callback can determine the type (MIX_UCODE or MIS_UCODE) according to the supported mixes.
Quoted 8 lines
Additionally list management generally requires locking. This function doesn't even have a comment saying what lock(s) is (are) necessary to be held (same elsewhere).
Finally please add blank lines ahead of the line above as well as (not just here) ahead of the main return statement of the function.
Will do
Quoted 6 lines
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 246
Quoted 3 lines
xen/arch/x86/microcode.c +2
-0+/* Find a ucode patch who has newer revision than the one in use */ +struct microcode_patch *find_patch (unsigned int cpu)Is the caller allowed to alter the returned object? If not, you want to return a pointer to const.
No. Will constify the return value.
Quoted 22 lines
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 253
Quoted 8 lines
xen/arch/x86/microcode.c +7
-0+{ +struct microcode_patch *microcode_patch; +struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); + +if ( microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig) ) + { +__microcode_fini_cpu (cpu);This is kind of a library function - I can't see how it could legitimately invoke "fini", the more that you do here but not ...
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 262
Quoted 9 lines
xen/arch/x86/microcode.c +9
-0+return NULL ; + } + +list_for_each_entry (microcode_patch, µcode_cache, list) + { +if ( microcode_ops->match_cpu (microcode_patch, cpu) ) +return microcode_patch; + } +return NULL ;
Quoted 1 line
... here.
My understanding is saving the cpu signature and ucode revision can reduce MSR reads to get such information. So "fini" is invoked when "collect" failed. The apply_microcode() following find_patch() can access "uci" without invoking "collect" again.
Quoted 37 lines
xen/arch/x86/microcode_amd.c static bool_t microcode_fits(const struct microcode_amd *mc_amd, · line 245
Quoted 8 lines
xen/arch/x86/microcode_amd.c +7
-0+static enum microcode_match_result replace_patch (struct microcode_patch *new, +struct microcode_patch *old) +{ +struct microcode_amd *new_mc = new->data; +struct microcode_header_amd *new_header = new_mc->mpb; +struct microcode_amd *old_mc = old->data; +struct microcode_header_amd *old_header = old_mc->mpb;const (all four of them afaict)
xen/arch/x86/microcode_intel.c static enum microcode_match_result microcode_update_match( · line 152
Quoted 9 lines
xen/arch/x86/microcode_intel.c +3
-0@@ -147,6 +147,15 @@ static enum microcode_match_result microcode_update_match(return MIS_UCODE ; } +static bool match_cpu (const struct microcode_patch *patch,unsigned int cpu) +{ +struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu);const
xen/arch/x86/microcode_intel.c static int microcode_sanity_check(void *mc) · line 258
Quoted 4 lines
xen/arch/x86/microcode_intel.c +4
-0+static enum microcode_match_result replace_patch (struct microcode_patch *new, +struct microcode_patch *old) +{ +struct microcode_header_intel *old_header = old->data;const
xen/arch/x86/microcode_intel.c static int microcode_sanity_check(void *mc) · line 264
Quoted 6 lines
xen/arch/x86/microcode_intel.c +6
-0+enum microcode_match_result ret = +microcode_update_match (new->data, old_header->sig, + old_header->pf, old_header->rev); + +if ( ret ==NEW_UCODE ) +list_replace (&old->list, &new->list);
Quoted 2 lines
I think it would be better to leave actual list maintenance to generic code. That way the function parameters can also be pointer-to-const.
Will do.
Quoted 9 lines
xen/arch/x86/microcode_intel.c static int get_matching_microcode(const void *mc, unsigned int cpu) · line 281
Quoted 7 lines
xen/arch/x86/microcode_intel.c +2
-0@@ -248,6 +277,25 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)const struct microcode_header_intel *mc_header = mc;unsigned long total_size =get_totalsize (mc_header);void *new_mc; + struct microcode_patch *microcode_patch = xmalloc(struct microcode_patch); +void *new_mc2 =xmalloc_bytes (total_size);Why don't you use the already existing
new_mchere?
Will reuse the new_mc here.
Quoted 16 lines
Quoted 16 lines
xen/arch/x86/microcode_intel.c +4
-1@@ -276,18 +324,24 @@ static int apply_microcode(unsigned int cpu)unsigned int val[2 ];unsigned int cpu_num =raw_smp_processor_id ();struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu_num); +struct microcode_intel *mc_intel; +struct microcode_patch *patch;/* We should bind the task to the CPU */ BUG_ON (cpu_num != cpu); -if ( uci->mc.mc_intel ==NULL ) + patch =find_patch (cpu); +if ( !patch )return -EINVAL ;
Quoted 6 lines
xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 338
Quoted 2 lines
xen/arch/x86/microcode_intel.c +2
-0+ mc_intel = patch->data; +BUG_ON (!mc_intel);I'm not convinced this is a useful check - you never
save_patch()anything that has a NULL pointer here. And general corruption might put bad values other than NULL into the field.
Then I will remove this check.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 3/8] microcode: introduce the global microcode cache
Quoted 41 lines
Quoted 1 line
Quoted 1 line · On 11.02.19 at 04:59, <[email protected]> wrote:
Quoted 34 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode.c +16
-0@@ -208,6 +210,58 @@ static void microcode_fini_cpu(unsigned int cpu)spin_unlock (µcode_mutex); } +/* Save a ucode patch to the global cache list */ +bool save_patch (struct microcode_patch *new_patch) +{ +struct microcode_patch *microcode_patch; + +list_for_each_entry (microcode_patch, µcode_cache, list) + { +enum microcode_match_result result = + microcode_ops->replace_patch (new_patch, microcode_patch); + +switch ( result ) + { +case OLD_UCODE : + microcode_ops->free_patch (new_patch); +return false; +xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 238
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+case NEW_UCODE : + microcode_ops->free_patch (microcode_patch); +return true; + +case MIS_UCODE : +continue ; + +default : +ASSERT_UNREACHABLE (); +return 0 ;false (or true; either value is going to be fine/wrong here afaict)
Quoted 4 lines
Anyway I'm having some difficulty seeing what the intended meaning of the return value is, and without that being clear I also can't make up my mind whether I agree with the cases here.
Quoted 5 lines
The return value means whether saving a ucode patch succeeded or failed. In another word, it also means whether the ucode cache is updated or not. According to the return value, the caller decides not to do the expensive late ucode update for some cases (i.e. when admin provides a old ucode).
Would you mind extending the comment to the effect?
Quoted 23 lines
Quoted 23 lines
xen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 253
Quoted 9 lines
xen/arch/x86/microcode.c +9
-0+/* Find a ucode patch who has newer revision than the one in use */ +struct microcode_patch *find_patch (unsigned int cpu) +{ +struct microcode_patch *microcode_patch; +struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); + +if ( microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig) ) + { +__microcode_fini_cpu (cpu);This is kind of a library function - I can't see how it could legitimately invoke "fini", the more that you do here but not ...
Quoted 9 lines
xen/arch/x86/microcode.c +9
-0+return NULL ; + } + +list_for_each_entry (microcode_patch, µcode_cache, list) + { +if ( microcode_ops->match_cpu (microcode_patch, cpu) ) +return microcode_patch; + } +return NULL ;
Quoted 6 lines
Quoted 1 line
... here.
My understanding is saving the cpu signature and ucode revision can reduce MSR reads to get such information. So "fini" is invoked when "collect" failed. The
apply_microcode()followingfind_patch()can access "uci" without invoking "collect" again.
I fail to see the connection between "saving the cpu signature and ucode revision" and the apparent layering violation here. Could you help me with this?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 4/8 microcode: delete 'mc' field from struct ucode_cpu_info
In reply to Chao Gaoapply_microcode() now gets ucode patch from the global cache rather than using the microcode stored in "mc" field of ucode_cpu_info. Also remove 'microcode_resume_match' from microcode_ops because the matching is done in find_patch(). The cpu status notifier is also removed. It was used to free the "mc" field to avoid memory leak.
--- xen/arch/x86/microcode.c | 71 +++--------------------------------- xen/arch/x86/microcode_amd.c | 80 ++--------------------------------------- xen/arch/x86/microcode_intel.c | 27 +++----------- xen/include/asm-x86/microcode.h | 6 ---- 4 files changed, 12 insertions(+), 172 deletions(-)
xen/arch/x86/microcode.c +5 -66
@@ -195,21 +195,13 @@ struct microcode_info {char buffer[1 ]; }; -static void __microcode_fini_cpu (unsigned int cpu) +static void microcode_fini_cpu (unsigned int cpu) {struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); -xfree (uci->mc.mc_valid);memset (uci,0 ,sizeof (*uci)); } -static void microcode_fini_cpu (unsigned int cpu) -{ -spin_lock (µcode_mutex); -__microcode_fini_cpu (cpu); -spin_unlock (µcode_mutex); -} -/* Save a ucode patch to the global cache list */ bool save_patch (struct microcode_patch *new_patch) {
@@ -250,7 +242,7 @@ struct microcode_patch *find_patch(unsigned int cpu)if ( microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig) ) { -__microcode_fini_cpu (cpu); +microcode_fini_cpu (cpu);return NULL ; }
@@ -266,8 +258,6 @@ int microcode_resume_cpu(unsigned int cpu) {int err;struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); -struct cpu_signature nsig; -unsigned int cpu2;if ( !microcode_ops )return 0 ;
@@ -277,40 +267,12 @@ int microcode_resume_cpu(unsigned int cpu) err = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig);if ( err ) { -__microcode_fini_cpu (cpu); +microcode_fini_cpu (cpu);spin_unlock (µcode_mutex);return err; } -if ( uci->mc.mc_valid ) - { - err = microcode_ops->microcode_resume_match (cpu, uci->mc.mc_valid); -if ( err >=0 ) - { -if ( err ) - err = microcode_ops->apply_microcode (cpu); -spin_unlock (µcode_mutex); -return err; - } - } - - nsig = uci->cpu_sig; -__microcode_fini_cpu (cpu); - uci->cpu_sig = nsig; - - err = -EIO ; -for_each_online_cpu ( cpu2 ) - { - uci = &per_cpu (ucode_cpu_info, cpu2); -if ( uci->mc.mc_valid && - microcode_ops->microcode_resume_match (cpu, uci->mc.mc_valid) >0 ) - { - err = microcode_ops->apply_microcode (cpu); -break ; - } - } - -__microcode_fini_cpu (cpu); + err = microcode_ops->apply_microcode (cpu);spin_unlock (µcode_mutex);return err;
@@ -328,7 +290,7 @@ static int microcode_update_cpu(const void *buf, size_t size)if (likely (!err) ) err = microcode_ops->cpu_request_microcode (cpu, buf, size);else -__microcode_fini_cpu (cpu); +microcode_fini_cpu (cpu);spin_unlock (µcode_mutex);
@@ -416,25 +378,6 @@ static int __init microcode_init(void) }__initcall (microcode_init); -static int microcode_percpu_callback ( -struct notifier_block *nfb,unsigned long action,void *hcpu) -{ -unsigned int cpu = (unsigned long )hcpu; - -switch ( action ) - { -case CPU_DEAD : -microcode_fini_cpu (cpu); -break ; - } - -return NOTIFY_DONE ; -} - -static struct notifier_block microcode_percpu_nfb = { - .notifier_call = microcode_percpu_callback, -}; -int __init early_microcode_update_cpu (bool start_update) {int rc =0 ;
@@ -478,12 +421,8 @@ int __init early_microcode_init(void)return rc;if ( microcode_ops ) - {if ( ucode_mod.mod_end || ucode_blob.size ) rc =early_microcode_update_cpu (true); -register_cpu_notifier (µcode_percpu_nfb); - } -return rc; }
xen/arch/x86/microcode_amd.c +2 -78
125 lines not on this page · raw message
xen/arch/x86/microcode_intel.c +5 -22
@@ -276,19 +276,18 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu);const struct microcode_header_intel *mc_header = mc;unsigned long total_size =get_totalsize (mc_header); -void *new_mc; struct microcode_patch *microcode_patch = xmalloc(struct microcode_patch); -void *new_mc2 =xmalloc_bytes (total_size); +void *new_mc =xmalloc_bytes (total_size); -if ( !microcode_patch || !new_mc2 ) +if ( !microcode_patch || !new_mc ) {xfree (microcode_patch); -xfree (new_mc2); +xfree (new_mc);printk (XENLOG_ERR "microcode: Can not allocate memory\n" );return -ENOMEM ; } -memcpy (new_mc2, mc, total_size); - microcode_patch->data = new_mc2; +memcpy (new_mc, mc, total_size); + microcode_patch->data = new_mc;/* * In order to support a system with mixed stepping CPUs, save this ucode
@@ -304,16 +303,6 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)pr_debug ("microcode: CPU%d found a matching microcode update with" " version %#x (current=%#x)\n" , cpu, mc_header->rev, uci->cpu_sig.rev); - new_mc =xmalloc_bytes (total_size); -if ( new_mc ==NULL ) - { -printk (KERN_ERR "microcode: error! Can not allocate memory\n" ); -return -ENOMEM ; - } - -memcpy (new_mc, mc, total_size); -xfree (uci->mc.mc_intel); - uci->mc.mc_intel = new_mc;return 1 ; }
@@ -439,13 +428,7 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,return error; } -static int microcode_resume_match (unsigned int cpu,const void *mc) -{ -return get_matching_microcode (mc, cpu); -} -static const struct microcode_ops microcode_intel_ops = { - .microcode_resume_match = microcode_resume_match, .cpu_request_microcode = cpu_request_microcode, .collect_cpu_info = collect_cpu_info, .apply_microcode = apply_microcode,
xen/include/asm-x86/microcode.h +0 -6
@@ -19,7 +19,6 @@ struct microcode_patch { };struct microcode_ops { -int (*microcode_resume_match)(unsigned int cpu,const void *mc);int (*cpu_request_microcode)(unsigned int cpu,const void *buf,size_t size);int (*collect_cpu_info)(unsigned int cpu,struct cpu_signature *csig);
@@ -39,11 +38,6 @@ struct cpu_signature {struct ucode_cpu_info {struct cpu_signature cpu_sig; -union { -struct microcode_intel *mc_intel; -struct microcode_amd *mc_amd; -void *mc_valid; - } mc; }; DECLARE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info);
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 4/8] microcode: delete 'mc' field from struct ucode_cpu_info
Thanks for the cleanup!
Quoted 13 lines · Chao Gao
xen/include/asm-x86/microcode.h +0
-1@@ -19,7 +19,6 @@ struct microcode_patch { };struct microcode_ops { -int (*microcode_resume_match)(unsigned int cpu,const void *mc);int (*cpu_request_microcode)(unsigned int cpu,const void *buf,size_t size);int (*collect_cpu_info)(unsigned int cpu,struct cpu_signature *csig);
xen/include/asm-x86/microcode.h struct cpu_signature { · line 41
Quoted 9 lines
xen/include/asm-x86/microcode.h +0
-5@@ -39,11 +38,6 @@ struct cpu_signature {struct ucode_cpu_info {struct cpu_signature cpu_sig; -union { -struct microcode_intel *mc_intel; -struct microcode_amd *mc_amd; -void *mc_valid; - } mc; };
Is there really a need for such structure since it only has one field now?
I'm trying to figure out whether this is expanded by further patches, but it seems like it's not, if so please remove the struct altogether.
I'm also wondering whether it's needed to store the cpu signature in the pcpu area, AFAICT you always call collect_cpu_info before apply_microcode, at which point cpu_signature could be stored in the stack and passed to apply_microcode as a parameter?
Or apply_microcode could call collect_cpu_info directly. Getting rid of the pcpu field would also allow you to get rid of microcode_fini_cpu, further cleaning the code.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 4/8] microcode: delete 'mc' field from struct ucode_cpu_info
Quoted 32 lines · Roger Pau Monné
Thanks for the cleanup!
Quoted 13 lines · Chao Gao
xen/include/asm-x86/microcode.h +0
-1@@ -19,7 +19,6 @@ struct microcode_patch { };struct microcode_ops { -int (*microcode_resume_match)(unsigned int cpu,const void *mc);int (*cpu_request_microcode)(unsigned int cpu,const void *buf,size_t size);int (*collect_cpu_info)(unsigned int cpu,struct cpu_signature *csig);xen/include/asm-x86/microcode.h struct cpu_signature { · line 41
Quoted 9 lines
xen/include/asm-x86/microcode.h +0
-5@@ -39,11 +38,6 @@ struct cpu_signature {struct ucode_cpu_info {struct cpu_signature cpu_sig; -union { -struct microcode_intel *mc_intel; -struct microcode_amd *mc_amd; -void *mc_valid; - } mc; };Is there really a need for such structure since it only has one field now?
I'm trying to figure out whether this is expanded by further patches, but it seems like it's not, if so please remove the struct altogether.
Quoted 8 lines
I'm also wondering whether it's needed to store the cpu signature in the pcpu area, AFAICT you always call
collect_cpu_infobeforeapply_microcode, at which pointcpu_signaturecould be stored in the stack and passed toapply_microcodeas a parameter?Or
apply_microcodecould callcollect_cpu_infodirectly. Getting rid of the pcpu field would also allow you to get rid ofmicrocode_fini_cpu, further cleaning the code.
Your suggestions are viable and will follow them.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 5/8 microcode: split out apply_microcode() from cpu_request_microcode()
In reply to Chao GaoDuring late microcode update, apply_microcode() is invoked in cpu_request_microcode(). To make late microcode update more reliable, we want to put the apply_microcode() into stop_machine context. So we split out it from cpu_request_microcode(). As a consequence, apply_microcode() should be invoked explicitly in the common code.
Also with the global ucode cache, microcode parsing only needs to be done once; cpu_request_microcode() is also moved out of microcode_update_cpu().
On AMD side, svm_host_osvw_init() is supposed to be called after microcode update. As apply_micrcode() won't be called by cpu_request_microcode() now, svm_host_osvw_init() is also moved to the end of apply_microcode().
--- xen/arch/x86/microcode.c | 80 +++++++++++++++++++++++++----------------- xen/arch/x86/microcode_amd.c | 23 +++++++----- xen/arch/x86/microcode_intel.c | 20 ++--------- 3 files changed, 64 insertions(+), 59 deletions(-)
xen/arch/x86/microcode.c +47 -33
@@ -254,47 +254,42 @@ struct microcode_patch *find_patch(unsigned int cpu)return NULL ; } -int microcode_resume_cpu (unsigned int cpu) +/* +* Return the number of ucode patch inserted to the global cache. +* Return negtive value on error. +*/ +static int parse_microcode_blob (const void *buffer,size_t len) { -int err; +unsigned int cpu =smp_processor_id ();struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); - -if ( !microcode_ops ) -return 0 ; +int ret;spin_lock (µcode_mutex); - - err = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig); -if ( err ) - { + ret = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig); +if (likely (!ret) ) + ret = microcode_ops->cpu_request_microcode (cpu, buffer, len); +else microcode_fini_cpu (cpu); -spin_unlock (µcode_mutex); -return err; - } - - err = microcode_ops->apply_microcode (cpu);spin_unlock (µcode_mutex); -return err; +return ret; } -static int microcode_update_cpu (const void *buf,size_t size) +static int microcode_update_cpu (void ) { -int err; -unsigned int cpu =smp_processor_id (); -struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); +int ret;spin_lock (µcode_mutex); - - err = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig); -if (likely (!err) ) - err = microcode_ops->cpu_request_microcode (cpu, buf, size); -else -microcode_fini_cpu (cpu); - + ret = microcode_ops->apply_microcode (smp_processor_id ());spin_unlock (µcode_mutex); -return err; +return ret; +} + +int microcode_resume_cpu (unsigned int cpu) +{ +BUG_ON (cpu !=smp_processor_id ()); +return microcode_ops ?microcode_update_cpu () :0 ; }static long do_microcode_update (void *_info)
@@ -304,7 +299,7 @@ static long do_microcode_update(void *_info)BUG_ON (info->cpu !=smp_processor_id ()); - error =microcode_update_cpu (info->buffer, info->buffer_size); + error =microcode_update_cpu ();if ( error ) info->error = error;
@@ -339,10 +334,6 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)return ret; } - info->buffer_size = len; - info->error =0 ; - info->cpu =cpumask_first (&cpu_online_map); -if ( microcode_ops->start_update ) { ret = microcode_ops->start_update ();
@@ -353,6 +344,18 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) } } + ret =parse_microcode_blob (info->buffer, len); +if ( ret <=0 ) + { +printk (XENLOG_ERR "No valid or newer ucode found. Update abort!\n" ); +xfree (info); +return -EINVAL ; + } + + info->buffer_size = len; + info->error =0 ; + info->cpu =cpumask_first (&cpu_online_map); +return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); }
@@ -396,13 +399,24 @@ int __init early_microcode_update_cpu(bool start_update) }if ( data ) { +static bool parsed = false; +if ( start_update && microcode_ops->start_update ) rc = microcode_ops->start_update ();if ( rc )return rc; -return microcode_update_cpu (data, len); +if ( !parsed ) + { + rc =parse_microcode_blob (data, len); + parsed = true; + +if ( rc <=0 ) +return -EINVAL ; + } + +return microcode_update_cpu (); }else return -ENOMEM ;
xen/arch/x86/microcode_amd.c +14 -9
@@ -297,6 +297,10 @@ static int apply_microcode(unsigned int cpu) uci->cpu_sig.rev = rev; +#if CONFIG_HVM +svm_host_osvw_init (); +#endif +return 0 ; }
@@ -463,6 +467,7 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu);unsigned int current_cpu_id;unsigned int equiv_cpu_id; +unsigned int matched_cnt =0 ;/* We should bind the task to the CPU */ BUG_ON (cpu !=raw_smp_processor_id ());
@@ -564,11 +569,15 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,* this ucode patch before checking whether it matches with * current CPU. */ -if (save_patch (microcode_patch) &µcode_fits (mc_amd, cpu) ) +if (save_patch (microcode_patch) ) { - error =apply_microcode (cpu); -if ( error ) -break ; + matched_cnt++; +if (microcode_fits (mc_amd, cpu) ) + { + error =apply_microcode (cpu); +if ( error ) +break ; + } }if ( offset >= bufsize )
@@ -600,17 +609,13 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf, } out: -#if CONFIG_HVM -svm_host_osvw_init (); -#endif -/* * In some cases we may return an error even if processor's microcode has * been updated. For example, the first patch in a container file is loaded * successfully but subsequent container file processing encounters a * failure. */ -return error; +return error ?: matched_cnt; }static int start_update (void )
xen/arch/x86/microcode_intel.c +3 -17
@@ -273,7 +273,6 @@ static enum microcode_match_result replace_patch(struct microcode_patch *new,*/ static int get_matching_microcode (const void *mc,unsigned int cpu) { -struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu);const struct microcode_header_intel *mc_header = mc;unsigned long total_size =get_totalsize (mc_header); struct microcode_patch *microcode_patch = xmalloc(struct microcode_patch);
@@ -291,19 +290,9 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)/* * In order to support a system with mixed stepping CPUs, save this ucode -* patch before checking whether it matches with current CPU. +* patch without checking whether it matches with current CPU. */ -if ( !save_patch (microcode_patch) ) -return 0 ; - -if (microcode_update_match (mc, uci->cpu_sig.sig, uci->cpu_sig.pf, - uci->cpu_sig.rev) !=NEW_UCODE ) -return 0 ; - -pr_debug ("microcode: CPU%d found a matching microcode update with" -" version %#x (current=%#x)\n" , - cpu, mc_header->rev, uci->cpu_sig.rev); -return 1 ; +return save_patch (microcode_patch); }static int apply_microcode (unsigned int cpu)
@@ -422,10 +411,7 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,if ( offset <0 ) error = offset; -if ( !error && matching_count ) - error =apply_microcode (cpu); - -return error; +return error ?: matching_count; }static const struct microcode_ops microcode_intel_ops = {
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 5/8] microcode: split out apply_microcode() from cpu_request_microcode()
Quoted 73 lines · Chao Gao
During late microcode update,
apply_microcode()is invoked incpu_request_microcode(). To make late microcode update more reliable, we want to put theapply_microcode()intostop_machinecontext. So we split out it fromcpu_request_microcode(). As a consequence,apply_microcode()should be invoked explicitly in the common code.Also with the global ucode cache, microcode parsing only needs to be done once;
cpu_request_microcode()is also moved out ofmicrocode_update_cpu().On AMD side,
svm_host_osvw_init()is supposed to be called after microcode update. Asapply_micrcode()won't be called bycpu_request_microcode()now,svm_host_osvw_init()is also moved to the end ofapply_microcode().Signed-off-by Chao Gao <[email protected]>--- xen/arch/x86/microcode.c | 80 +++++++++++++++++++++++++----------------- xen/arch/x86/microcode_amd.c | 23 +++++++----- xen/arch/x86/microcode_intel.c | 20 ++--------- 3 files changed, 64 insertions(+), 59 deletions(-)xen/arch/x86/microcode.c +14
-19@@ -254,47 +254,42 @@ struct microcode_patch *find_patch(unsigned int cpu)return NULL ; } -int microcode_resume_cpu (unsigned int cpu) +/* +* Return the number of ucode patch inserted to the global cache. +* Return negtive value on error. +*/ +static int parse_microcode_blob (const void *buffer,size_t len) { -int err; +unsigned int cpu =smp_processor_id ();struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); - -if ( !microcode_ops ) -return 0 ; +int ret;spin_lock (µcode_mutex); - - err = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig); -if ( err ) - { + ret = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig); +if (likely (!ret) ) + ret = microcode_ops->cpu_request_microcode (cpu, buffer, len); +else microcode_fini_cpu (cpu); -spin_unlock (µcode_mutex); -return err; - } - - err = microcode_ops->apply_microcode (cpu);spin_unlock (µcode_mutex); -return err; +return ret; } -static int microcode_update_cpu (const void *buf,size_t size) +static int microcode_update_cpu (void ) { -int err; -unsigned int cpu =smp_processor_id (); -struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); +int ret;
xen/arch/x86/microcode.c struct microcode_patch *find_patch(unsigned int cpu) · line 283
Quoted 10 lines
xen/arch/x86/microcode.c +1
-7spin_lock (µcode_mutex); - - err = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig); -if (likely (!err) ) - err = microcode_ops->cpu_request_microcode (cpu, buf, size); -else -microcode_fini_cpu (cpu); - + ret = microcode_ops->apply_microcode (smp_processor_id ());
I've realized that most of this helpers take a cpu id parameter (apply_microcode, collect_cpu_info and cpu_request_microcode), but that at least on Intel they are required to be called on the current CPU, at which point I wonder about their purpose. IMO they just make the interface more messy, without adding any functionality.
xen/arch/x86/microcode.c struct microcode_patch *find_patch(unsigned int cpu) · line 291
Quoted 9 lines
xen/arch/x86/microcode.c +6
-1spin_unlock (µcode_mutex); -return err; +return ret; +} + +int microcode_resume_cpu (unsigned int cpu) +{ +BUG_ON (cpu !=smp_processor_id ());
Same here, what's the point of passing a cpu id as parameter when there's only one valid value?
Quoted 29 lines
xen/arch/x86/microcode.c +3
-5+return microcode_ops ?microcode_update_cpu () :0 ; }static long do_microcode_update (void *_info)@@ -304,7 +299,7 @@ static long do_microcode_update(void *_info)BUG_ON (info->cpu !=smp_processor_id ()); - error =microcode_update_cpu (info->buffer, info->buffer_size); + error =microcode_update_cpu ();if ( error ) info->error = error;@@ -339,10 +334,6 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)return ret; } - info->buffer_size = len; - info->error =0 ; - info->cpu =cpumask_first (&cpu_online_map); -if ( microcode_ops->start_update ) { ret = microcode_ops->start_update ();@@ -353,6 +344,18 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) } } + ret =parse_microcode_blob (info->buffer, len);
xen/arch/x86/microcode.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 357
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+if ( ret <=0 ) + { +printk (XENLOG_ERR "No valid or newer ucode found. Update abort!\n" ); +xfree (info); +return -EINVAL ; + } + + info->buffer_size = len; + info->error =0 ; + info->cpu =cpumask_first (&cpu_online_map);
It looks like you can also get rid of the cpu field in microcode_info, since it's always smp_processor_id?
Quoted 17 lines
xen/arch/x86/microcode.c +3
-1+return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); }@@ -396,13 +399,24 @@ int __init early_microcode_update_cpu(bool start_update) }if ( data ) { +static bool parsed = false; +if ( start_update && microcode_ops->start_update ) rc = microcode_ops->start_update ();if ( rc )return rc; -return microcode_update_cpu (data, len);
xen/arch/x86/microcode.c int __init early_microcode_update_cpu(bool start_update) · line 419
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+if ( !parsed ) + { + rc =parse_microcode_blob (data, len); + parsed = true; + +if ( rc <=0 ) +return -EINVAL ; + } + +return microcode_update_cpu ();
Hm, the usage of parsed here is kind of dangerous. I assume this works fine because early_microcode_update_cpu is called from the BSP in early_microcode_init, and then concurrent calls done by the APs always see parsed as true.
I would however recommend that you move the parsing to early_microcode_init, and that early_microcode_update_cpu always assume the blob has been parsed.
If that doesn't work for some reason, I would then recommend that you gate the parsing based on cpu_id, so "smp_processor_id() == 0"
Quoted 33 lines
xen/arch/x86/microcode.c +0
-0}else return -ENOMEM ;xen/arch/x86/microcode_amd.c +6
-1@@ -297,6 +297,10 @@ static int apply_microcode(unsigned int cpu) uci->cpu_sig.rev = rev; +#if CONFIG_HVM +svm_host_osvw_init (); +#endif +return 0 ; }@@ -463,6 +467,7 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu);unsigned int current_cpu_id;unsigned int equiv_cpu_id; +unsigned int matched_cnt =0 ;/* We should bind the task to the CPU */ BUG_ON (cpu !=raw_smp_processor_id ());@@ -564,11 +569,15 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,* this ucode patch before checking whether it matches with * current CPU. */ -if (save_patch (microcode_patch) &µcode_fits (mc_amd, cpu) ) +if (save_patch (microcode_patch) ) {
xen/arch/x86/microcode_amd.c static int cpu_request_microcode(unsigned int cpu, const void *buf, · line 580
Quoted 10 lines
xen/arch/x86/microcode_amd.c +7
-3- error =apply_microcode (cpu); -if ( error ) -break ; + matched_cnt++; +if (microcode_fits (mc_amd, cpu) ) + { + error =apply_microcode (cpu); +if ( error ) +break ; + }
In the commit message you mention that apply_microcode won't be called by cpu_request_microcode, yet it seems it's called?
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 5/8] microcode: split out apply_microcode() from cpu_request_microcode()
Quoted 75 lines · Roger Pau Monné
Quoted 74 lines · Chao Gao
During late microcode update,
apply_microcode()is invoked incpu_request_microcode(). To make late microcode update more reliable, we want to put theapply_microcode()intostop_machinecontext. So we split out it fromcpu_request_microcode(). As a consequence,apply_microcode()should be invoked explicitly in the common code.Also with the global ucode cache, microcode parsing only needs to be done once;
cpu_request_microcode()is also moved out ofmicrocode_update_cpu().On AMD side,
svm_host_osvw_init()is supposed to be called after microcode update. Asapply_micrcode()won't be called bycpu_request_microcode()now,svm_host_osvw_init()is also moved to the end ofapply_microcode().Signed-off-by Chao Gao <[email protected]>--- xen/arch/x86/microcode.c | 80 +++++++++++++++++++++++++----------------- xen/arch/x86/microcode_amd.c | 23 +++++++----- xen/arch/x86/microcode_intel.c | 20 ++--------- 3 files changed, 64 insertions(+), 59 deletions(-)xen/arch/x86/microcode.c +14
-19@@ -254,47 +254,42 @@ struct microcode_patch *find_patch(unsigned int cpu)return NULL ; } -int microcode_resume_cpu (unsigned int cpu) +/* +* Return the number of ucode patch inserted to the global cache. +* Return negtive value on error. +*/ +static int parse_microcode_blob (const void *buffer,size_t len) { -int err; +unsigned int cpu =smp_processor_id ();struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); - -if ( !microcode_ops ) -return 0 ; +int ret;spin_lock (µcode_mutex); - - err = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig); -if ( err ) - { + ret = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig); +if (likely (!ret) ) + ret = microcode_ops->cpu_request_microcode (cpu, buffer, len); +else microcode_fini_cpu (cpu); -spin_unlock (µcode_mutex); -return err; - } - - err = microcode_ops->apply_microcode (cpu);spin_unlock (µcode_mutex); -return err; +return ret; } -static int microcode_update_cpu (const void *buf,size_t size) +static int microcode_update_cpu (void ) { -int err; -unsigned int cpu =smp_processor_id (); -struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); +int ret;
Quoted 15 lines
xen/arch/x86/microcode.c struct microcode_patch *find_patch(unsigned int cpu) · line 283
Quoted 9 lines
xen/arch/x86/microcode.c +1
-7spin_lock (µcode_mutex); - - err = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig); -if (likely (!err) ) - err = microcode_ops->cpu_request_microcode (cpu, buf, size); -else -microcode_fini_cpu (cpu); - + ret = microcode_ops->apply_microcode (smp_processor_id ());I've realized that most of this helpers take a cpu id parameter (
apply_microcode,collect_cpu_infoandcpu_request_microcode), but that at least on Intel they are required to be called on the current CPU, at which point I wonder about their purpose. IMO they just make the interface more messy, without adding any functionality.
I agree with you and will remove the 'cpu' parameter from the interfaces you mentioned above.
Quoted 42 lines
xen/arch/x86/microcode.c struct microcode_patch *find_patch(unsigned int cpu) · line 291
Quoted 10 lines
xen/arch/x86/microcode.c +6
-1spin_unlock (µcode_mutex); -return err; +return ret; +} + +int microcode_resume_cpu (unsigned int cpu) +{ +BUG_ON (cpu !=smp_processor_id ());Same here, what's the point of passing a cpu id as parameter when there's only one valid value?
Quoted 28 lines
xen/arch/x86/microcode.c +2
-5+return microcode_ops ?microcode_update_cpu () :0 ; }static long do_microcode_update (void *_info)@@ -304,7 +299,7 @@ static long do_microcode_update(void *_info)BUG_ON (info->cpu !=smp_processor_id ()); - error =microcode_update_cpu (info->buffer, info->buffer_size); + error =microcode_update_cpu ();if ( error ) info->error = error;@@ -339,10 +334,6 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)return ret; } - info->buffer_size = len; - info->error =0 ; - info->cpu =cpumask_first (&cpu_online_map); -if ( microcode_ops->start_update ) { ret = microcode_ops->start_update ();@@ -353,6 +344,18 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) } }
Quoted 14 lines
xen/arch/x86/microcode.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 357
Quoted 11 lines
xen/arch/x86/microcode.c +11
-0+ ret =parse_microcode_blob (info->buffer, len); +if ( ret <=0 ) + { +printk (XENLOG_ERR "No valid or newer ucode found. Update abort!\n" ); +xfree (info); +return -EINVAL ; + } + + info->buffer_size = len; + info->error =0 ; + info->cpu =cpumask_first (&cpu_online_map);It looks like you can also get rid of the cpu field in
microcode_info, since it's alwayssmp_processor_id?
Yes. Will remove the 'cpu' field.
Quoted 17 lines
Quoted 17 lines
xen/arch/x86/microcode.c +3
-0+return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); }@@ -396,13 +399,24 @@ int __init early_microcode_update_cpu(bool start_update) }if ( data ) { +static bool parsed = false; +if ( start_update && microcode_ops->start_update ) rc = microcode_ops->start_update ();if ( rc )return rc;
Quoted 16 lines
xen/arch/x86/microcode.c int __init early_microcode_update_cpu(bool start_update) · line 419
Quoted 11 lines
xen/arch/x86/microcode.c +10
-1-return microcode_update_cpu (data, len); +if ( !parsed ) + { + rc =parse_microcode_blob (data, len); + parsed = true; + +if ( rc <=0 ) +return -EINVAL ; + } + +return microcode_update_cpu ();Hm, the usage of parsed here is kind of dangerous. I assume this works fine because
early_microcode_update_cpuis called from the BSP inearly_microcode_init, and then concurrent calls done by the APs always see parsed as true.
I think APs are woken up one-by-one. See the call site of cpu_up in __start_xen.
Quoted 7 lines
I would however recommend that you move the parsing to
early_microcode_init, and thatearly_microcode_update_cpualways assume the blob has been parsed.If that doesn't work for some reason, I would then recommend that you gate the parsing based on
cpu_id, so "smp_processor_id()== 0"
Do you think it is better:
remove the parsed flag and check whether current cpu is bsp in early_microcode_init(); bsp calls the early_microcode_update_cpu(). APs just call microcode_update_cpu().
Quoted 39 lines
Quoted 39 lines
xen/arch/x86/microcode.c +0
-0}else return -ENOMEM ;xen/arch/x86/microcode_amd.c +8
-4@@ -297,6 +297,10 @@ static int apply_microcode(unsigned int cpu) uci->cpu_sig.rev = rev; +#if CONFIG_HVM +svm_host_osvw_init (); +#endif +return 0 ; }@@ -463,6 +467,7 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu);unsigned int current_cpu_id;unsigned int equiv_cpu_id; +unsigned int matched_cnt =0 ;/* We should bind the task to the CPU */ BUG_ON (cpu !=raw_smp_processor_id ());@@ -564,11 +569,15 @@ static int cpu_request_microcode(unsigned int cpu, const void *buf,* this ucode patch before checking whether it matches with * current CPU. */ -if (save_patch (microcode_patch) &µcode_fits (mc_amd, cpu) ) +if (save_patch (microcode_patch) ) { - error =apply_microcode (cpu); -if ( error ) -break ; + matched_cnt++; +if (microcode_fits (mc_amd, cpu) )
Quoted 8 lines
xen/arch/x86/microcode_amd.c static int cpu_request_microcode(unsigned int cpu, const void *buf, · line 580
Quoted 5 lines
xen/arch/x86/microcode_amd.c +5
-0+ { + error =apply_microcode (cpu); +if ( error ) +break ; + }In the commit message you mention that
apply_microcodewon't be called bycpu_request_microcode, yet it seems it's called?
lines below "matched_cnt++" should be removed. They slipped in when I did a rebasing.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 5/8] microcode: split out apply_microcode() from cpu_request_microcode()
In reply to Chao Gaoxen/arch/x86/microcode_amd.c static int apply_microcode(unsigned int cpu) · line 302
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode_amd.c +3
-0@@ -297,6 +297,10 @@ static int apply_microcode(unsigned int cpu) uci->cpu_sig.rev = rev; +#if CONFIG_HVM +svm_host_osvw_init (); +#endif
#ifdef (it's been wrong at where you move it from)
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 6/8 microcode: delete microcode pointer and size from microcode_info
In reply to Chao Gaomicrocode pointer and size were passed to other CPUs to parse microcode locally. Now, parsing microcode is done on one CPU. Other CPUs needn't parse the microcode blob; the pointer and size can be removed.
--- xen/arch/x86/microcode.c | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-)
xen/arch/x86/microcode.c +17 -16
@@ -190,9 +190,7 @@ DEFINE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info);struct microcode_info {unsigned int cpu; -uint32_t buffer_size;int error; -char buffer[1 ]; };static void microcode_fini_cpu (unsigned int cpu)
@@ -316,6 +314,7 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) {int ret;struct microcode_info *info; +void * buffer;if ( len != (uint32_t )len )return -E2BIG ;
@@ -323,28 +322,26 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)if ( microcode_ops ==NULL )return -EINVAL ; - info =xmalloc_bytes (sizeof (*info) + len); -if ( info ==NULL ) -return -ENOMEM ; - - ret =copy_from_guest (info->buffer, buf, len); -if ( ret !=0 ) + info = xmalloc(struct microcode_info); + buffer =xmalloc_bytes (len); +if ( !info || !buffer ) { -xfree (info); -return ret; + ret = -ENOMEM ; + goto free; } + ret =copy_from_guest (buffer, buf, len); +if ( ret !=0 ) + goto free; +if ( microcode_ops->start_update ) { ret = microcode_ops->start_update ();if ( ret !=0 ) - { -xfree (info); -return ret; - } + goto free; } - ret =parse_microcode_blob (info->buffer, len); + ret =parse_microcode_blob (buffer, len);if ( ret <=0 ) {printk (XENLOG_ERR "No valid or newer ucode found. Update abort!\n" );
@@ -352,11 +349,15 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)return -EINVAL ; } - info->buffer_size = len; info->error =0 ; info->cpu =cpumask_first (&cpu_online_map);return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); + + free: +xfree (info); +xfree (buffer); +return ret; }static int __init microcode_init (void )
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 6/8] microcode: delete microcode pointer and size from microcode_info
Quoted 15 lines · Chao Gao
microcode pointer and size were passed to other CPUs to parse microcode locally. Now, parsing microcode is done on one CPU. Other CPUs needn't parse the microcode blob; the pointer and size can be removed.
Signed-off-by Chao Gao <[email protected]>--- xen/arch/x86/microcode.c | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-)xen/arch/x86/microcode.c +0
-0
xen/arch/x86/microcode.c DEFINE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info); · line 192
Quoted 3 lines
xen/arch/x86/microcode.c +0
-0@@ -190,9 +190,7 @@ DEFINE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info);struct microcode_info {unsigned int cpu;
I think cpu can also be removed as my previous reply to patch 5, at which point you only need to store an error? In which case you should also remove this struct then.
Quoted 27 lines
xen/arch/x86/microcode.c +4
-8-uint32_t buffer_size;int error; -char buffer[1 ]; };static void microcode_fini_cpu (unsigned int cpu)@@ -316,6 +314,7 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) {int ret;struct microcode_info *info; +void * buffer;if ( len != (uint32_t )len )return -E2BIG ;@@ -323,28 +322,26 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)if ( microcode_ops ==NULL )return -EINVAL ; - info =xmalloc_bytes (sizeof (*info) + len); -if ( info ==NULL ) -return -ENOMEM ; - - ret =copy_from_guest (info->buffer, buf, len); -if ( ret !=0 ) + info = xmalloc(struct microcode_info); + buffer =xmalloc_bytes (len); +if ( !info || !buffer )
xen/arch/x86/microcode.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 335
Quoted 10 lines
xen/arch/x86/microcode.c +5
-2{ -xfree (info); -return ret; + ret = -ENOMEM ; + goto free; } + ret =copy_from_guest (buffer, buf, len); +if ( ret !=0 ) + goto free;
copy_from_guest doesn't return an errno value, you have to set ret to EFAULT:
if ( copy_from_guest(buffer, buf, len) )
{
ret = -EFAULT;
goto free;
}Quoted 5 lines
xen/arch/x86/microcode.c +1
-0+if ( microcode_ops->start_update ) { ret = microcode_ops->start_update ();if ( ret !=0 )
xen/arch/x86/microcode.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 345
Quoted 10 lines
xen/arch/x86/microcode.c +2
-5- { -xfree (info); -return ret; - } + goto free; } - ret =parse_microcode_blob (info->buffer, len); + ret =parse_microcode_blob (buffer, len);if ( ret <=0 )
Don't you need to free info and buffer here also?
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 6/8] microcode: delete microcode pointer and size from microcode_info
Quoted 17 lines · Roger Pau Monné
Quoted 16 lines · Chao Gao
microcode pointer and size were passed to other CPUs to parse microcode locally. Now, parsing microcode is done on one CPU. Other CPUs needn't parse the microcode blob; the pointer and size can be removed.
Signed-off-by Chao Gao <[email protected]>--- xen/arch/x86/microcode.c | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-)xen/arch/x86/microcode.c +0
-0@@ -190,9 +190,7 @@ DEFINE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info);
Quoted 6 lines
xen/arch/x86/microcode.c DEFINE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info); · line 192
Quoted 2 lines
xen/arch/x86/microcode.c +0
-0struct microcode_info {unsigned int cpu;I think cpu can also be removed as my previous reply to patch 5, at which point you only need to store an error? In which case you should also remove this struct then.
I agree to all your comments on this patch. Will correct them.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 7/8 x86/microcode: Synchronize late microcode loading
In reply to Chao GaoThis patch ports microcode improvement patches from linux kernel.
Before you read any further: the early loading method is still the preferred one and you should always do that. The following patch is improving the late loading mechanism for long running jobs and cloud use cases.
Gather all cores and serialize the microcode update on them by doing it one-by-one to make the late update process as reliable as possible and avoid potential issues caused by the microcode update.
--- xen/arch/x86/microcode.c | 125 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 98 insertions(+), 27 deletions(-)
xen/arch/x86/microcode.c +98 -27
@@ -22,6 +22,7 @@*/ #include <xen/cpu.h> +#include <xen/cpumask.h> #include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h>
@@ -30,18 +31,25 @@#include <xen/smp.h> #include <xen/softirq.h> #include <xen/spinlock.h> +#include <xen/stop_machine.h> #include <xen/tasklet.h> #include <xen/guest_access.h> #include <xen/earlycpio.h> +#include <xen/watchdog.h> +#include <asm/delay.h> #include <asm/msr.h> #include <asm/processor.h> #include <asm/setup.h> #include <asm/microcode.h> +/* By default, wait for 30000us */ +#define MICROCODE_DEFAULT_TIMEOUT_US 30000 +static module_t __initdata ucode_mod;static signed int __initdata ucode_mod_idx;static bool_t __initdata ucode_mod_forced; +static unsigned int nr_cores;/* * If we scan the initramfs.cpio for the early microcode code
@@ -188,10 +196,11 @@ static DEFINE_SPINLOCK(microcode_mutex); DEFINE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info); -struct microcode_info { -unsigned int cpu; -int error; -}; +/* +* Count the CPUs that have entered and exited the rendezvous +* during late microcode update. +*/ +static atomic_t cpu_in, cpu_out;static void microcode_fini_cpu (unsigned int cpu) {
@@ -290,30 +299,60 @@ int microcode_resume_cpu(unsigned int cpu)return microcode_ops ?microcode_update_cpu () :0 ; } -static long do_microcode_update (void *_info) +/* Wait for all CPUs to rendezvous with a timeout (us) */ +static int wait_for_cpus (atomic_t *cnt,unsigned int timeout) { -struct microcode_info *info = _info; -int error; +unsigned int cpus =num_online_cpus (); -BUG_ON (info->cpu !=smp_processor_id ()); +atomic_inc (cnt); + +while (atomic_read (cnt) != cpus ) + { +if ( timeout <=0 ) + { +printk ("CPU%d: Timeout when waiting for CPUs calling in\n" , +smp_processor_id ()); +return -EBUSY ; + } +udelay (1 ); + timeout--; + } + +return 0 ; +} + +static int do_microcode_update (void *unused) +{ +unsigned int cpu =smp_processor_id (); +int ret; - error =microcode_update_cpu (); -if ( error ) - info->error = error; + ret =wait_for_cpus (&cpu_in,MICROCODE_DEFAULT_TIMEOUT_US ); +if ( ret ) +return ret; - info->cpu =cpumask_next (info->cpu, &cpu_online_map); -if ( info->cpu < nr_cpu_ids ) -return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); +/* +* Initiate an update on all processors which don't have an online sibling +* thread with a lower thread id. Other sibling threads just await the +* completion of microcode update. +*/ +if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + ret =microcode_update_cpu (); +/* +* Increase the wait timeout to a safe value here since we're serializing +* the microcode update and that could take a while on a large number of +* CPUs. And that is fine as the *actual* timeout will be determined by +* the last CPU finished updating and thus cut short +*/ +if (wait_for_cpus (&cpu_out,MICROCODE_DEFAULT_TIMEOUT_US * nr_cores) ) +panic ("Timeout when finishing updating microcode" ); - error = info->error; -xfree (info); -return error; +return ret; }int microcode_update (XEN_GUEST_HANDLE_PARAM (const_void) buf,unsigned long len) {int ret; -struct microcode_info *info; +unsigned int cpu;void * buffer;if ( len != (uint32_t )len )
@@ -322,9 +361,8 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)if ( microcode_ops ==NULL )return -EINVAL ; - info = xmalloc(struct microcode_info); buffer =xmalloc_bytes (len); -if ( !info || !buffer ) +if ( !buffer ) { ret = -ENOMEM ; goto free;
@@ -334,28 +372,61 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)if ( ret !=0 ) goto free; +/* cpu_online_map must not change during update */ +if ( !get_cpu_maps () ) + { + ret = -EBUSY ; + goto free; + } +if ( microcode_ops->start_update ) { ret = microcode_ops->start_update ();if ( ret !=0 ) - goto free; + goto put; } ret =parse_microcode_blob (buffer, len);if ( ret <=0 ) {printk (XENLOG_ERR "No valid or newer ucode found. Update abort!\n" ); -xfree (info); -return -EINVAL ; + ret = -EINVAL ; + goto put; } - info->error =0 ; - info->cpu =cpumask_first (&cpu_online_map); +atomic_set (&cpu_in,0 ); +atomic_set (&cpu_out,0 ); + +/* Calculate the number of online CPU core */ + nr_cores =0 ; +for_each_online_cpu (cpu) +if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + nr_cores++; + +printk (XENLOG_INFO "%d cores are to update their microcode\n" , nr_cores); -return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); +/* +* We intend to disable interrupt for long time, which may lead to +* watchdog timeout. +*/ +watchdog_disable (); +/* +* Late loading dance. Why the heavy-handed stop_machine effort? +* +* - HT siblings must be idle and not execute other code while the other +* sibling is loading microcode in order to avoid any negative +* interactions cause by the loading. +* +* - In addition, microcode update on the cores must be serialized until +* this requirement can be relaxed in the future. Right now, this is +* conservative and good. +*/ + ret =stop_machine_run (do_microcode_update,NULL ,NR_CPUS ); +watchdog_enable (); + put: +put_cpu_maps (); free: -xfree (info);xfree (buffer);return ret; }
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 7/8] x86/microcode: Synchronize late microcode loading
Quoted 94 lines · Chao Gao
This patch ports microcode improvement patches from linux kernel.
Before you read any further: the early loading method is still the preferred one and you should always do that. The following patch is improving the late loading mechanism for long running jobs and cloud use cases.
Gather all cores and serialize the microcode update on them by doing it one-by-one to make the late update process as reliable as possible and avoid potential issues caused by the microcode update.
Signed-off-by Chao Gao <[email protected]>Tested-by Chao Gao <[email protected]>[linux commit: a5321aec6412b20b5ad15db2d6b916c05349dbff][linux commit: bb8c13d61a629276a162c1d2b1a20a815cbcfbb7]Cc Kevin Tian <[email protected]>Cc Jun Nakajima <[email protected]>Cc Ashok Raj <[email protected]>Cc Borislav Petkov <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Andrew Cooper <[email protected]>Cc Jan Beulich <[email protected]>--- xen/arch/x86/microcode.c | 125 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 98 insertions(+), 27 deletions(-)xen/arch/x86/microcode.c +17
-8@@ -22,6 +22,7 @@*/ #include <xen/cpu.h> +#include <xen/cpumask.h> #include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> @@ -30,18 +31,25 @@#include <xen/smp.h> #include <xen/softirq.h> #include <xen/spinlock.h> +#include <xen/stop_machine.h> #include <xen/tasklet.h> #include <xen/guest_access.h> #include <xen/earlycpio.h> +#include <xen/watchdog.h> +#include <asm/delay.h> #include <asm/msr.h> #include <asm/processor.h> #include <asm/setup.h> #include <asm/microcode.h> +/* By default, wait for 30000us */ +#define MICROCODE_DEFAULT_TIMEOUT_US 30000 +static module_t __initdata ucode_mod;static signed int __initdata ucode_mod_idx;static bool_t __initdata ucode_mod_forced; +static unsigned int nr_cores;/* * If we scan the initramfs.cpio for the early microcode code @@ -188,10 +196,11 @@ static DEFINE_SPINLOCK(microcode_mutex); DEFINE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info); -struct microcode_info { -unsigned int cpu; -int error; -}; +/* +* Count the CPUs that have entered and exited the rendezvous +* during late microcode update. +*/ +static atomic_t cpu_in, cpu_out;static void microcode_fini_cpu (unsigned int cpu) {@@ -290,30 +299,60 @@ int microcode_resume_cpu(unsigned int cpu)return microcode_ops ?microcode_update_cpu () :0 ; } -static long do_microcode_update (void *_info) +/* Wait for all CPUs to rendezvous with a timeout (us) */ +static int wait_for_cpus (atomic_t *cnt,unsigned int timeout) { -struct microcode_info *info = _info; -int error; +unsigned int cpus =num_online_cpus (); -BUG_ON (info->cpu !=smp_processor_id ()); +atomic_inc (cnt);
xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 317
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+ +while (atomic_read (cnt) != cpus ) + { +if ( timeout <=0 ) + { +printk ("CPU%d: Timeout when waiting for CPUs calling in\n" , +smp_processor_id ()); +return -EBUSY ; + } +udelay (1 );
udelay will call the rdtsc instruction, is it fine to use it on a sibling thread while there's a microcode update in process on the same core?
Quoted 18 lines
xen/arch/x86/microcode.c +13
-3+ timeout--; + } + +return 0 ; +} + +static int do_microcode_update (void *unused) +{ +unsigned int cpu =smp_processor_id (); +int ret; - error =microcode_update_cpu (); -if ( error ) - info->error = error; + ret =wait_for_cpus (&cpu_in,MICROCODE_DEFAULT_TIMEOUT_US ); +if ( ret ) +return ret;
xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 339
Quoted 10 lines
xen/arch/x86/microcode.c +7
-3- info->cpu =cpumask_next (info->cpu, &cpu_online_map); -if ( info->cpu < nr_cpu_ids ) -return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); +/* +* Initiate an update on all processors which don't have an online sibling +* thread with a lower thread id. Other sibling threads just await the +* completion of microcode update. +*/ +if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + ret =microcode_update_cpu ();
The description says "Gather all cores and serialize the microcode update on them by doing it one-by-one" but it looks like you are doing the update in parallel actually?
Quoted 70 lines
xen/arch/x86/microcode.c +34
-10+/* +* Increase the wait timeout to a safe value here since we're serializing +* the microcode update and that could take a while on a large number of +* CPUs. And that is fine as the *actual* timeout will be determined by +* the last CPU finished updating and thus cut short +*/ +if (wait_for_cpus (&cpu_out,MICROCODE_DEFAULT_TIMEOUT_US * nr_cores) ) +panic ("Timeout when finishing updating microcode" ); - error = info->error; -xfree (info); -return error; +return ret; }int microcode_update (XEN_GUEST_HANDLE_PARAM (const_void) buf,unsigned long len) {int ret; -struct microcode_info *info; +unsigned int cpu;void * buffer;if ( len != (uint32_t )len )@@ -334,28 +372,61 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)if ( ret !=0 ) goto free; +/* cpu_online_map must not change during update */ +if ( !get_cpu_maps () ) + { + ret = -EBUSY ; + goto free; + } +if ( microcode_ops->start_update ) { ret = microcode_ops->start_update ();if ( ret !=0 ) - goto free; + goto put; } ret =parse_microcode_blob (buffer, len);if ( ret <=0 ) {printk (XENLOG_ERR "No valid or newer ucode found. Update abort!\n" ); -xfree (info); -return -EINVAL ; + ret = -EINVAL ; + goto put; } - info->error =0 ; - info->cpu =cpumask_first (&cpu_online_map); +atomic_set (&cpu_in,0 ); +atomic_set (&cpu_out,0 ); + +/* Calculate the number of online CPU core */ + nr_cores =0 ; +for_each_online_cpu (cpu) +if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + nr_cores++; + +printk (XENLOG_INFO "%d cores are to update their microcode\n" , nr_cores); -return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); +/* +* We intend to disable interrupt for long time, which may lead to +* watchdog timeout. +*/
xen/arch/x86/microcode.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 421
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+watchdog_disable (); +/* +* Late loading dance. Why the heavy-handed stop_machine effort? +* +* - HT siblings must be idle and not execute other code while the other +* sibling is loading microcode in order to avoid any negative +* interactions cause by the loading. +* +* - In addition, microcode update on the cores must be serialized until +* this requirement can be relaxed in the future. Right now, this is
As said above, I'm not sure what you are doing here could be considered serialized, the previous method was clearly serialized moving from one CPU to the next one.
Here you are likely updating multiple cores at the same time, which I'm not saying it's wrong, but doesn't seem to match the commit description or the comments in the code.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 7/8] x86/microcode: Synchronize late microcode loading
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Roger Pau Monné <[email protected]> 01/29/19 11:39 AM >>>
xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 339
Quoted 7 lines · Chao Gao
xen/arch/x86/microcode.c +7
-0+/* +* Initiate an update on all processors which don't have an online sibling +* thread with a lower thread id. Other sibling threads just await the +* completion of microcode update. +*/ +if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + ret =microcode_update_cpu ();
Quoted 3 lines
The description says "Gather all cores and serialize the microcode update on them by doing it one-by-one" but it looks like you are doing the update in parallel actually?
As discussed in the context of earlier versions, the apparent parallelism here goes away immediately inside microcode_update_cpu(), which
acquires a global spin lock first thing.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 7/8] x86/microcode: Synchronize late microcode loading
In reply to Roger Pau MonnéQuoted 93 lines · Roger Pau Monné
Quoted 92 lines · Chao Gao
This patch ports microcode improvement patches from linux kernel.
Before you read any further: the early loading method is still the preferred one and you should always do that. The following patch is improving the late loading mechanism for long running jobs and cloud use cases.
Gather all cores and serialize the microcode update on them by doing it one-by-one to make the late update process as reliable as possible and avoid potential issues caused by the microcode update.
Signed-off-by Chao Gao <[email protected]>Tested-by Chao Gao <[email protected]>[linux commit: a5321aec6412b20b5ad15db2d6b916c05349dbff][linux commit: bb8c13d61a629276a162c1d2b1a20a815cbcfbb7]Cc Kevin Tian <[email protected]>Cc Jun Nakajima <[email protected]>Cc Ashok Raj <[email protected]>Cc Borislav Petkov <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Andrew Cooper <[email protected]>Cc Jan Beulich <[email protected]>--- xen/arch/x86/microcode.c | 125 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 98 insertions(+), 27 deletions(-)xen/arch/x86/microcode.c +16
-7@@ -22,6 +22,7 @@*/ #include <xen/cpu.h> +#include <xen/cpumask.h> #include <xen/lib.h> #include <xen/kernel.h> #include <xen/init.h> @@ -30,18 +31,25 @@#include <xen/smp.h> #include <xen/softirq.h> #include <xen/spinlock.h> +#include <xen/stop_machine.h> #include <xen/tasklet.h> #include <xen/guest_access.h> #include <xen/earlycpio.h> +#include <xen/watchdog.h> +#include <asm/delay.h> #include <asm/msr.h> #include <asm/processor.h> #include <asm/setup.h> #include <asm/microcode.h> +/* By default, wait for 30000us */ +#define MICROCODE_DEFAULT_TIMEOUT_US 30000 +static module_t __initdata ucode_mod;static signed int __initdata ucode_mod_idx;static bool_t __initdata ucode_mod_forced; +static unsigned int nr_cores;/* * If we scan the initramfs.cpio for the early microcode code @@ -188,10 +196,11 @@ static DEFINE_SPINLOCK(microcode_mutex); DEFINE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info); -struct microcode_info { -unsigned int cpu; -int error; -}; +/* +* Count the CPUs that have entered and exited the rendezvous +* during late microcode update. +*/ +static atomic_t cpu_in, cpu_out;static void microcode_fini_cpu (unsigned int cpu) {@@ -290,30 +299,60 @@ int microcode_resume_cpu(unsigned int cpu)return microcode_ops ?microcode_update_cpu () :0 ; } -static long do_microcode_update (void *_info) +/* Wait for all CPUs to rendezvous with a timeout (us) */ +static int wait_for_cpus (atomic_t *cnt,unsigned int timeout) { -struct microcode_info *info = _info; -int error; +unsigned int cpus =num_online_cpus ();
Quoted 16 lines
xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 317
Quoted 12 lines
xen/arch/x86/microcode.c +11
-1-BUG_ON (info->cpu !=smp_processor_id ()); +atomic_inc (cnt); + +while (atomic_read (cnt) != cpus ) + { +if ( timeout <=0 ) + { +printk ("CPU%d: Timeout when waiting for CPUs calling in\n" , +smp_processor_id ()); +return -EBUSY ; + } +udelay (1 );udelay will call the rdtsc instruction, is it fine to use it on a sibling thread while there's a microcode update in process on the same core?
I think it is ok. I just checked the kernel code. It uses the ndelay, which in turn calls the rdtsc instruction in some cases.
Ashok, what's your opinion?
Quoted 119 lines
Quoted 19 lines
xen/arch/x86/microcode.c +13
-3+ timeout--; + } + +return 0 ; +} + +static int do_microcode_update (void *unused) +{ +unsigned int cpu =smp_processor_id (); +int ret; - error =microcode_update_cpu (); -if ( error ) - info->error = error; + ret =wait_for_cpus (&cpu_in,MICROCODE_DEFAULT_TIMEOUT_US ); +if ( ret ) +return ret;xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 339
Quoted 10 lines
xen/arch/x86/microcode.c +7
-3- info->cpu =cpumask_next (info->cpu, &cpu_online_map); -if ( info->cpu < nr_cpu_ids ) -return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); +/* +* Initiate an update on all processors which don't have an online sibling +* thread with a lower thread id. Other sibling threads just await the +* completion of microcode update. +*/ +if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + ret =microcode_update_cpu ();The description says "Gather all cores and serialize the microcode update on them by doing it one-by-one" but it looks like you are doing the update in parallel actually?
Quoted 70 lines
xen/arch/x86/microcode.c +34
-10+/* +* Increase the wait timeout to a safe value here since we're serializing +* the microcode update and that could take a while on a large number of +* CPUs. And that is fine as the *actual* timeout will be determined by +* the last CPU finished updating and thus cut short +*/ +if (wait_for_cpus (&cpu_out,MICROCODE_DEFAULT_TIMEOUT_US * nr_cores) ) +panic ("Timeout when finishing updating microcode" ); - error = info->error; -xfree (info); -return error; +return ret; }int microcode_update (XEN_GUEST_HANDLE_PARAM (const_void) buf,unsigned long len) {int ret; -struct microcode_info *info; +unsigned int cpu;void * buffer;if ( len != (uint32_t )len )@@ -334,28 +372,61 @@ int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len)if ( ret !=0 ) goto free; +/* cpu_online_map must not change during update */ +if ( !get_cpu_maps () ) + { + ret = -EBUSY ; + goto free; + } +if ( microcode_ops->start_update ) { ret = microcode_ops->start_update ();if ( ret !=0 ) - goto free; + goto put; } ret =parse_microcode_blob (buffer, len);if ( ret <=0 ) {printk (XENLOG_ERR "No valid or newer ucode found. Update abort!\n" ); -xfree (info); -return -EINVAL ; + ret = -EINVAL ; + goto put; } - info->error =0 ; - info->cpu =cpumask_first (&cpu_online_map); +atomic_set (&cpu_in,0 ); +atomic_set (&cpu_out,0 ); + +/* Calculate the number of online CPU core */ + nr_cores =0 ; +for_each_online_cpu (cpu) +if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + nr_cores++; + +printk (XENLOG_INFO "%d cores are to update their microcode\n" , nr_cores); -return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); +/* +* We intend to disable interrupt for long time, which may lead to +* watchdog timeout. +*/ xen/arch/x86/microcode.c int microcode_update(XEN_GUEST_HANDLE_PARAM(const_void) buf, unsigned long len) · line 421
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+watchdog_disable (); +/* +* Late loading dance. Why the heavy-handed stop_machine effort? +* +* - HT siblings must be idle and not execute other code while the other +* sibling is loading microcode in order to avoid any negative +* interactions cause by the loading. +* +* - In addition, microcode update on the cores must be serialized until +* this requirement can be relaxed in the future. Right now, this is As said above, I'm not sure what you are doing here could be considered serialized, the previous method was clearly serialized moving from one CPU to the next one.
Quoted 3 lines
Here you are likely updating multiple cores at the same time, which I'm not saying it's wrong, but doesn't seem to match the commit description or the comments in the code.
Indeed, it is a little confusing. will explain why it is serialized in the commit message.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 7/8] x86/microcode: Synchronize late microcode loading
In reply to Chao GaoQuoted 9 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode.c +1
-0@@ -30,18 +31,25 @@#include <xen/smp.h> #include <xen/softirq.h> #include <xen/spinlock.h> +#include <xen/stop_machine.h> #include <xen/tasklet.h> #include <xen/guest_access.h> #include <xen/earlycpio.h>
xen/arch/x86/microcode.c line 47
Quoted 10 lines
xen/arch/x86/microcode.c +4
-0+#include <xen/watchdog.h> +#include <asm/delay.h> #include <asm/msr.h> #include <asm/processor.h> #include <asm/setup.h> #include <asm/microcode.h> +/* By default, wait for 30000us */ +#define MICROCODE_DEFAULT_TIMEOUT_US 30000
This value deserves some explanation as to how it was chosen.
xen/arch/x86/microcode.c line 52
Quoted 4 lines
xen/arch/x86/microcode.c +1
-0static module_t __initdata ucode_mod;static signed int __initdata ucode_mod_idx;static bool_t __initdata ucode_mod_forced; +static unsigned int nr_cores;
Could you see about avoiding such a static? You have ...
xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 324
Quoted 1 line
xen/arch/x86/microcode.c +1
-0+static int do_microcode_update (void *unused)
... an "unused" here after all, and it's the (indirect) caller of the function which calculates the value.
Quoted 19 lines
xen/arch/x86/microcode.c +11
-6+{ +unsigned int cpu =smp_processor_id (); +int ret; - error =microcode_update_cpu (); -if ( error ) - info->error = error; + ret =wait_for_cpus (&cpu_in,MICROCODE_DEFAULT_TIMEOUT_US ); +if ( ret ) +return ret; - info->cpu =cpumask_next (info->cpu, &cpu_online_map); -if ( info->cpu < nr_cpu_ids ) -return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); +/* +* Initiate an update on all processors which don't have an online sibling +* thread with a lower thread id. Other sibling threads just await the +* completion of microcode update. +*/
xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 347
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + ret =microcode_update_cpu (); +/* +* Increase the wait timeout to a safe value here since we're serializing +* the microcode update and that could take a while on a large number of +* CPUs. And that is fine as the *actual* timeout will be determined by +* the last CPU finished updating and thus cut short +*/ +if (wait_for_cpus (&cpu_out,MICROCODE_DEFAULT_TIMEOUT_US * nr_cores) ) +panic ("Timeout when finishing updating microcode" );
While I expect this to go away again in the next patch, I'd still like to see this improved, in particular in case the patch here goes in independently of the next one. After all on a system with 100 cores the timeout totals to a whopping 3 seconds.
Generally the time needed to wait scales by the number of CPUs still in need of doing the update. And if a timeout is really to occur, it's perhaps because of one bad core or socket, not because nothing works at all. Hence it would seem both nice and possible to scale the "remaining time to wait" by the (known) number of remaining processors to respond.
Additionally it would be confusing to see dozens of panics in case this actually triggers, because all CPUs would hit this at about the same time.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 7/8] x86/microcode: Synchronize late microcode loading
Quoted 17 lines · Jan Beulich
Quoted 17 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode.c +3
-0@@ -30,18 +31,25 @@#include <xen/smp.h> #include <xen/softirq.h> #include <xen/spinlock.h> +#include <xen/stop_machine.h> #include <xen/tasklet.h> #include <xen/guest_access.h> #include <xen/earlycpio.h> +#include <xen/watchdog.h> +#include <asm/delay.h> #include <asm/msr.h> #include <asm/processor.h> #include <asm/setup.h> #include <asm/microcode.h>
Quoted 4 lines
xen/arch/x86/microcode.c line 47
Quoted 2 lines
xen/arch/x86/microcode.c +2
-0+/* By default, wait for 30000us */ +#define MICROCODE_DEFAULT_TIMEOUT_US 30000This value deserves some explanation as to how it was chosen.
Will do.
Quoted 6 lines
xen/arch/x86/microcode.c line 52
Quoted 5 lines
xen/arch/x86/microcode.c +1
-0static module_t __initdata ucode_mod;static signed int __initdata ucode_mod_idx;static bool_t __initdata ucode_mod_forced; +static unsigned int nr_cores;
Quoted 6 lines
Could you see about avoiding such a static? You have ...
xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 324
Quoted 1 line
xen/arch/x86/microcode.c +1
-0+static int do_microcode_update (void *unused)... an "unused" here after all, and it's the (indirect) caller of the function which calculates the value.
Will do.
Quoted 31 lines
Quoted 20 lines
xen/arch/x86/microcode.c +11
-6+{ +unsigned int cpu =smp_processor_id (); +int ret; - error =microcode_update_cpu (); -if ( error ) - info->error = error; + ret =wait_for_cpus (&cpu_in,MICROCODE_DEFAULT_TIMEOUT_US ); +if ( ret ) +return ret; - info->cpu =cpumask_next (info->cpu, &cpu_online_map); -if ( info->cpu < nr_cpu_ids ) -return continue_hypercall_on_cpu (info->cpu, do_microcode_update, info); +/* +* Initiate an update on all processors which don't have an online sibling +* thread with a lower thread id. Other sibling threads just await the +* completion of microcode update. +*/ xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 347
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + ret =microcode_update_cpu (); +/* +* Increase the wait timeout to a safe value here since we're serializing +* the microcode update and that could take a while on a large number of +* CPUs. And that is fine as the *actual* timeout will be determined by +* the last CPU finished updating and thus cut short +*/ +if (wait_for_cpus (&cpu_out,MICROCODE_DEFAULT_TIMEOUT_US * nr_cores) ) +panic ("Timeout when finishing updating microcode" );
Quoted 4 lines
While I expect this to go away again in the next patch, I'd still like to see this improved, in particular in case the patch here goes in independently of the next one. After all on a system with 100 cores the timeout totals to a whopping 3 seconds.
To be clear, the timeout remains the same in the next patch due to the serial print clause in apply_microcode().
Quoted 7 lines
Generally the time needed to wait scales by the number of CPUs still in need of doing the update. And if a timeout is really to occur, it's perhaps because of one bad core or socket, not because nothing works at all. Hence it would seem both nice and possible to scale the "remaining time to wait" by the (known) number of remaining processors to respond.
Basically, I think the benefit is we can recognize the failure earlier if no core called in in a given interval (i.e. 30ms), and trigger a panic. Considering for such case, even with this optimization, the system needs reboot, which generally takes several minutes, what's the value of this optimization?
Quoted 4 lines
Additionally it would be confusing to see dozens of panics in case this actually triggers, because all CPUs would hit this at about the same time.
Will avoid this confusion.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 7/8] x86/microcode: Synchronize late microcode loading
Quoted 35 lines
Quoted 1 line
Quoted 1 line · On 11.02.19 at 06:40, <[email protected]> wrote:
Quoted 17 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode.c +5
-0+/* +* Initiate an update on all processors which don't have an online sibling +* thread with a lower thread id. Other sibling threads just await the +* completion of microcode update. +*/ xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 347
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + ret =microcode_update_cpu (); +/* +* Increase the wait timeout to a safe value here since we're serializing +* the microcode update and that could take a while on a large number of +* CPUs. And that is fine as the *actual* timeout will be determined by +* the last CPU finished updating and thus cut short +*/ +if (wait_for_cpus (&cpu_out,MICROCODE_DEFAULT_TIMEOUT_US * nr_cores) ) +panic ("Timeout when finishing updating microcode" );Quoted 4 lines
While I expect this to go away again in the next patch, I'd still like to see this improved, in particular in case the patch here goes in independently of the next one. After all on a system with 100 cores the timeout totals to a whopping 3 seconds.
To be clear, the timeout remains the same in the next patch due to the serial print clause in
apply_microcode().Quoted 7 lines
Generally the time needed to wait scales by the number of CPUs still in need of doing the update. And if a timeout is really to occur, it's perhaps because of one bad core or socket, not because nothing works at all. Hence it would seem both nice and possible to scale the "remaining time to wait" by the (known) number of remaining processors to respond.
Quoted 5 lines
Basically, I think the benefit is we can recognize the failure earlier if no core called in in a given interval (i.e. 30ms), and trigger a panic. Considering for such case, even with this optimization, the system needs reboot, which generally takes several minutes, what's the value of this optimization?
Hmm, on one hand this is a fair point you make. Otoh, why do you add any timeout at all, if we say we're hosed anyway if the timeout expires? You could then as well log a message (say once a second) about how many (or which) CPUs still didn't respond. The admin can then still reboot the system if desired.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 7/8] x86/microcode: Synchronize late microcode loading
Quoted 41 lines · Jan Beulich
Quoted 35 lines
Quoted 1 line
Quoted 1 line · On 11.02.19 at 06:40, <[email protected]> wrote:
Quoted 17 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode.c +5
-0+/* +* Initiate an update on all processors which don't have an online sibling +* thread with a lower thread id. Other sibling threads just await the +* completion of microcode update. +*/ xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 347
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + ret =microcode_update_cpu (); +/* +* Increase the wait timeout to a safe value here since we're serializing +* the microcode update and that could take a while on a large number of +* CPUs. And that is fine as the *actual* timeout will be determined by +* the last CPU finished updating and thus cut short +*/ +if (wait_for_cpus (&cpu_out,MICROCODE_DEFAULT_TIMEOUT_US * nr_cores) ) +panic ("Timeout when finishing updating microcode" );Quoted 4 lines
While I expect this to go away again in the next patch, I'd still like to see this improved, in particular in case the patch here goes in independently of the next one. After all on a system with 100 cores the timeout totals to a whopping 3 seconds.
To be clear, the timeout remains the same in the next patch due to the serial print clause in
apply_microcode().Quoted 7 lines
Generally the time needed to wait scales by the number of CPUs still in need of doing the update. And if a timeout is really to occur, it's perhaps because of one bad core or socket, not because nothing works at all. Hence it would seem both nice and possible to scale the "remaining time to wait" by the (known) number of remaining processors to respond.
Quoted 5 lines
Basically, I think the benefit is we can recognize the failure earlier if no core called in in a given interval (i.e. 30ms), and trigger a panic. Considering for such case, even with this optimization, the system needs reboot, which generally takes several minutes, what's the value of this optimization?
Quoted 5 lines
Hmm, on one hand this is a fair point you make. Otoh, why do you add any timeout at all, if we say we're hosed anyway if the timeout expires? You could then as well log a message (say once a second) about how many (or which) CPUs still didn't respond. The admin can then still reboot the system if desired.
That's not a data center friendly approach.
The ability to do microcode update in an online system might by risky, but in case of failure requiring access to the console or power settings of the system isn't nice.
I think doing a panic() after some timeout is a sensible way to handle a failure.
In case you'd like having a way to wait longer: we could allow the "noreboot" parameter to be modified at runtime and do the panic only if opt_noreboot isn't set.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 7/8] x86/microcode: Synchronize late microcode loading
Quoted 48 lines · Juergen Gross
Quoted 41 lines · Jan Beulich
Quoted 35 lines
Quoted 1 line
Quoted 1 line · On 11.02.19 at 06:40, <[email protected]> wrote:
Quoted 17 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode.c +5
-0+/* +* Initiate an update on all processors which don't have an online sibling +* thread with a lower thread id. Other sibling threads just await the +* completion of microcode update. +*/ xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 347
Quoted 10 lines
xen/arch/x86/microcode.c +10
-0+if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + ret =microcode_update_cpu (); +/* +* Increase the wait timeout to a safe value here since we're serializing +* the microcode update and that could take a while on a large number of +* CPUs. And that is fine as the *actual* timeout will be determined by +* the last CPU finished updating and thus cut short +*/ +if (wait_for_cpus (&cpu_out,MICROCODE_DEFAULT_TIMEOUT_US * nr_cores) ) +panic ("Timeout when finishing updating microcode" );Quoted 4 lines
While I expect this to go away again in the next patch, I'd still like to see this improved, in particular in case the patch here goes in independently of the next one. After all on a system with 100 cores the timeout totals to a whopping 3 seconds.
To be clear, the timeout remains the same in the next patch due to the serial print clause in
apply_microcode().Quoted 7 lines
Generally the time needed to wait scales by the number of CPUs still in need of doing the update. And if a timeout is really to occur, it's perhaps because of one bad core or socket, not because nothing works at all. Hence it would seem both nice and possible to scale the "remaining time to wait" by the (known) number of remaining processors to respond.
Quoted 5 lines
Basically, I think the benefit is we can recognize the failure earlier if no core called in in a given interval (i.e. 30ms), and trigger a panic. Considering for such case, even with this optimization, the system needs reboot, which generally takes several minutes, what's the value of this optimization?
Quoted 5 lines
Hmm, on one hand this is a fair point you make. Otoh, why do you add any timeout at all, if we say we're hosed anyway if the timeout expires? You could then as well log a message (say once a second) about how many (or which) CPUs still didn't respond. The admin can then still reboot the system if desired.
Quoted 5 lines
That's not a data center friendly approach.
The ability to do microcode update in an online system might by risky, but in case of failure requiring access to the console or power settings of the system isn't nice.
Thats right... the reason it cannnot be a perfect number is because it really depends on several factors. If the other thread is in a long flow instruction we can only break at instruction boundary, even ipi. Say if you were in wbinvd() for example, or some new ISA taking its sweet time. In the grand scheme of things its less important to fine optimnize this number.
On the other hand not panicking and waiting for sysadmin certainly isn't datacenter friendly as you had pointed out.
Quoted 10 lines
I think doing a
panic()after some timeout is a sensible way to handle a failure.In case you'd like having a way to wait longer: we could allow the "noreboot" parameter to be modified at runtime and do the panic only if
opt_norebootisn't set.Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 7/8] x86/microcode: Synchronize late microcode loading
In reply to Juergen GrossQuoted 52 lines
Quoted 1 line
Quoted 1 line · On 11.02.19 at 14:35, <[email protected]> wrote:
Quoted 5 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 11.02.19 at 06:40, <[email protected]> wrote:
Quoted 3 lines · Jan Beulich
Review
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
(quoted diff) +2
-0+ /* + * Initiate an update on all processors which don't have an onlinesibling
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu)
Quoted 7 lines
xen/arch/x86/microcode.c +7
-0+* thread with a lower thread id. Other sibling threads just await the +* completion of microcode update. +*/ +if ( cpu ==cpumask_first (per_cpu (cpu_sibling_mask, cpu)) ) + ret =microcode_update_cpu (); +/* +* Increase the wait timeout to a safe value here since we're serializing
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
+ * the microcode update and that could take a while on a large number
of
Quoted 28 lines
Quoted 22 lines
Quoted 10 lines
xen/arch/x86/microcode.c int microcode_resume_cpu(unsigned int cpu) · line 347
Quoted 5 lines
xen/arch/x86/microcode.c +5
-0+* CPUs. And that is fine as the *actual* timeout will be determined by +* the last CPU finished updating and thus cut short +*/ +if (wait_for_cpus (&cpu_out,MICROCODE_DEFAULT_TIMEOUT_US * nr_cores) ) +panic ("Timeout when finishing updating microcode" );While I expect this to go away again in the next patch, I'd still like to see this improved, in particular in case the patch here goes in independently of the next one. After all on a system with 100 cores the timeout totals to a whopping 3 seconds.
To be clear, the timeout remains the same in the next patch due to the serial print clause in
apply_microcode().Quoted 7 lines
Generally the time needed to wait scales by the number of CPUs still in need of doing the update. And if a timeout is really to occur, it's perhaps because of one bad core or socket, not because nothing works at all. Hence it would seem both nice and possible to scale the "remaining time to wait" by the (known) number of remaining processors to respond.
Quoted 5 lines
Basically, I think the benefit is we can recognize the failure earlier if no core called in in a given interval (i.e. 30ms), and trigger a panic. Considering for such case, even with this optimization, the system needs reboot, which generally takes several minutes, what's the value of this optimization?
Quoted 5 lines
Hmm, on one hand this is a fair point you make. Otoh, why do you add any timeout at all, if we say we're hosed anyway if the timeout expires? You could then as well log a message (say once a second) about how many (or which) CPUs still didn't respond. The admin can then still reboot the system if desired.
Quoted 5 lines
That's not a data center friendly approach.
The ability to do microcode update in an online system might by risky, but in case of failure requiring access to the console or power settings of the system isn't nice.
I realize this, but I also wouldn't bet the system would reboot cleanly in such a case (i.e. a power cycle may be required anyway).
Quoted 2 lines
I think doing a
panic()after some timeout is a sensible way to handle a failure.
I don't disagree; I'm just not convinced this is the only "sensible" way.
Quoted 3 lines
In case you'd like having a way to wait longer: we could allow the "noreboot" parameter to be modified at runtime and do the panic only if
opt_norebootisn't set.
Why would runtime modification come into play here?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v5 8/8 microcode: update microcode on cores in parallel
In reply to Chao GaoCurrently, microcode_update_lock and microcode_mutex prevent cores from updating microcode in parallel. Below changes are made to support parallel microcode update on cores.
microcode_update_lock is removed. The purpose of this lock is to prevent logic threads of a same core from updating microcode at the same time. But due to using a global lock, it also prevents parallel microcode updating on different cores. The original purpose of microcode_update_lock is already enforced at the level of apply_microcode()'s caller:
1. For late microcode update, only one sibiling thread of a core will call the apply_microcode().
2. For microcode update during system startup or CPU-hotplug, each logical thread is woken up one-by-one.
3. get/put_cpu_bitmaps() prevents the concurrency of CPU-hotplug and late microcode update.
microcode_mutex is replaced by a rwlock. microcode_mutex was used to prevent concurrent accesses to 'uci' and microcode_cache. Now the per-cpu variable, 'uci', won't be accessed by remote cpus after most fields in 'uci' have been removed; The only shared resource which needs to be protected is the microcode_cache. A rwlock allows multiple readers (one thread of each core) to access the global cache and update microcode simultaneously. Because the rwlock may be held in stop_machine context, where interrupt is disabled, irq{save, restore} variants are used to get/release the rwlock.
Note that printk in apply_microcode() and svm_host_osvm_init() (for AMD only) are still processed sequentially.
---
Changes in v5:
- newly add
--- xen/arch/x86/microcode.c | 51 ++++++++++++++++++++++++++++++------------ xen/arch/x86/microcode_amd.c | 9 +------- xen/arch/x86/microcode_intel.c | 9 +------- 3 files changed, 39 insertions(+), 30 deletions(-)
xen/arch/x86/microcode.c +37 -14
@@ -192,7 +192,7 @@ scan:const struct microcode_ops *microcode_ops; -static DEFINE_SPINLOCK (microcode_mutex); +static DEFINE_RWLOCK (cache_rwlock); DEFINE_PER_CPU(struct ucode_cpu_info, ucode_cpu_info);
@@ -206,6 +206,7 @@ static void microcode_fini_cpu(unsigned int cpu) {struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); +ASSERT (cpu ==smp_processor_id ());memset (uci,0 ,sizeof (*uci)); }
@@ -213,21 +214,25 @@ static void microcode_fini_cpu(unsigned int cpu)bool save_patch (struct microcode_patch *new_patch) {struct microcode_patch *microcode_patch; +enum microcode_match_result result =MIS_UCODE ; +bool ret; +unsigned long flag; + +write_lock_irqsave (&cache_rwlock, flag);list_for_each_entry (microcode_patch, µcode_cache, list) { -enum microcode_match_result result = - microcode_ops->replace_patch (new_patch, microcode_patch); + result = microcode_ops->replace_patch (new_patch, microcode_patch);switch ( result ) {case OLD_UCODE : - microcode_ops->free_patch (new_patch); -return false; + ret = false; + goto out;case NEW_UCODE : - microcode_ops->free_patch (microcode_patch); -return true; + ret = true; + goto out;case MIS_UCODE :continue ;
@@ -238,7 +243,27 @@ bool save_patch(struct microcode_patch *new_patch) } }list_add_tail (&new_patch->list, µcode_cache); -return true; + ret = true; + + out: +write_unlock_irqrestore (&cache_rwlock, flag); + +/* free useless patches after interrupt enabled */ +switch ( result ) + { +case OLD_UCODE : + microcode_ops->free_patch (new_patch); +break ; + +case NEW_UCODE : + microcode_ops->free_patch (microcode_patch); +break ; + +default : +break ; + } + +return ret; }/* Find a ucode patch who has newer revision than the one in use */
@@ -269,15 +294,12 @@ static int parse_microcode_blob(const void *buffer, size_t len) {unsigned int cpu =smp_processor_id ();struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu); -int ret; +int ret = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig); -spin_lock (µcode_mutex); - ret = microcode_ops->collect_cpu_info (cpu, &uci->cpu_sig);if (likely (!ret) ) ret = microcode_ops->cpu_request_microcode (cpu, buffer, len);else microcode_fini_cpu (cpu); -spin_unlock (µcode_mutex);return ret; }
@@ -285,10 +307,11 @@ static int parse_microcode_blob(const void *buffer, size_t len)static int microcode_update_cpu (void ) {int ret; +unsigned long flag; -spin_lock (µcode_mutex); +read_lock_irqsave (&cache_rwlock, flag); ret = microcode_ops->apply_microcode (smp_processor_id ()); -spin_unlock (µcode_mutex); +read_unlock_irqrestore (&cache_rwlock, flag);return ret; }
xen/arch/x86/microcode_amd.c +1 -8
@@ -74,9 +74,6 @@ struct mpbhdr {uint8_t data[]; }; -/* serialize access to the physical write */ -static DEFINE_SPINLOCK (microcode_update_lock); -/* See comment in start_update() for cases when this routine fails */ static int collect_cpu_info (unsigned int cpu,struct cpu_signature *csig) {
@@ -258,7 +255,6 @@ static enum microcode_match_result replace_patch(struct microcode_patch *new,static int apply_microcode (unsigned int cpu) { -unsigned long flags;struct ucode_cpu_info *uci = &per_cpu (ucode_cpu_info, cpu);uint32_t rev;struct microcode_header_amd *hdr;
@@ -274,16 +270,13 @@ static int apply_microcode(unsigned int cpu) hdr = patch->data;BUG_ON (!hdr); - -spin_lock_irqsave (µcode_update_lock, flags); +BUG_ON (local_irq_is_enabled ()); hw_err =wrmsr_safe (MSR_AMD_PATCHLOADER , (unsigned long )hdr);/* get patch id after patching */ rdmsrl (MSR_AMD_PATCHLEVEL , rev); -spin_unlock_irqrestore (µcode_update_lock, flags); -/* check current patch id and patch's id for match */ if ( hw_err || (rev != hdr->patch_id) ) {
xen/arch/x86/microcode_intel.c +1 -8
@@ -86,9 +86,6 @@ struct extended_sigtable {#define exttable_size (et) ((et)->count * EXT_SIGNATURE_SIZE + EXT_HEADER_SIZE) -/* serialize access to the physical write to MSR 0x79 */ -static DEFINE_SPINLOCK (microcode_update_lock); -static int collect_cpu_info (unsigned int cpu_num,struct cpu_signature *csig) {struct cpuinfo_x86 *c = &cpu_data[cpu_num];
@@ -297,7 +294,6 @@ static int get_matching_microcode(const void *mc, unsigned int cpu)static int apply_microcode (unsigned int cpu) { -unsigned long flags;uint64_t msr_content;unsigned int val[2 ];unsigned int cpu_num =raw_smp_processor_id ();
@@ -314,9 +310,7 @@ static int apply_microcode(unsigned int cpu) mc_intel = patch->data;BUG_ON (!mc_intel); - -/* serialize access to the physical write to MSR 0x79 */ -spin_lock_irqsave (µcode_update_lock, flags); +BUG_ON (local_irq_is_enabled ());/* write microcode via MSR 0x79 */ wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )mc_intel->bits);
@@ -329,7 +323,6 @@ static int apply_microcode(unsigned int cpu)rdmsrl (MSR_IA32_UCODE_REV , msr_content); val[1 ] = (uint32_t )(msr_content >>32 ); -spin_unlock_irqrestore (µcode_update_lock, flags);if ( val[1 ] != mc_intel->hdr.rev ) { printk(KERN_ERR "microcode: CPU%d update from revision "
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 8/8] microcode: update microcode on cores in parallel
Quoted 3 lines · Chao Gao
Currently,
microcode_update_lockandmicrocode_mutexprevent cores from updating microcode in parallel. Below changes are made to support parallel microcode update on cores.
Oh, that's what I missed from the previous patch then, and what serialises the applying of the microcode update.
Quoted 24 lines
microcode_update_lockis removed. The purpose of this lock is to prevent logic threads of a same core from updating microcode at the same time. But due to using a global lock, it also prevents parallel microcode updating on different cores. The original purpose ofmicrocode_update_lockis already enforced at the level of apply_microcode()'s caller:
1. For late microcode update, only one sibiling thread of a core will call theapply_microcode().
2. For microcode update during system startup or CPU-hotplug, each logical thread is woken up one-by-one.
3. get/put_cpu_bitmaps() prevents the concurrency of CPU-hotplug and late microcode update.
microcode_mutexis replaced by a rwlock.microcode_mutexwas used to prevent concurrent accesses to 'uci' andmicrocode_cache. Now the per-cpu variable, 'uci', won't be accessed by remote cpus after most fields in 'uci' have been removed; The only shared resource which needs to be protected is themicrocode_cache. A rwlock allows multiple readers (one thread of each core) to access the global cache and update microcode simultaneously. Because the rwlock may be held instop_machinecontext, where interrupt is disabled, irq{save, restore} variants are used to get/release the rwlock.
Quoted 4 lines
Note that printk in
apply_microcode()andsvm_host_osvm_init()(for AMD only) are still processed sequentially.Signed-off-by Chao Gao <[email protected]>
Thanks, this LGTM, just one question below.
xen/arch/x86/microcode.c static int parse_microcode_blob(const void *buffer, size_t len) · line 314
Quoted 11 lines
xen/arch/x86/microcode.c +3
-2@@ -285,10 +307,11 @@ static int parse_microcode_blob(const void *buffer, size_t len)static int microcode_update_cpu (void ) {int ret; +unsigned long flag; -spin_lock (µcode_mutex); +read_lock_irqsave (&cache_rwlock, flag); ret = microcode_ops->apply_microcode (smp_processor_id ()); -spin_unlock (µcode_mutex); +read_unlock_irqrestore (&cache_rwlock, flag);
Why do you take the lock here, wouldn't it be better to just take it for find_patch? (ie: like you do for save_patch)
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 8/8] microcode: update microcode on cores in parallel
Quoted 45 lines · Roger Pau Monné
Quoted 3 lines · Chao Gao
Currently,
microcode_update_lockandmicrocode_mutexprevent cores from updating microcode in parallel. Below changes are made to support parallel microcode update on cores.Oh, that's what I missed from the previous patch then, and what serialises the applying of the microcode update.
Quoted 24 lines
microcode_update_lockis removed. The purpose of this lock is to prevent logic threads of a same core from updating microcode at the same time. But due to using a global lock, it also prevents parallel microcode updating on different cores. The original purpose ofmicrocode_update_lockis already enforced at the level of apply_microcode()'s caller:
1. For late microcode update, only one sibiling thread of a core will call theapply_microcode().
2. For microcode update during system startup or CPU-hotplug, each logical thread is woken up one-by-one.
3. get/put_cpu_bitmaps() prevents the concurrency of CPU-hotplug and late microcode update.
microcode_mutexis replaced by a rwlock.microcode_mutexwas used to prevent concurrent accesses to 'uci' andmicrocode_cache. Now the per-cpu variable, 'uci', won't be accessed by remote cpus after most fields in 'uci' have been removed; The only shared resource which needs to be protected is themicrocode_cache. A rwlock allows multiple readers (one thread of each core) to access the global cache and update microcode simultaneously. Because the rwlock may be held instop_machinecontext, where interrupt is disabled, irq{save, restore} variants are used to get/release the rwlock.Quoted 4 lines
Note that printk in
apply_microcode()andsvm_host_osvm_init()(for AMD only) are still processed sequentially.Signed-off-by Chao Gao <[email protected]>Thanks, this LGTM, just one question below.
Quoted 6 lines
xen/arch/x86/microcode.c +1
-0@@ -285,10 +307,11 @@ static int parse_microcode_blob(const void *buffer, size_t len)static int microcode_update_cpu (void ) {int ret; +unsigned long flag;
Quoted 8 lines
xen/arch/x86/microcode.c static int parse_microcode_blob(const void *buffer, size_t len) · line 314
Quoted 5 lines
xen/arch/x86/microcode.c +2
-2-spin_lock (µcode_mutex); +read_lock_irqsave (&cache_rwlock, flag); ret = microcode_ops->apply_microcode (smp_processor_id ()); -spin_unlock (µcode_mutex); +read_unlock_irqrestore (&cache_rwlock, flag);Why do you take the lock here, wouldn't it be better to just take it for
find_patch? (ie: like you do forsave_patch)
Because find_patch() is expected to return a pointer to a ucode patch. If a thread is to load this ucode patch, we should hold the lock to avoid it being freed by another thread.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 8/8] microcode: update microcode on cores in parallel
In reply to Chao Gaoxen/arch/x86/microcode.c static void microcode_fini_cpu(unsigned int cpu) · line 221
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode.c +5
-0@@ -213,21 +214,25 @@ static void microcode_fini_cpu(unsigned int cpu)bool save_patch (struct microcode_patch *new_patch) {struct microcode_patch *microcode_patch; +enum microcode_match_result result =MIS_UCODE ; +bool ret; +unsigned long flag; + +write_lock_irqsave (&cache_rwlock, flag);
Can the new variable please be named "flags", like we do (almost?) everywhere else?
Quoted 23 lines
xen/arch/x86/microcode.c +5
-6list_for_each_entry (microcode_patch, µcode_cache, list) { -enum microcode_match_result result = - microcode_ops->replace_patch (new_patch, microcode_patch); + result = microcode_ops->replace_patch (new_patch, microcode_patch);switch ( result ) {case OLD_UCODE : - microcode_ops->free_patch (new_patch); -return false; + ret = false; + goto out;case NEW_UCODE : - microcode_ops->free_patch (microcode_patch); -return true; + ret = true; + goto out;case MIS_UCODE :continue ;
xen/arch/x86/microcode.c bool save_patch(struct microcode_patch *new_patch) · line 246
Quoted 5 lines
xen/arch/x86/microcode.c +1
-1@@ -238,7 +243,27 @@ bool save_patch(struct microcode_patch *new_patch) } }list_add_tail (&new_patch->list, µcode_cache); -return true; + ret = true;
I don't see the need to update "ret" upwards from here. Afaict all derivation can be done from "result" below here. Which then puts under question the utility of the entire switch() above.
I have to admit that I also dislike the use of "goto" here: While I've learned to accept its use in particular on some error handling paths, I'm unconvinced that this function can't be written without its use.
Quoted 12 lines
xen/arch/x86/microcode_intel.c +1
-3@@ -314,9 +310,7 @@ static int apply_microcode(unsigned int cpu) mc_intel = patch->data;BUG_ON (!mc_intel); - -/* serialize access to the physical write to MSR 0x79 */ -spin_lock_irqsave (µcode_update_lock, flags); +BUG_ON (local_irq_is_enabled ());/* write microcode via MSR 0x79 */ wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )mc_intel->bits);
xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 328
Quoted 7 lines
xen/arch/x86/microcode_intel.c +0
-1@@ -329,7 +323,6 @@ static int apply_microcode(unsigned int cpu)rdmsrl (MSR_IA32_UCODE_REV , msr_content); val[1 ] = (uint32_t )(msr_content >>32 ); -spin_unlock_irqrestore (µcode_update_lock, flags);if ( val[1 ] != mc_intel->hdr.rev ) { printk(KERN_ERR "microcode: CPU%d update from revision "
Am I understanding right that you now rely on upper layers in the call tree to avoid calling into here in parallel for two hyperthreads of the same core? I can't see how you avoid this situation during AP bringup, for example. Did I overlook anything in this regard?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 8/8] microcode: update microcode on cores in parallel
Quoted 16 lines · Jan Beulich
Quoted 16 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +1
-3@@ -314,9 +310,7 @@ static int apply_microcode(unsigned int cpu) mc_intel = patch->data;BUG_ON (!mc_intel); - -/* serialize access to the physical write to MSR 0x79 */ -spin_lock_irqsave (µcode_update_lock, flags); +BUG_ON (local_irq_is_enabled ());/* write microcode via MSR 0x79 */ wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )mc_intel->bits);@@ -329,7 +323,6 @@ static int apply_microcode(unsigned int cpu)rdmsrl (MSR_IA32_UCODE_REV , msr_content); val[1 ] = (uint32_t )(msr_content >>32 );
Quoted 9 lines
xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 328
Quoted 4 lines
xen/arch/x86/microcode_intel.c +0
-1-spin_unlock_irqrestore (µcode_update_lock, flags);if ( val[1 ] != mc_intel->hdr.rev ) { printk(KERN_ERR "microcode: CPU%d update from revision "Am I understanding right that you now rely on upper layers in the call tree to avoid calling into here in parallel for two hyperthreads of the same core? I can't see how you avoid this situation during AP bringup, for example. Did I overlook anything in this regard?
IIRC microcode update is done in the serialized part of AP bringup, before the call to smp_callin, which guarantees serialization.
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 8/8] microcode: update microcode on cores in parallel
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 12.02.19 at 14:25, <[email protected]> wrote:
Quoted 16 lines · Jan Beulich
Quoted 16 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +1
-3@@ -314,9 +310,7 @@ static int apply_microcode(unsigned int cpu) mc_intel = patch->data;BUG_ON (!mc_intel); - -/* serialize access to the physical write to MSR 0x79 */ -spin_lock_irqsave (µcode_update_lock, flags); +BUG_ON (local_irq_is_enabled ());/* write microcode via MSR 0x79 */ wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )mc_intel->bits);@@ -329,7 +323,6 @@ static int apply_microcode(unsigned int cpu)rdmsrl (MSR_IA32_UCODE_REV , msr_content); val[1 ] = (uint32_t )(msr_content >>32 );Quoted 9 lines
xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 328
Quoted 4 lines
xen/arch/x86/microcode_intel.c +0
-1-spin_unlock_irqrestore (µcode_update_lock, flags);if ( val[1 ] != mc_intel->hdr.rev ) { printk(KERN_ERR "microcode: CPU%d update from revision "Am I understanding right that you now rely on upper layers in the call tree to avoid calling into here in parallel for two hyperthreads of the same core? I can't see how you avoid this situation during AP bringup, for example. Did I overlook anything in this regard?
Quoted 2 lines
IIRC microcode update is done in the serialized part of AP bringup, before the call to
smp_callin, which guarantees serialization.
Hmm, yes, right now it is. But I'd call this "happens to be that way" rather than "is guaranteed to be that way" - prior to commit f97838bbd9 it did happen later.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 8/8] microcode: update microcode on cores in parallel
Quoted 28 lines · Jan Beulich
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 12.02.19 at 14:25, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +1
-3@@ -314,9 +310,7 @@ static int apply_microcode(unsigned int cpu) mc_intel = patch->data;BUG_ON (!mc_intel); - -/* serialize access to the physical write to MSR 0x79 */ -spin_lock_irqsave (µcode_update_lock, flags); +BUG_ON (local_irq_is_enabled ());/* write microcode via MSR 0x79 */ wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )mc_intel->bits);xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 328
Quoted 7 lines
xen/arch/x86/microcode_intel.c +0
-1@@ -329,7 +323,6 @@ static int apply_microcode(unsigned int cpu)rdmsrl (MSR_IA32_UCODE_REV , msr_content); val[1 ] = (uint32_t )(msr_content >>32 ); -spin_unlock_irqrestore (µcode_update_lock, flags);if ( val[1 ] != mc_intel->hdr.rev ) { printk(KERN_ERR "microcode: CPU%d update from revision "Am I understanding right that you now rely on upper layers in the call tree to avoid calling into here in parallel for two hyperthreads of the same core? I can't see how you avoid this situation during AP bringup, for example. Did I overlook anything in this regard?
Quoted 6 lines
Quoted 2 lines
IIRC microcode update is done in the serialized part of AP bringup, before the call to
smp_callin, which guarantees serialization.Hmm, yes, right now it is. But I'd call this "happens to be that way" rather than "is guaranteed to be that way" - prior to commit
f97838bbd9it did happen later.
How about employing another lock, "early_ucode_update_lock", to guarantee serialization.
In particular, early_microcode_update_cpu() and microcode_resume_cpu() will acquire this lock before ucode update.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 8/8] microcode: update microcode on cores in parallel
Quoted 37 lines
Quoted 1 line
Quoted 1 line · On 13.02.19 at 03:30, <[email protected]> wrote:
Quoted 28 lines · Jan Beulich
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 12.02.19 at 14:25, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +1
-3@@ -314,9 +310,7 @@ static int apply_microcode(unsigned int cpu) mc_intel = patch->data;BUG_ON (!mc_intel); - -/* serialize access to the physical write to MSR 0x79 */ -spin_lock_irqsave (µcode_update_lock, flags); +BUG_ON (local_irq_is_enabled ());/* write microcode via MSR 0x79 */ wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )mc_intel->bits);xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 328
Quoted 7 lines
xen/arch/x86/microcode_intel.c +0
-1@@ -329,7 +323,6 @@ static int apply_microcode(unsigned int cpu)rdmsrl (MSR_IA32_UCODE_REV , msr_content); val[1 ] = (uint32_t )(msr_content >>32 ); -spin_unlock_irqrestore (µcode_update_lock, flags);if ( val[1 ] != mc_intel->hdr.rev ) { printk(KERN_ERR "microcode: CPU%d update from revision "Am I understanding right that you now rely on upper layers in the call tree to avoid calling into here in parallel for two hyperthreads of the same core? I can't see how you avoid this situation during AP bringup, for example. Did I overlook anything in this regard?
Quoted 6 lines
Quoted 2 lines
IIRC microcode update is done in the serialized part of AP bringup, before the call to
smp_callin, which guarantees serialization.Hmm, yes, right now it is. But I'd call this "happens to be that way" rather than "is guaranteed to be that way" - prior to commit
f97838bbd9it did happen later.
Quoted 5 lines
How about employing another lock, "
early_ucode_update_lock", to guarantee serialization.In particular,
early_microcode_update_cpu()andmicrocode_resume_cpu()will acquire this lock before ucode update.
That's a (temporary) option, but would over-serialize things again. I was rather trying to hint towards a per-core lock.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 8/8] microcode: update microcode on cores in parallel
Quoted 37 lines · Jan Beulich
Quoted 37 lines
Quoted 1 line
Quoted 1 line · On 13.02.19 at 03:30, <[email protected]> wrote:
Quoted 34 lines · Jan Beulich
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 12.02.19 at 14:25, <[email protected]> wrote:
Quoted 16 lines · Jan Beulich
Quoted 16 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
xen/arch/x86/microcode_intel.c +1
-3@@ -314,9 +310,7 @@ static int apply_microcode(unsigned int cpu) mc_intel = patch->data;BUG_ON (!mc_intel); - -/* serialize access to the physical write to MSR 0x79 */ -spin_lock_irqsave (µcode_update_lock, flags); +BUG_ON (local_irq_is_enabled ());/* write microcode via MSR 0x79 */ wrmsrl (MSR_IA32_UCODE_WRITE , (unsigned long )mc_intel->bits);@@ -329,7 +323,6 @@ static int apply_microcode(unsigned int cpu)rdmsrl (MSR_IA32_UCODE_REV , msr_content); val[1 ] = (uint32_t )(msr_content >>32 );Quoted 9 lines
xen/arch/x86/microcode_intel.c static int apply_microcode(unsigned int cpu) · line 328
Quoted 4 lines
xen/arch/x86/microcode_intel.c +0
-1-spin_unlock_irqrestore (µcode_update_lock, flags);if ( val[1 ] != mc_intel->hdr.rev ) { printk(KERN_ERR "microcode: CPU%d update from revision "Am I understanding right that you now rely on upper layers in the call tree to avoid calling into here in parallel for two hyperthreads of the same core? I can't see how you avoid this situation during AP bringup, for example. Did I overlook anything in this regard?
Quoted 2 lines
IIRC microcode update is done in the serialized part of AP bringup, before the call to
smp_callin, which guarantees serialization.Hmm, yes, right now it is. But I'd call this "happens to be that way" rather than "is guaranteed to be that way" - prior to commit
f97838bbd9it did happen later.
Quoted 8 lines
Quoted 5 lines
How about employing another lock, "
early_ucode_update_lock", to guarantee serialization.In particular,
early_microcode_update_cpu()andmicrocode_resume_cpu()will acquire this lock before ucode update.That's a (temporary) option, but would over-serialize things again. I was rather trying to hint towards a per-core lock.
To implement a per-core lock, a core should be conscious of its siblings. Unfortunately, during AP bringup, ucode update is done prior to the initialization of 'cpu_sibling_mask'.
Thanks
Chao
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH v5 8/8] microcode: update microcode on cores in parallel
Quoted 45 lines
Quoted 1 line
Quoted 1 line · On 13.02.19 at 09:50, <[email protected]> wrote:
Quoted 43 lines · Jan Beulich
Quoted 42 lines
Quoted 1 line
Quoted 1 line · On 13.02.19 at 03:30, <[email protected]> wrote:
Quoted 28 lines · Jan Beulich
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 12.02.19 at 14:25, <[email protected]> wrote:
Quoted 25 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 28.01.19 at 08:06, <[email protected]> wrote:
(quoted diff) +1
-3@@ -314,9 +310,7 @@ static int apply_microcode(unsigned int cpu) mc_intel = patch->data; BUG_ON(!mc_intel); - - /* serialize access to the physical write to MSR 0x79 */ - spin_lock_irqsave(µcode_update_lock, flags); + BUG_ON(local_irq_is_enabled()); /* write microcode via MSR 0x79 */ wrmsrl(MSR_IA32_UCODE_WRITE, (unsigned long)mc_intel->bits);Review static int apply_microcode(unsigned int cpu) · line 328
Quoted 7 lines
(quoted diff) +0
-1@@ -329,7 +323,6 @@ static int apply_microcode(unsigned int cpu) rdmsrl(MSR_IA32_UCODE_REV, msr_content); val[1] = (uint32_t)(msr_content >> 32); - spin_unlock_irqrestore(µcode_update_lock, flags); if ( val[1] != mc_intel->hdr.rev ) { printk(KERN_ERR "microcode: CPU%d update from revision "Am I understanding right that you now rely on upper layers in the call tree to avoid calling into here in parallel for two hyperthreads of the same core? I can't see how you avoid this situation during AP bringup, for example. Did I overlook anything in this regard?
Quoted 6 lines
Quoted 2 lines
IIRC microcode update is done in the serialized part of AP bringup, before the call to
smp_callin, which guarantees serialization.Hmm, yes, right now it is. But I'd call this "happens to be that way" rather than "is guaranteed to be that way" - prior to commit
f97838bbd9it did happen later.How about employing another lock, "
early_ucode_update_lock", to guarantee serialization.In particular,
early_microcode_update_cpu()andmicrocode_resume_cpu()will acquire this lock before ucode update.
Quoted 6 lines
Quoted 2 lines
That's a (temporary) option, but would over-serialize things again. I was rather trying to hint towards a per-core lock.
To implement a per-core lock, a core should be conscious of its siblings. Unfortunately, during AP bringup, ucode update is done prior to the initialization of '
cpu_sibling_mask'.
Okay, in this case a global lock will perhaps do for now (as boot time AP bringup is serialized in this regard already anyway). I do think though that the sibling maps should be set up earlier, but this doesn't look to be as simple as moving ahead the call to set_cpu_sibling_map(). I wonder anyway why all sorts of stuff have accumulated ahead of smp_callin(), thus delaying the boot process in particular on large systems. Clearly something taking as long as microcode load should, if at all possible, happen afterwards.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 26 lines · Chao Gao
Changes in this version:
- support parallel microcode updates for all cores (see patch 8)
- Address Roger's comments on the last version.The intention of this series is to make the late microcode loading more reliable by rendezvousing all cpus in
stop_machinecontext. This idea comes from Ashok. I am porting his linux patch to Xen (see patch 7 for more details).This series makes three changes:
1. Patch 1-6: introduce a global microcode cache
2. Patch 7: synchronize late microcode loading
3. Patch 8: support parallel microcodes update on different coresCurrently, late microcode loading does a lot of things including parsing microcode blob, checking the signature/revision and performing update. Putting all of them into
stop_machinecontext is a bad idea because of complexity (One issue I observed is memory allocation triggered one assertion instop_machinecontext). In order to simplify the load process, I move parsing microcode out of the load process. The microcode blob is parsed and a global microcode cache is built on a single CPU before rendezvousing all cpus to update microcode. Other CPUs just get and load a suitable microcode from the global cache. With this global cache, it is safe to put simplified load process tostop_machinecontext.
Quoted 8 lines
Regarding changes to AMD side, I didn't do any test for them due to lack of hardware. Could you help to test this series on an AMD machine? At least, two basic tests are needed:
* do a microcode update after system bootup
* don't bring all pCPUs up at bootup by specifying maxcpus option in xen command line and then do a microcode update and online all offlined CPUs via 'xen-hptool'.
Thanks for the series, I think it's a good improvement to current microcode loading.
I would like to ask how have you tested the series, I don't seem to find any tool in the current tree to load a microcode to Xen. The only thing I've found is:
https://lists.xen.org/archives/html/xen-devel/2013-07/txtpyXvYZGRwb.txt
Have you used this tool to test the code?
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 39 lines · Roger Pau Monné
Quoted 26 lines · Chao Gao
Changes in this version:
- support parallel microcode updates for all cores (see patch 8)
- Address Roger's comments on the last version.The intention of this series is to make the late microcode loading more reliable by rendezvousing all cpus in
stop_machinecontext. This idea comes from Ashok. I am porting his linux patch to Xen (see patch 7 for more details).This series makes three changes:
1. Patch 1-6: introduce a global microcode cache
2. Patch 7: synchronize late microcode loading
3. Patch 8: support parallel microcodes update on different coresCurrently, late microcode loading does a lot of things including parsing microcode blob, checking the signature/revision and performing update. Putting all of them into
stop_machinecontext is a bad idea because of complexity (One issue I observed is memory allocation triggered one assertion instop_machinecontext). In order to simplify the load process, I move parsing microcode out of the load process. The microcode blob is parsed and a global microcode cache is built on a single CPU before rendezvousing all cpus to update microcode. Other CPUs just get and load a suitable microcode from the global cache. With this global cache, it is safe to put simplified load process tostop_machinecontext.Quoted 8 lines
Regarding changes to AMD side, I didn't do any test for them due to lack of hardware. Could you help to test this series on an AMD machine? At least, two basic tests are needed:
* do a microcode update after system bootup
* don't bring all pCPUs up at bootup by specifying maxcpus option in xen command line and then do a microcode update and online all offlined CPUs via 'xen-hptool'.Thanks for the series, I think it's a good improvement to current microcode loading.
Quoted 7 lines
I would like to ask how have you tested the series, I don't seem to find any tool in the current tree to load a microcode to Xen. The only thing I've found is:
https://lists.xen.org/archives/html/xen-devel/2013-07/txtpyXvYZGRwb.txt
Have you used this tool to test the code?
Yes. I am using this patch with some issues fixed.
Thanks
Chao
Quoted 1 line
From
ff00662f3e35f661a06dbf143150f300664d5e93Mon Sep 17 00:00:00 2001
From: Chao Gao <[email protected]>
Date: Fri, 9 Mar 2018 20:01:53 +0800
Subject: [PATCH 1/9] misc/xenmicrocode: Upload /lib/firmware/<some blob> to
the hypervisor
--- tools/libxc/include/xenctrl.h | 1 + tools/libxc/xc_misc.c | 19 +++++++++++ tools/misc/Makefile | 4 +++ tools/misc/xenmicrocode.c | 73 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 97 insertions(+) create mode 100644 tools/misc/xenmicrocode.c
tools/libxc/include/xenctrl.h +1 -0
@@ -1245,6 +1245,7 @@ typedef uint32_t xc_node_to_node_dist_t;int xc_physinfo (xc_interface *xch,xc_physinfo_t *info);int xc_cputopoinfo (xc_interface *xch,unsigned *max_cpus,xc_cputopo_t *cputopo); +int xc_platform_op (xc_interface *xch,struct xen_platform_op *op);int xc_numainfo (xc_interface *xch,unsigned *max_nodes,xc_meminfo_t *meminfo,uint32_t *distance); int xc_pcitopoinfo(xc_interface *xch, unsigned num_devs,
tools/libxc/xc_misc.c +19 -0
@@ -226,6 +226,25 @@ int xc_physinfo(xc_interface *xch,return 0 ; } +int xc_platform_op (xc_interface *xch,struct xen_platform_op *op) +{ +int ret =0 ; +DECLARE_PLATFORM_OP ; +DECLARE_HYPERCALL_BOUNCE (op,sizeof (*op),XC_HYPERCALL_BUFFER_BOUNCE_BOTH ); + +if (xc_hypercall_bounce_pre (xch, op) ) + { +PERROR ("Could not bounce xen_platform_op memory buffer" ); +return -1 ; + } + op->interface_version =XENPF_INTERFACE_VERSION ; + + platform_op = *op; + ret =do_platform_op (xch, &platform_op); +xc_hypercall_bounce_post (xch, op); +return ret; +} +int xc_cputopoinfo (xc_interface *xch,unsigned *max_cpus,xc_cputopo_t *cputopo) {
tools/misc/Makefile +4 -0
@@ -23,6 +23,7 @@ INSTALL_SBIN-$(CONFIG_X86) += xen-hvmcrash INSTALL_SBIN-$(CONFIG_X86) += xen-hvmctx INSTALL_SBIN-$(CONFIG_X86) += xen-lowmemd INSTALL_SBIN-$(CONFIG_X86) += xen-mfndump +INSTALL_SBIN-$(CONFIG_X86) += xenmicrocode INSTALL_SBIN += xen-ringwatch INSTALL_SBIN += xen-tmem-list-parse INSTALL_SBIN += xencov
@@ -118,4 +119,7 @@ xen-lowmemd: xen-lowmemd.o xencov: xencov.o $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) +xenmicrocode: xenmicrocode.o + $(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS) + -include $(DEPS_INCLUDE)
tools/misc/xenmicrocode.c new file +73 -0
73 lines not on this page · raw message
Signature · 1 line
1.8.3.1
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 44 lines · Chao Gao
Quoted 43 lines · Roger Pau Monné
Quoted 26 lines · Chao Gao
Changes in this version:
- support parallel microcode updates for all cores (see patch 8)
- Address Roger's comments on the last version.The intention of this series is to make the late microcode loading more reliable by rendezvousing all cpus in
stop_machinecontext. This idea comes from Ashok. I am porting his linux patch to Xen (see patch 7 for more details).This series makes three changes:
1. Patch 1-6: introduce a global microcode cache
2. Patch 7: synchronize late microcode loading
3. Patch 8: support parallel microcodes update on different coresCurrently, late microcode loading does a lot of things including parsing microcode blob, checking the signature/revision and performing update. Putting all of them into
stop_machinecontext is a bad idea because of complexity (One issue I observed is memory allocation triggered one assertion instop_machinecontext). In order to simplify the load process, I move parsing microcode out of the load process. The microcode blob is parsed and a global microcode cache is built on a single CPU before rendezvousing all cpus to update microcode. Other CPUs just get and load a suitable microcode from the global cache. With this global cache, it is safe to put simplified load process tostop_machinecontext.Quoted 8 lines
Regarding changes to AMD side, I didn't do any test for them due to lack of hardware. Could you help to test this series on an AMD machine? At least, two basic tests are needed:
* do a microcode update after system bootup
* don't bring all pCPUs up at bootup by specifying maxcpus option in xen command line and then do a microcode update and online all offlined CPUs via 'xen-hptool'.Thanks for the series, I think it's a good improvement to current microcode loading.
I would like to ask how have you tested the series, I don't seem to find any tool in the current tree to load a microcode to Xen. The only thing I've found is:
Quoted 5 lines
Quoted 3 lines
https://lists.xen.org/archives/html/xen-devel/2013-07/txtpyXvYZGRwb.txt
Have you used this tool to test the code?
Yes. I am using this patch with some issues fixed.
Could you please include it in the next version of the series?
We really need this tool in-tree IMO.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel