x86/microcode: support for microcode update in Xen dom0
Subsystems (3)
cover x86/microcode: support for microcode update in Xen dom0
From: Jeremy Fitzhardinge <[email protected]>
Hi all,
I'm proposing this for 2.6.39.
This series adds a new "Xen" microcode update type, in addition to Intel and AMD.
The Xen hypervisor is responsible for performing the actual microcode update (since only it knows what physical CPUs are in the system and has sufficient privilege to access them), but it requires the dom0 kernel to provide the actual microcode update data.
Xen update mechanism is uniform independent of the CPU type, but the driver must know where to find the data file, which depends on the CPU type. And since the update hypercall updates all CPUs, we only need to execute it once on any CPU - but for simplicity it just runs it only on (V)CPU 0.
Thanks,
J
Jeremy Fitzhardinge (1):
xen: add CPU microcode update driver
Stephen Tweedie (1):
xen dom0: Add support for the platform_ops hypercall
arch/x86/include/asm/microcode.h | 9 ++ arch/x86/include/asm/xen/hypercall.h | 8 ++ arch/x86/kernel/Makefile | 1 + arch/x86/kernel/microcode_core.c | 5 +- arch/x86/kernel/microcode_xen.c | 198 ++++++++++++++++++++++++++++++ arch/x86/xen/Kconfig | 4 + include/xen/interface/platform.h | 222 ++++++++++++++++++++++++++++++++++ include/xen/interface/xen.h | 2 + 8 files changed, 448 insertions(+), 1 deletions(-) create mode 100644 arch/x86/kernel/microcode_xen.c create mode 100644 include/xen/interface/platform.h
Signature · 1 line
1.7.3.5
PATCH 1/2 xen dom0: Add support for the platform_ops hypercall
From: Stephen Tweedie <[email protected]>
Minimal changes to get platform ops (renamed dom0_ops on pv_ops) working on pv_ops builds. Pulls in upstream linux-2.6.18-xen.hg's platform.h
[ Impact: add Xen hypercall definitions ]
--- arch/x86/include/asm/xen/hypercall.h | 8 ++ include/xen/interface/platform.h | 222 ++++++++++++++++++++++++++++++++++ include/xen/interface/xen.h | 2 + 3 files changed, 232 insertions(+), 0 deletions(-) create mode 100644 include/xen/interface/platform.h
arch/x86/include/asm/xen/hypercall.h +8 -0
@@ -45,6 +45,7 @@#include <xen/interface/xen.h> #include <xen/interface/sched.h> #include <xen/interface/physdev.h> +#include <xen/interface/platform.h> /* * The hypercall asms have to meet several constraints:
@@ -299,6 +300,13 @@ HYPERVISOR_set_timer_op(u64 timeout) }static inline int +HYPERVISOR_dom0_op (struct xen_platform_op *platform_op) +{ + platform_op->interface_version =XENPF_INTERFACE_VERSION ; +return _hypercall1 (int, dom0_op, platform_op); +} + +static inline int HYPERVISOR_set_debugreg (int reg,unsigned long value) {return _hypercall2 (int, set_debugreg, reg, value);
include/xen/interface/platform.h new file +222 -0
222 lines not on this page · raw message
include/xen/interface/xen.h +2 -0
@@ -461,6 +461,8 @@ typedef uint8_t xen_domain_handle_t[16];#define __mk_unsigned_long (x) x ## UL#define mk_unsigned_long (x) __mk_unsigned_long(x) +DEFINE_GUEST_HANDLE (uint64_t); +#else /* __ASSEMBLY__ */ /* In assembly code we cannot use C numeric constant suffixes. */
Signature · 1 line
1.7.3.5
PATCH 2/2 xen: add CPU microcode update driver
In reply to Jeremy FitzhardingeFrom: Jeremy Fitzhardinge <[email protected]>
Xen does all the hard work for us, including choosing the right update method for this cpu type and actually doing it for all cpus. We just need to supply it with the firmware blob.
Because Xen updates all CPUs (and the kernel's virtual cpu numbers have no fixed relationship with the underlying physical cpus), we only bother doing anything for cpu "0".
--- arch/x86/include/asm/microcode.h | 9 ++ arch/x86/kernel/Makefile | 1 + arch/x86/kernel/microcode_core.c | 5 +- arch/x86/kernel/microcode_xen.c | 198 ++++++++++++++++++++++++++++++++++++++ arch/x86/xen/Kconfig | 4 + 5 files changed, 216 insertions(+), 1 deletions(-) create mode 100644 arch/x86/kernel/microcode_xen.c
arch/x86/include/asm/microcode.h +9 -0
@@ -61,4 +61,13 @@ static inline struct microcode_ops * __init init_amd_microcode(void) }#endif +#ifdef CONFIG_MICROCODE_XEN +extern struct microcode_ops *__init init_xen_microcode (void ); +#else +static inline struct microcode_ops *__init init_xen_microcode (void ) +{ +return NULL ; +} +#endif +#endif /* _ASM_X86_MICROCODE_H */
arch/x86/kernel/Makefile +1 -0
@@ -104,6 +104,7 @@ obj-$(CONFIG_PCSPKR_PLATFORM) += pcspeaker.o microcode-y := microcode_core.o microcode-$(CONFIG_MICROCODE_INTEL) += microcode_intel.o microcode-$(CONFIG_MICROCODE_AMD) += microcode_amd.o +microcode-$(CONFIG_MICROCODE_XEN) += microcode_xen.o obj-$(CONFIG_MICROCODE) += microcode.o obj-$(CONFIG_X86_CHECK_BIOS_CORRUPTION) += check.o
arch/x86/kernel/microcode_core.c +4 -1
@@ -83,6 +83,7 @@#include <linux/fs.h> #include <linux/mm.h> +#include <xen/xen.h> #include <asm/microcode.h> #include <asm/processor.h>
@@ -506,7 +507,9 @@ static int __init microcode_init(void)struct cpuinfo_x86 *c = &cpu_data (0 );int error; -if (c->x86_vendor ==X86_VENDOR_INTEL ) +if (xen_pv_domain ()) + microcode_ops =init_xen_microcode (); +else if (c->x86_vendor ==X86_VENDOR_INTEL ) microcode_ops =init_intel_microcode ();else if (c->x86_vendor ==X86_VENDOR_AMD ) microcode_ops =init_amd_microcode ();
arch/x86/kernel/microcode_xen.c new file +198 -0
198 lines not on this page · raw message
arch/x86/xen/Kconfig +4 -0
@@ -48,3 +48,7 @@ config XEN_DEBUG_FS help Enable statistics output and various tuning options in debugfs. Enabling this option may incur a significant performance overhead. + +config MICROCODE_XEN + def_bool y + depends on XEN_DOM0 && MICROCODE
Signature · 1 line
1.7.3.5
Borislav Petkov
maintainerQuoted 14 lines · Jeremy Fitzhardinge
From: Jeremy Fitzhardinge <[email protected]>
Hi all,
I'm proposing this for 2.6.39.
This series adds a new "Xen" microcode update type, in addition to Intel and AMD.
The Xen hypervisor is responsible for performing the actual microcode update (since only it knows what physical CPUs are in the system and has sufficient privilege to access them), but it requires the dom0 kernel to provide the actual microcode update data.
Quoted 5 lines
Xen update mechanism is uniform independent of the CPU type, but the driver must know where to find the data file, which depends on the CPU type. And since the update hypercall updates all CPUs, we only need to execute it once on any CPU - but for simplicity it just runs it only on (V)CPU 0.
I don't like this for several reasons:
- ucode should be loaded as early as possible and the perfect place for that should be the hypervisor and not the dom0 kernel. But I'm not that familiar with xen design and I guess it would be pretty hard to do. (Btw, x86 bare metal could also try to improve the situation there but that's also hard due to the design of the firmware loader (needs userspace)).
- you're adding a microcode_xen.c as if this is another hw vendor and you're figuring out which is the proper firmware image based on the vendor, then you load it and then do the hypercall. This is duplicating code and inviting future bugs. Everytime the hw vendors decide to change something to their fw loading mechanism, xen needs to be updated. Also, you don't do any sanity checks to the ucode image as the vendor code does which is inacceptable.
What it should do instead, is call into the hw vendor-specific ucode loading mechanism and do all the image loading/verification there. The only thing you'd need to supply is a xen-specific ->apply_microcode() doing the hypercall _after_ the ucode image has been verified and is good to go. I'm guessing the XENPF_microcode_update does call into the hypervisor and calls a xen-specific ucode update mechanism based on the vendor instead of using the vendor-supplied code...
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 20 lines · Borislav Petkov
Quoted 14 lines · Jeremy Fitzhardinge
From: Jeremy Fitzhardinge <[email protected]>
Hi all,
I'm proposing this for 2.6.39.
This series adds a new "Xen" microcode update type, in addition to Intel and AMD.
The Xen hypervisor is responsible for performing the actual microcode update (since only it knows what physical CPUs are in the system and has sufficient privilege to access them), but it requires the dom0 kernel to provide the actual microcode update data.
Quoted 5 lines
Xen update mechanism is uniform independent of the CPU type, but the driver must know where to find the data file, which depends on the CPU type. And since the update hypercall updates all CPUs, we only need to execute it once on any CPU - but for simplicity it just runs it only on (V)CPU 0.
Quoted 8 lines
I don't like this for several reasons:
- ucode should be loaded as early as possible and the perfect place for that should be the hypervisor and not the dom0 kernel. But I'm not that familiar with xen design and I guess it would be pretty hard to do. (Btw, x86 bare metal could also try to improve the situation there but that's also hard due to the design of the firmware loader (needs userspace)).
Yes, its the same issue with Xen or without. The Xen hypervisor has no devices, storage or anything else with which it can get microcode data. It depends on the dom0 kernel getting it from somewhere and supplying it to the hypervisor. In practice this is no different from relying on usermode to get the firmware update.
In general firmware updates are not critical and not required very early, so "as soon as reasonably possible" is OK. (If the firmware update is critical, it should be supplied as a BIOS update.)
So I think this is moot with respect to this particular patch.
Quoted 7 lines
- you're adding a
microcode_xen.cas if this is another hw vendor and you're figuring out which is the proper firmware image based on the vendor, then you load it and then do the hypercall. This is duplicating code and inviting future bugs. Everytime the hw vendors decide to change something to their fw loading mechanism, xen needs to be updated. Also, you don't do any sanity checks to the ucode image as the vendor code does which is inacceptable.
The code within the hypervisor is more or less the same as the kernel's:
it does all the required vendor-specific checks on the firmware and loads it on all the CPUs with the appropriate mechanism. The hypercall ABI is vendor-agnostic, but it assumes that dom0 will supply suitable microcode for the current CPU vendor (though if it doesn't, the image will presumably be rejected).
So from that perspective, it is a distinct microcode loading mechanism, akin to a vendor-specific one. The awkward part is getting the filename to actually request from usermode, which is Intel/AMD/whatever specific, which results in duplicated code to generate that pathname.
Quoted 7 lines
What it should do instead, is call into the hw vendor-specific ucode loading mechanism and do all the image loading/verification there. The only thing you'd need to supply is a xen-specific ->apply_microcode() doing the hypercall _after_ the ucode image has been verified and is good to go. I'm guessing the
XENPF_microcode_updatedoes call into the hypervisor and calls a xen-specific ucode update mechanism based on the vendor instead of using the vendor-supplied code...
Well, I was trying to avoid putting Xen-specific code into the existing Intel/AMD loaders. That doesn't seem any cleaner.
I could export "my firmware pathname" functions from them and have the Xen driver call those, rather than duplicating the pathname construction code. Would that help address your concerns?
J
Quoted 20 lines · Borislav Petkov
Quoted 14 lines · Jeremy Fitzhardinge
From: Jeremy Fitzhardinge <[email protected]>
Hi all,
I'm proposing this for 2.6.39.
This series adds a new "Xen" microcode update type, in addition to Intel and AMD.
The Xen hypervisor is responsible for performing the actual microcode update (since only it knows what physical CPUs are in the system and has sufficient privilege to access them), but it requires the dom0 kernel to provide the actual microcode update data.
Quoted 5 lines
Xen update mechanism is uniform independent of the CPU type, but the driver must know where to find the data file, which depends on the CPU type. And since the update hypercall updates all CPUs, we only need to execute it once on any CPU - but for simplicity it just runs it only on (V)CPU 0.
Quoted 8 lines
I don't like this for several reasons:
- ucode should be loaded as early as possible and the perfect place for that should be the hypervisor and not the dom0 kernel. But I'm not that familiar with xen design and I guess it would be pretty hard to do. (Btw, x86 bare metal could also try to improve the situation there but that's also hard due to the design of the firmware loader (needs userspace)).
Yes, its the same issue with Xen or without. The Xen hypervisor has no devices, storage or anything else with which it can get microcode data. It depends on the dom0 kernel getting it from somewhere and supplying it to the hypervisor. In practice this is no different from relying on usermode to get the firmware update.
In general firmware updates are not critical and not required very early, so "as soon as reasonably possible" is OK. (If the firmware update is critical, it should be supplied as a BIOS update.)
So I think this is moot with respect to this particular patch.
Quoted 7 lines
- you're adding a
microcode_xen.cas if this is another hw vendor and you're figuring out which is the proper firmware image based on the vendor, then you load it and then do the hypercall. This is duplicating code and inviting future bugs. Everytime the hw vendors decide to change something to their fw loading mechanism, xen needs to be updated. Also, you don't do any sanity checks to the ucode image as the vendor code does which is inacceptable.
The code within the hypervisor is more or less the same as the kernel's:
it does all the required vendor-specific checks on the firmware and loads it on all the CPUs with the appropriate mechanism. The hypercall ABI is vendor-agnostic, but it assumes that dom0 will supply suitable microcode for the current CPU vendor (though if it doesn't, the image will presumably be rejected).
So from that perspective, it is a distinct microcode loading mechanism, akin to a vendor-specific one. The awkward part is getting the filename to actually request from usermode, which is Intel/AMD/whatever specific, which results in duplicated code to generate that pathname.
Quoted 7 lines
What it should do instead, is call into the hw vendor-specific ucode loading mechanism and do all the image loading/verification there. The only thing you'd need to supply is a xen-specific ->apply_microcode() doing the hypercall _after_ the ucode image has been verified and is good to go. I'm guessing the
XENPF_microcode_updatedoes call into the hypervisor and calls a xen-specific ucode update mechanism based on the vendor instead of using the vendor-supplied code...
Well, I was trying to avoid putting Xen-specific code into the existing Intel/AMD loaders. That doesn't seem any cleaner.
I could export "my firmware pathname" functions from them and have the Xen driver call those, rather than duplicating the pathname construction code. Would that help address your concerns?
J
Borislav Petkov
maintainerQuoted 7 lines · Jeremy Fitzhardinge
Quoted 6 lines
- ucode should be loaded as early as possible and the perfect place for that should be the hypervisor and not the dom0 kernel. But I'm not that familiar with xen design and I guess it would be pretty hard to do. (Btw, x86 bare metal could also try to improve the situation there but that's also hard due to the design of the firmware loader (needs userspace)).
Quoted 8 lines
Yes, its the same issue with Xen or without. The Xen hypervisor has no devices, storage or anything else with which it can get microcode data. It depends on the dom0 kernel getting it from somewhere and supplying it to the hypervisor. In practice this is no different from relying on usermode to get the firmware update.
In general firmware updates are not critical and not required very early, so "as soon as reasonably possible" is OK.
Yep.
I can imagine cases where CPU ucode might need to get applied as early as possible.
Quoted 2 lines
(If the firmware update is critical, it should be supplied as a BIOS update.)
I can also imagine cases where BIOS update is not an option/is not provided and we'd need the ucode applied by the OS too :). Think BIOS quirks in the kernel whereas all that can be fixed in the BIOS.
Quoted 30 lines
So I think this is moot with respect to this particular patch.
Quoted 7 lines
- you're adding a
microcode_xen.cas if this is another hw vendor and you're figuring out which is the proper firmware image based on the vendor, then you load it and then do the hypercall. This is duplicating code and inviting future bugs. Everytime the hw vendors decide to change something to their fw loading mechanism, xen needs to be updated. Also, you don't do any sanity checks to the ucode image as the vendor code does which is inacceptable.The code within the hypervisor is more or less the same as the kernel's:
it does all the required vendor-specific checks on the firmware and loads it on all the CPUs with the appropriate mechanism. The hypercall ABI is vendor-agnostic, but it assumes that dom0 will supply suitable microcode for the current CPU vendor (though if it doesn't, the image will presumably be rejected).So from that perspective, it is a distinct microcode loading mechanism, akin to a vendor-specific one. The awkward part is getting the filename to actually request from usermode, which is Intel/AMD/whatever specific, which results in duplicated code to generate that pathname.
Quoted 7 lines
What it should do instead, is call into the hw vendor-specific ucode loading mechanism and do all the image loading/verification there. The only thing you'd need to supply is a xen-specific ->apply_microcode() doing the hypercall _after_ the ucode image has been verified and is good to go. I'm guessing the
XENPF_microcode_updatedoes call into the hypervisor and calls a xen-specific ucode update mechanism based on the vendor instead of using the vendor-supplied code...
Quoted 6 lines
Well, I was trying to avoid putting Xen-specific code into the existing Intel/AMD loaders. That doesn't seem any cleaner.
I could export "my firmware pathname" functions from them and have the Xen driver call those, rather than duplicating the pathname construction code. Would that help address your concerns?
Well, I was thinking even more radically than that. How about
1. microcode_xen.c figures out which struct microcode_ops to use based on the hw vendor;
2. overwrites the ->apply_microcode ptr with the hypercall wrapper
3. dom0 uses it to load the firmware image and do all checks to it
4. eventually, the hypervisor gets to apply the _verified_ microcode image (no more checks needed) using the vendor-specific application method.
This way there's almost no code duplication, you'll be reusing the vendor-supplied code in baremetal which gets tested and updated everytime it needs to and will save you a bunch of work everytime there's change to it needed to replicate it into the hypervisor.
Btw, if the code within the hypervisor is similar to the kernel's, you could even save the original ->apply_microcode() pointer from step 2 and call it in the hypervisor when the XENPF_microcode_update hypercall op gets called. This way you have 0 code duplication.
I think this'll be very cool :).
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 6 lines · Borislav Petkov
Quoted 6 lines
Well, I was trying to avoid putting Xen-specific code into the existing Intel/AMD loaders. That doesn't seem any cleaner.
I could export "my firmware pathname" functions from them and have the Xen driver call those, rather than duplicating the pathname construction code. Would that help address your concerns?
Quoted 8 lines
Well, I was thinking even more radically than that. How about
1.
microcode_xen.cfigures out which structmicrocode_opsto use based on the hw vendor;2. overwrites the ->apply_microcode ptr with the hypercall wrapper
3. dom0 uses it to load the firmware image and do all checks to it
That could be made to work, but I don't really see it as being an improvement. The whole "overwriting bits of other people's ops structures" thing seems like a pretty bad idea for long term maintainability.
Quoted 8 lines
4. eventually, the hypervisor gets to apply the _verified_ microcode image (no more checks needed) using the vendor-specific application method.
This way there's almost no code duplication, you'll be reusing the vendor-supplied code in baremetal which gets tested and updated everytime it needs to and will save you a bunch of work everytime there's change to it needed to replicate it into the hypervisor.
In general Xen tries to avoid trusting its domains. Admittedly, dom0 is special in that it is already somewhat trusted, but even dom0 is constrained by Xen. For microcode, Xen just depends on it to provide a best-possible microcode file, then Xen+the CPU do the work of fully validating it and installing it.
Quoted 4 lines
Btw, if the code within the hypervisor is similar to the kernel's, you could even save the original ->apply_microcode() pointer from step 2 and call it in the hypervisor when the
XENPF_microcode_updatehypercall op gets called. This way you have 0 code duplication.
The hypervisor and its domains are completely separate pieces of code. This is akin to suggesting the kernel directly jump through a pointer and to run some usermode code.
J
Borislav Petkov
maintainerQuoted 12 lines · Jeremy Fitzhardinge
Quoted 11 lines · Borislav Petkov
Quoted 6 lines
Well, I was trying to avoid putting Xen-specific code into the existing Intel/AMD loaders. That doesn't seem any cleaner.
I could export "my firmware pathname" functions from them and have the Xen driver call those, rather than duplicating the pathname construction code. Would that help address your concerns?
Well, I was thinking even more radically than that. How about
1.
microcode_xen.cfigures out which structmicrocode_opsto use based on the hw vendor;
Quoted 6 lines
Quoted 3 lines
2. overwrites the ->apply_microcode ptr with the hypercall wrapper
3. dom0 uses it to load the firmware image and do all checks to it
That could be made to work, but I don't really see it as being an improvement.
WTF? How is
* almost no code duplication
* not adding Xen-specific checks to generic arch code
* relying on already tested codepaths
not an improvement?
Quoted 2 lines
The whole "overwriting bits of other people's ops structures" thing seems like a pretty bad idea for long term maintainability.
I don't think that's an issue: you either load microcode_xen in dom0 (x)or the respective vendor driver on baremetal.
Quoted 9 lines
Quoted 8 lines
4. eventually, the hypervisor gets to apply the _verified_ microcode image (no more checks needed) using the vendor-specific application method.
This way there's almost no code duplication, you'll be reusing the vendor-supplied code in baremetal which gets tested and updated everytime it needs to and will save you a bunch of work everytime there's change to it needed to replicate it into the hypervisor.
Quoted 5 lines
In general Xen tries to avoid trusting its domains. Admittedly, dom0 is special in that it is already somewhat trusted, but even dom0 is constrained by Xen. For microcode, Xen just depends on it to provide a best-possible microcode file, then Xen+the CPU do the work of fully validating it and installing it.
Well, the CPU doesn't trust the microcode provided by the software either. And why should it?
All I'm saying is, you should try as best as possible to avoid code duplication and the need for replicating functionality to Xen, thus doubling - even multiplying - the effort for coding/testing baremetal and then Xen. Microcode is a perfect example since the vendors do all their testing/verification on baremetal anyway and the rest should benefit from that work.
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 18 lines · Borislav Petkov
Quoted 9 lines · Jeremy Fitzhardinge
Quoted 8 lines · Borislav Petkov
Quoted 6 lines
Well, I was trying to avoid putting Xen-specific code into the existing Intel/AMD loaders. That doesn't seem any cleaner.
I could export "my firmware pathname" functions from them and have the Xen driver call those, rather than duplicating the pathname construction code. Would that help address your concerns?
Well, I was thinking even more radically than that. How about
Quoted 8 lines
Quoted 6 lines
1.
microcode_xen.cfigures out which structmicrocode_opsto use based on the hw vendor;2. overwrites the ->apply_microcode ptr with the hypercall wrapper
3. dom0 uses it to load the firmware image and do all checks to it
That could be made to work, but I don't really see it as being an improvement.
Quoted 7 lines
WTF? How is
* almost no code duplication
* not adding Xen-specific checks to generic arch code
* relying on already tested codepathsnot an improvement?
My understanding of what you're proposing is:
1. the normal CPU-specific microcode driver starts up
2. an if (xen) test overrides the apply_microcode to call a Xen-specific function
3. the driver requests the firmware, loads and verifies it as normal
4. the microcode is applied via the Xen apply_microcode function
With (2) needing a Xen-specific change to both the Intel and AMD drivers.
If not, what are you proposing in detail?
Are you instead thinking:
* Xen-specific microcode driver starts up
* It calls either the intel or amd request_microcode_fw as needed
* apply the the loaded data via hypercall
?
But all this is flawed because the microcode_intel/amd.c drivers assume they can see all the CPUs present in the system, and load suitable microcode for each specific one. But a kernel in a Xen domain only has virtual CPUs - not physical ones - and has no idea how to get appropriate microcode data for all the physical CPUs in the system.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
4. eventually, the hypervisor gets to apply the _verified_ microcode image (no more checks needed) using the vendor-specific application method.
This way there's almost no code duplication, you'll be reusing the vendor-supplied code in baremetal which gets tested and updated everytime it needs to and will save you a bunch of work everytime there's change to it needed to replicate it into the hypervisor.
Quoted 7 lines
Quoted 5 lines
In general Xen tries to avoid trusting its domains. Admittedly, dom0 is special in that it is already somewhat trusted, but even dom0 is constrained by Xen. For microcode, Xen just depends on it to provide a best-possible microcode file, then Xen+the CPU do the work of fully validating it and installing it.
Well, the CPU doesn't trust the microcode provided by the software either. And why should it?
Right. There's nothing really for the driver to do (kernel or Xen). The Intel driver does a bunch of checksum checking, which is presumably useful for detecting a corrupted firmware update early, and splits out the chunks specific to the current cpus. The AMD driver does no checking per-se, but also looks for processor-specific microcode update chunks.
In the Xen case, since a domain is not necessarily seeing all the details of the underlying physical cpus, it makes most sense for usermode to just pass in the whole microcode file and let Xen do all the checking/filtering based on complete knowledge of the system.
Quoted 6 lines
All I'm saying is, you should try as best as possible to avoid code duplication and the need for replicating functionality to Xen, thus doubling - even multiplying - the effort for coding/testing baremetal and then Xen. Microcode is a perfect example since the vendors do all their testing/verification on baremetal anyway and the rest should benefit from that work.
CPU vendors test Xen, and Intel is particularly interested in getting this microcode driver upstream. The amount of duplicated code is trivial, and the basic structure of the microcode updates doesn't seem set to change. Since Xen has to have all sorts of other CPU-specific code which at least somewhat overlaps with what's in the kernel a bit more doesn't matter.
(It's interesting that nobody seems to have been interested enough in Via to have ever implemented a microcode driver for it - perhaps they only ever do it from BIOS.)
J
Quoted 18 lines · Borislav Petkov
Quoted 9 lines · Jeremy Fitzhardinge
Quoted 8 lines · Borislav Petkov
Quoted 6 lines
Well, I was trying to avoid putting Xen-specific code into the existing Intel/AMD loaders. That doesn't seem any cleaner.
I could export "my firmware pathname" functions from them and have the Xen driver call those, rather than duplicating the pathname construction code. Would that help address your concerns?
Well, I was thinking even more radically than that. How about
Quoted 8 lines
Quoted 6 lines
1.
microcode_xen.cfigures out which structmicrocode_opsto use based on the hw vendor;2. overwrites the ->apply_microcode ptr with the hypercall wrapper
3. dom0 uses it to load the firmware image and do all checks to it
That could be made to work, but I don't really see it as being an improvement.
Quoted 7 lines
WTF? How is
* almost no code duplication
* not adding Xen-specific checks to generic arch code
* relying on already tested codepathsnot an improvement?
My understanding of what you're proposing is:
1. the normal CPU-specific microcode driver starts up
2. an if (xen) test overrides the apply_microcode to call a Xen-specific function
3. the driver requests the firmware, loads and verifies it as normal
4. the microcode is applied via the Xen apply_microcode function
With (2) needing a Xen-specific change to both the Intel and AMD drivers.
If not, what are you proposing in detail?
Are you instead thinking:
* Xen-specific microcode driver starts up
* It calls either the intel or amd request_microcode_fw as needed
* apply the the loaded data via hypercall
?
But all this is flawed because the microcode_intel/amd.c drivers assume they can see all the CPUs present in the system, and load suitable microcode for each specific one. But a kernel in a Xen domain only has virtual CPUs - not physical ones - and has no idea how to get appropriate microcode data for all the physical CPUs in the system.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
4. eventually, the hypervisor gets to apply the _verified_ microcode image (no more checks needed) using the vendor-specific application method.
This way there's almost no code duplication, you'll be reusing the vendor-supplied code in baremetal which gets tested and updated everytime it needs to and will save you a bunch of work everytime there's change to it needed to replicate it into the hypervisor.
Quoted 7 lines
Quoted 5 lines
In general Xen tries to avoid trusting its domains. Admittedly, dom0 is special in that it is already somewhat trusted, but even dom0 is constrained by Xen. For microcode, Xen just depends on it to provide a best-possible microcode file, then Xen+the CPU do the work of fully validating it and installing it.
Well, the CPU doesn't trust the microcode provided by the software either. And why should it?
Right. There's nothing really for the driver to do (kernel or Xen). The Intel driver does a bunch of checksum checking, which is presumably useful for detecting a corrupted firmware update early, and splits out the chunks specific to the current cpus. The AMD driver does no checking per-se, but also looks for processor-specific microcode update chunks.
In the Xen case, since a domain is not necessarily seeing all the details of the underlying physical cpus, it makes most sense for usermode to just pass in the whole microcode file and let Xen do all the checking/filtering based on complete knowledge of the system.
Quoted 6 lines
All I'm saying is, you should try as best as possible to avoid code duplication and the need for replicating functionality to Xen, thus doubling - even multiplying - the effort for coding/testing baremetal and then Xen. Microcode is a perfect example since the vendors do all their testing/verification on baremetal anyway and the rest should benefit from that work.
CPU vendors test Xen, and Intel is particularly interested in getting this microcode driver upstream. The amount of duplicated code is trivial, and the basic structure of the microcode updates doesn't seem set to change. Since Xen has to have all sorts of other CPU-specific code which at least somewhat overlaps with what's in the kernel a bit more doesn't matter.
(It's interesting that nobody seems to have been interested enough in Via to have ever implemented a microcode driver for it - perhaps they only ever do it from BIOS.)
J
H. Peter Anvin
reviewerNote: Xen may not have devices, but it is already using multiboot to load multiple modules. It could load the microcode blob that way.
That would enable an earlier loading of microcode, which is a very good thing.
-hpa
Borislav Petkov
maintainerQuoted 12 lines · Jeremy Fitzhardinge
My understanding of what you're proposing is:
1. the normal CPU-specific microcode driver starts up
2. an if (xen) test overrides theapply_microcodeto call a Xen-specific function
3. the driver requests the firmware, loads and verifies it as normal
4. the microcode is applied via the Xenapply_microcodefunctionWith (2) needing a Xen-specific change to both the Intel and AMD drivers.
If not, what are you proposing in detail?
Quoted 7 lines
Are you instead thinking:
* Xen-specific microcode driver starts up
* It calls either the intel or amdrequest_microcode_fwas needed
* apply the the loaded data via hypercall?
I am thinking something in the sense of the above. For example, in the AMD case you take
static struct microcode_ops microcode_amd_ops = {
.request_microcode_user = request_microcode_user, .request_microcode_fw = request_microcode_fw, .collect_cpu_info = collect_cpu_info_amd, .apply_microcode = apply_microcode_amd, .microcode_fini_cpu = microcode_fini_cpu_amd,
};
and reuse the ->request_microcode_fw, ->collect_cpu_info and ->microcode_fini_cpu on dom0 as if you're running on baremetal. Up to the point where you need to apply the microcode. Then, you use your supplied ->apply_microcode hypercall wrapper to call into the hypervisor.
Quoted 5 lines
But all this is flawed because the
microcode_intel/amd.cdrivers assume they can see all the CPUs present in the system, and load suitable microcode for each specific one. But a kernel in a Xen domain only has virtual CPUs - not physical ones - and has no idea how to get appropriate microcode data for all the physical CPUs in the system.
Well, let me quote you:
Quoted 5 lines · Jeremy Fitzhardinge
Xen update mechanism is uniform independent of the CPU type, but the driver must know where to find the data file, which depends on the CPU type. And since the update hypercall updates all CPUs, we only need to execute it once on any CPU - but for simplicity it just runs it only on (V)CPU 0.
so you only do it once and exit early in the rest of the cases. I wouldn't worry about performance since ucode is applied only once upon boot.
[..]
Quoted 11 lines
Right. There's nothing really for the driver to do (kernel or Xen). The Intel driver does a bunch of checksum checking, which is presumably useful for detecting a corrupted firmware update early, and splits out the chunks specific to the current cpus. The AMD driver does no checking per-se, but also looks for processor-specific microcode update chunks.
In the Xen case, since a domain is not necessarily seeing all the details of the underlying physical cpus, it makes most sense for usermode to just pass in the whole microcode file and let Xen do all the checking/filtering based on complete knowledge of the system.
This is exactly what I'm talking about - why copy all that checking/filtering code from baremetal to Xen instead of simply reusing it? Especially if you'd need to update the copy from time to time when baremetal changes.
Quoted 11 lines
Quoted 6 lines
All I'm saying is, you should try as best as possible to avoid code duplication and the need for replicating functionality to Xen, thus doubling - even multiplying - the effort for coding/testing baremetal and then Xen. Microcode is a perfect example since the vendors do all their testing/verification on baremetal anyway and the rest should benefit from that work.
CPU vendors test Xen, and Intel is particularly interested in getting this microcode driver upstream. The amount of duplicated code is trivial, and the basic structure of the microcode updates doesn't seem set to change.
Uuh, I wouldn't bet on that though :).
Quoted 3 lines
Since Xen has to have all sorts of other CPU-specific code which at least somewhat overlaps with what's in the kernel a bit more doesn't matter.
Well, I'll let x86 people decide on that but last time I checked they opposed "if (xen)" sprinkling all over the place.
Btw, hpa has a point, if you can load microcode using multiboot, all that discussion will become moot since you'll be better at loading microcode even than baremetal. We need a similar mechanism in x86 too since the current one loads the microcode definitely too late.
The optimal case for baremetal would be to load it as early as possible on each CPU's bootstrapping path and if you can do that in the hypervisor, before even dom0 starts, you're very much fine.
Quoted 3 lines
(It's interesting that nobody seems to have been interested enough in Via to have ever implemented a microcode driver for it - perhaps they only ever do it from BIOS.)
Yeah, that's one possibility. I bet they'll do it though, when BIOS cannot be updated/is out of life on some of their platforms and they want to get a ucode patch applied.
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 5 lines · H. Peter Anvin
Note: Xen may not have devices, but it is already using multiboot to load multiple modules. It could load the microcode blob that way.
That would enable an earlier loading of microcode, which is a very good thing.
Yes, that is a thought, but it would require some distro support to make sure the firmware is copied into /boot, and grub updated appropriately.
The principle advantage of updating the microcode driver is that it Just Works regardless of whether the system is booting native or Xen.
J
Quoted 9 lines · Borislav Petkov
I am thinking something in the sense of the above. For example, in the AMD case you take
static struct
microcode_opsmicrocode_amd_ops= {.request_microcode_user = request_microcode_user, .request_microcode_fw = request_microcode_fw, .collect_cpu_info = collect_cpu_info_amd, .apply_microcode = apply_microcode_amd, .microcode_fini_cpu = microcode_fini_cpu_amd,
Quoted 7 lines
};
and reuse the ->request_microcode_fw, ->collect_cpu_info and ->microcode_fini_cpu on dom0 as if you're running on baremetal. Up to the point where you need to apply the microcode. Then, you use your supplied ->apply_microcode hypercall wrapper to call into the hypervisor.
collect_cpu_info can't work, because the domain doesn't have access to all the host's physical CPUs.
However, even aside from that, it means exporting a pile of internal details from microcode_amd and reusing them within microcode_xen. And it requires that it be done again for each vendor.
But all that's really needed is a dead simple "request" that loads the entire file (with a vendor-specific name) and shoves it into Xen. There's no need for any vendor-specific code beyond the filename.
Quoted 8 lines
Quoted 5 lines
But all this is flawed because the
microcode_intel/amd.cdrivers assume they can see all the CPUs present in the system, and load suitable microcode for each specific one. But a kernel in a Xen domain only has virtual CPUs - not physical ones - and has no idea how to get appropriate microcode data for all the physical CPUs in the system.Well, let me quote you:
On Fri, Jan 28, 2011 at 04:26:52PM -0800, Jeremy Fitzhardinge wrote:
Quoted 8 lines
Quoted 5 lines
Xen update mechanism is uniform independent of the CPU type, but the driver must know where to find the data file, which depends on the CPU type. And since the update hypercall updates all CPUs, we only need to execute it once on any CPU - but for simplicity it just runs it only on (V)CPU 0.
so you only do it once and exit early in the rest of the cases. I wouldn't worry about performance since ucode is applied only once upon boot.
Its not a performance question. The Intel and AMD microcode drivers parse the full blob loaded from userspace, and just extract the chunk needed for each CPU. It does this for each separate CPU, so in principle you could have a mixture of models within one machine or something (the driver certainly assumes that could happen; perhaps it could on a larger multinode machine).
The point is that if it does this on (what the domain sees as ) "cpu 0", then it may throw away microcode chunks needed for other CPUs. That's why we need to hand Xen the entire microcode file, and let the hypervisor do the work of splitting it up and installing it on the CPUs.
Quoted 4 lines
This is exactly what I'm talking about - why copy all that checking/filtering code from baremetal to Xen instead of simply reusing it? Especially if you'd need to update the copy from time to time when baremetal changes.
The code in the kernel is in the wrong place. It has to be done in Xen. When Xen is present, the code in the kernel is redundant, not the other way around.
Quoted 5 lines
Quoted 4 lines
CPU vendors test Xen, and Intel is particularly interested in getting this microcode driver upstream. The amount of duplicated code is trivial, and the basic structure of the microcode updates doesn't seem set to change.
Uuh, I wouldn't bet on that though :).
Shrug. AFAICT the mechanism hasn't changed since it was first introduced. If there's a change, then both Linux and Xen will have to change, and most likely the same CPU vendor engineer will provide a patch for both. Xen has a good record for tracking new CPU features.
Quoted 5 lines
Quoted 3 lines
Since Xen has to have all sorts of other CPU-specific code which at least somewhat overlaps with what's in the kernel a bit more doesn't matter.
Well, I'll let x86 people decide on that but last time I checked they opposed "if (xen)" sprinkling all over the place.
Eh? I'm talking about code within Xen; it doesn't involve any if (xen)s within the kernel.
Quoted 8 lines
Btw, hpa has a point, if you can load microcode using multiboot, all that discussion will become moot since you'll be better at loading microcode even than baremetal. We need a similar mechanism in x86 too since the current one loads the microcode definitely too late.
The optimal case for baremetal would be to load it as early as possible on each CPU's bootstrapping path and if you can do that in the hypervisor, before even dom0 starts, you're very much fine.
It is possible, but it requires that vendors install the microcode updates in /boot and update the grub entries accordingly. I'd prefer a solution which works with current distros as-is.
J
Borislav Petkov
maintainerOk, I'm reading your answers but I keep getting the impression that this discussion is not moving anywhere. You keep finding reasons why it can't be done and trying to persuade me that your way is the only way. Why is that?
And I'm telling you microcode_xen has nothing to do among vendor-specific code. Since when is xen a hw vendor and deserves special attention? And don't tell me because people use it. It is absolutely inacceptable to add a bunch of code to arch/x86 just because you're telling me it can't be done differently, not intermixed with hw vendor code and just because a hypervisor vendor needs it. Does that mean that if every other virtualization booth wants to add their special code to arch/x86, we just go and slap it in without questioning its design?
I don't think so.
Quoted 12 lines · Jeremy Fitzhardinge
Quoted 11 lines · Borislav Petkov
I am thinking something in the sense of the above. For example, in the AMD case you take
static struct
microcode_opsmicrocode_amd_ops= {.request_microcode_user = request_microcode_user, .request_microcode_fw = request_microcode_fw, .collect_cpu_info = collect_cpu_info_amd, .apply_microcode = apply_microcode_amd, .microcode_fini_cpu = microcode_fini_cpu_amd,};
Quoted 8 lines
Quoted 5 lines
and reuse the ->request_microcode_fw, ->collect_cpu_info and ->microcode_fini_cpu on dom0 as if you're running on baremetal. Up to the point where you need to apply the microcode. Then, you use your supplied ->apply_microcode hypercall wrapper to call into the hypervisor.
collect_cpu_infocan't work, because the domain doesn't have access to all the host's physical CPUs.
Why would you need that? You can safely assume that the ucode patch level on all cores across the system are identical - I've yet to see a machine running with different patch levels for the same reason that mixed silicon systems is a large pain in the ass and you're better off buying yourself a completely new system.
Quoted 3 lines
However, even aside from that, it means exporting a pile of internal details from
microcode_amdand reusing them withinmicrocode_xen. And it requires that it be done again for each vendor.
Why can't you load the appropriate, unmodified microcode_<vendor> module in dom0 and let it call the hypercall?
Quoted 3 lines
But all that's really needed is a dead simple "request" that loads the entire file (with a vendor-specific name) and shoves it into Xen. There's no need for any vendor-specific code beyond the filename.
But that adds this funny chunk
- if (c->x86_vendor == X86_VENDOR_INTEL)
+ if (xen_pv_domain())
+ microcode_ops = init_xen_microcode();
+ else if (c->x86_vendor == X86_VENDOR_INTEL)
microcode_ops = init_intel_microcode();
else if (c->x86_vendor == X86_VENDOR_AMD)
microcode_ops = init_amd_microcode();which can clearly be avoided. Now imagine the if-else thing contained a dozen or more virt solutions in there, bloat! Now imagine this not only in the microcode driver but in a bunch of other places across arch/x86.
Quoted 24 lines
Quoted 8 lines
Quoted 5 lines
But all this is flawed because the
microcode_intel/amd.cdrivers assume they can see all the CPUs present in the system, and load suitable microcode for each specific one. But a kernel in a Xen domain only has virtual CPUs - not physical ones - and has no idea how to get appropriate microcode data for all the physical CPUs in the system.Well, let me quote you:
On Fri, Jan 28, 2011 at 04:26:52PM -0800, Jeremy Fitzhardinge wrote:
Quoted 8 lines
Quoted 5 lines
Xen update mechanism is uniform independent of the CPU type, but the driver must know where to find the data file, which depends on the CPU type. And since the update hypercall updates all CPUs, we only need to execute it once on any CPU - but for simplicity it just runs it only on (V)CPU 0.
so you only do it once and exit early in the rest of the cases. I wouldn't worry about performance since ucode is applied only once upon boot.
Its not a performance question. The Intel and AMD microcode drivers parse the full blob loaded from userspace, and just extract the chunk needed for each CPU. It does this for each separate CPU, so in principle you could have a mixture of models within one machine or something (the driver certainly assumes that could happen; perhaps it could on a larger multinode machine).
Quoted 4 lines
The point is that if it does this on (what the domain sees as ) "cpu 0", then it may throw away microcode chunks needed for other CPUs. That's why we need to hand Xen the entire microcode file, and let the hypervisor do the work of splitting it up and installing it on the CPUs.
see comment about mixed silicon above.
Quoted 8 lines
Quoted 4 lines
This is exactly what I'm talking about - why copy all that checking/filtering code from baremetal to Xen instead of simply reusing it? Especially if you'd need to update the copy from time to time when baremetal changes.
The code in the kernel is in the wrong place. It has to be done in Xen. When Xen is present, the code in the kernel is redundant, not the other way around.
Ok, this says it all. Let's remove the arch code and replace it with xen-friendly version - we won't need the baremetal one anyway. Jeez!
Quoted 10 lines
Quoted 5 lines
Quoted 4 lines
CPU vendors test Xen, and Intel is particularly interested in getting this microcode driver upstream. The amount of duplicated code is trivial, and the basic structure of the microcode updates doesn't seem set to change.
Uuh, I wouldn't bet on that though :).
Shrug. AFAICT the mechanism hasn't changed since it was first introduced. If there's a change, then both Linux and Xen will have to change, and most likely the same CPU vendor engineer will provide a patch for both. Xen has a good record for tracking new CPU features.
I don't think that the same CPU vendor engineer will do that, believe me!
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Since Xen has to have all sorts of other CPU-specific code which at least somewhat overlaps with what's in the kernel a bit more doesn't matter.
Well, I'll let x86 people decide on that but last time I checked they opposed "if (xen)" sprinkling all over the place.
Eh? I'm talking about code within Xen; it doesn't involve any if (xen)s within the kernel.
Ok, I'm getting tired of this. I'll gladly read all your answers if you have constructive suggestions and better proposals - if the only thing you have to come back are reasons why xen's is the only way to do it, then don't even bother to answer - at least I won't. If x86 people want to take your code, they'll do it since it is their call anyway.
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 2 lines · Borislav Petkov
Why would you need that? You can safely assume that the ucode patch level on all cores across the system are identical - I've yet to see a
No. You can safely assume that the ucode patch level on all cores on a package are identical, and that's it. Mixing processors with different processor flags (and thus potentially different microcode) is not uncommon. If it boots, it *will* be done, especially later in the product cycle, when specific spare parts are harder to come by. Sometimes you can even mix processors of different steppings.
Quoted 2 lines
mixed silicon systems is a large pain in the ass and you're better off buying yourself a completely new system.
Well, I have some of them at work.
Mixed CPU steppings or processor flags happen when you expand the system later to fill in processor packages (reasonably common), or when you replace a CPU that is flaky (rare). This sort of mixing of different processors is really common on older Xeon systems, and most of them need OS-assisted microcode updates to get the latest microcode available.
I am not arguing anything about the way Xen decided to go about implementing the feature, but they got the requirement "must pass through all the microcodes without removing any" right. It exists.
Quoted 6 lines
Quoted 3 lines
However, even aside from that, it means exporting a pile of internal details from
microcode_amdand reusing them withinmicrocode_xen. And it requires that it be done again for each vendor.Why can't you load the appropriate, unmodified microcode_<vendor> module in dom0 and let it call the hypercall?
Or add a microcode_virtual passthrough device, for that matter. Wouldn't that be much cleaner and more palatable?
Signature · 4 lines
"One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh
Quoted 2 lines · Borislav Petkov
Why would you need that? You can safely assume that the ucode patch level on all cores across the system are identical - I've yet to see a
No. You can safely assume that the ucode patch level on all cores on a package are identical, and that's it. Mixing processors with different processor flags (and thus potentially different microcode) is not uncommon. If it boots, it *will* be done, especially later in the product cycle, when specific spare parts are harder to come by. Sometimes you can even mix processors of different steppings.
Quoted 2 lines
mixed silicon systems is a large pain in the ass and you're better off buying yourself a completely new system.
Well, I have some of them at work.
Mixed CPU steppings or processor flags happen when you expand the system later to fill in processor packages (reasonably common), or when you replace a CPU that is flaky (rare). This sort of mixing of different processors is really common on older Xeon systems, and most of them need OS-assisted microcode updates to get the latest microcode available.
I am not arguing anything about the way Xen decided to go about implementing the feature, but they got the requirement "must pass through all the microcodes without removing any" right. It exists.
Quoted 6 lines
Quoted 3 lines
However, even aside from that, it means exporting a pile of internal details from
microcode_amdand reusing them withinmicrocode_xen. And it requires that it be done again for each vendor.Why can't you load the appropriate, unmodified microcode_<vendor> module in dom0 and let it call the hypercall?
Or add a microcode_virtual passthrough device, for that matter. Wouldn't that be much cleaner and more palatable?
Signature · 4 lines
"One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh
Quoted 4 lines · Borislav Petkov
Ok, I'm reading your answers but I keep getting the impression that this discussion is not moving anywhere. You keep finding reasons why it can't be done and trying to persuade me that your way is the only way. Why is that?
I agree that this conversation has got bogged down. I'm getting the impression that you're not really understanding my answers within the context of how Xen works, and so things are going in circles. I'm happy to go into more detail if you're interested.
I'm not trying to be obstructionist here. We've often changed the way things work on the Xen side to smooth the path into the kernel, and I'm perfectly willing to do it again for the microcode driver if it makes sense to do so.
Quoted 3 lines
And I'm telling you
microcode_xenhas nothing to do among vendor-specific code. Since when is xen a hw vendor and deserves special attention? And don't tell me because people use it.
Who's asking for special attention? I'm just trying to use the existing microcode infrastructure for handling different methods of microcode update to add one more. Why is "because people use it" not a useful thing to say? If I said "but nobody uses it", then that would be a strong argument for not including it.
Quoted 4 lines
It is absolutely
inacceptable to add a bunch of code toarch/x86just because you're telling me it can't be done differently, not intermixed with hw vendor code and just because a hypervisor vendor needs it.
You seem to have staked out a very... specific... position here, but I don't agree with your premise that microcode_foo is specifically microcode_<cpu-vendor>. If you view it as microcode_<method of loading microcode> then adding microcode_xen makes perfect sense. Since it is a small, self-contained piece of code that doesn't have any effects on any other code, nor any dependencies aside from the microcode driver's internal interface, I think it is a clean way to approach the problem.
Or to turn it around, what specific problems do you see arising from implementing the Xen microcode loader in this way? Why is adding a third microcode_foo.c a problem?
Quoted 3 lines
Does that mean that
if every other virtualization booth wants to add their special code toarch/x86, we just go and slap it in without questioning its design?
Of course not; the whole point of posting code for review is to get feedback on both general and specific issues, and I appreciate the time you've spent on this. But I have to say I don't really understand what your objections are. Are you objecting to the very principle of adding a new microcode driver, or is there something specific about the code I posted that you have issues with?
Thanks,
J
Quoted 4 lines · Borislav Petkov
Ok, I'm reading your answers but I keep getting the impression that this discussion is not moving anywhere. You keep finding reasons why it can't be done and trying to persuade me that your way is the only way. Why is that?
I agree that this conversation has got bogged down. I'm getting the impression that you're not really understanding my answers within the context of how Xen works, and so things are going in circles. I'm happy to go into more detail if you're interested.
I'm not trying to be obstructionist here. We've often changed the way things work on the Xen side to smooth the path into the kernel, and I'm perfectly willing to do it again for the microcode driver if it makes sense to do so.
Quoted 3 lines
And I'm telling you
microcode_xenhas nothing to do among vendor-specific code. Since when is xen a hw vendor and deserves special attention? And don't tell me because people use it.
Who's asking for special attention? I'm just trying to use the existing microcode infrastructure for handling different methods of microcode update to add one more. Why is "because people use it" not a useful thing to say? If I said "but nobody uses it", then that would be a strong argument for not including it.
Quoted 4 lines
It is absolutely
inacceptable to add a bunch of code toarch/x86just because you're telling me it can't be done differently, not intermixed with hw vendor code and just because a hypervisor vendor needs it.
You seem to have staked out a very... specific... position here, but I don't agree with your premise that microcode_foo is specifically microcode_<cpu-vendor>. If you view it as microcode_<method of loading microcode> then adding microcode_xen makes perfect sense. Since it is a small, self-contained piece of code that doesn't have any effects on any other code, nor any dependencies aside from the microcode driver's internal interface, I think it is a clean way to approach the problem.
Or to turn it around, what specific problems do you see arising from implementing the Xen microcode loader in this way? Why is adding a third microcode_foo.c a problem?
Quoted 3 lines
Does that mean that
if every other virtualization booth wants to add their special code toarch/x86, we just go and slap it in without questioning its design?
Of course not; the whole point of posting code for review is to get feedback on both general and specific issues, and I appreciate the time you've spent on this. But I have to say I don't really understand what your objections are. Are you objecting to the very principle of adding a new microcode driver, or is there something specific about the code I posted that you have issues with?
Thanks,
J
Sorry, missed your specific comments below.
Quoted 7 lines · Borislav Petkov
Quoted 2 lines
collect_cpu_infocan't work, because the domain doesn't have access to all the host's physical CPUs.Why would you need that? You can safely assume that the ucode patch level on all cores across the system are identical - I've yet to see a machine running with different patch levels for the same reason that mixed silicon systems is a large pain in the ass and you're better off buying yourself a completely new system.
I was thinking specifically about a large multi-chassis system with lots of nodes. In that case you could well imagine different nodes/chassis having variations of CPU steppings.
But even on a single board, I don't know what limitations there are on mixing steppings between sockets, but I suspect it is possible. Obviously I wouldn't expect there to be large variations between CPUs (different models or features) - that would be a problem.
Quoted 5 lines
Quoted 3 lines
However, even aside from that, it means exporting a pile of internal details from
microcode_amdand reusing them withinmicrocode_xen. And it requires that it be done again for each vendor.Why can't you load the appropriate, unmodified microcode_<vendor> module in dom0 and let it call the hypercall?
Well, if it can call hypercalls, then to some extent it is already "modified". Either I change the source to change the wrmsrs into hypercalls, or overwrite its operations structure to change the apply_microcode function pointer (which would require - at the very least - making it non-static) to a hyper-calling function.
Either way it implies that we're delving into the Intel/AMD drivers in order to implement Xen-specific functionality, which is clearly (to me, at least) much worse than just making the Xen-specific code completely self-contained.
Quoted 13 lines
Quoted 3 lines
But all that's really needed is a dead simple "request" that loads the entire file (with a vendor-specific name) and shoves it into Xen. There's no need for any vendor-specific code beyond the filename.
But that adds this funny chunk
(quoted diff) +3
-1- if (c->x86_vendor == X86_VENDOR_INTEL) + if (xen_pv_domain()) + microcode_ops = init_xen_microcode(); + else if (c->x86_vendor == X86_VENDOR_INTEL) microcode_ops = init_intel_microcode(); else if (c->x86_vendor == X86_VENDOR_AMD) microcode_ops = init_amd_microcode();
Quoted 2 lines
which can clearly be avoided. Now imagine the if-else thing contained a dozen or more virt solutions in there, bloat!
This is just the way microcode_core.c is structured. It doesn't have a dozen or more virt solutions, but if it did, we could change the way that microcode loaders are probed for to make it cleaner. Exactly the same issue arises if you say "there might be a dozen or more cpu vendors
- bloat!". True, there might be, and we should fix it in that case. Or conversely, there might be a new CPU vendor who decides to implement the Intel microcode loading interface, in which case the tests on a specific vendor become wrong.
Quoted 2 lines
Now imagine this not only
in the microcode driver but in a bunch of other places acrossarch/x86.
Why imagine that? We're not talking about the rest of arch/x86.
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
This is exactly what I'm talking about - why copy all that checking/filtering code from baremetal to Xen instead of simply reusing it? Especially if you'd need to update the copy from time to time when baremetal changes.
The code in the kernel is in the wrong place. It has to be done in Xen. When Xen is present, the code in the kernel is redundant, not the other way around.
Ok, this says it all. Let's remove the arch code and replace it with xen-friendly version - we won't need the baremetal one anyway. Jeez!
I didn't say anything about removing the code from Linux. But the simple fact is that a kernel running under a hypervisor isn't the ultimate owner of the machine's hardware - the hypervisor is, and it controls all access to the hardware. That's deeply fundamental to the way any hypervisor works, and Xen is just one example. And the consequence of that is that Linux has to use hypervisor facilities to perform certain operations rather than using its own code.
Quoted 5 lines
Quoted 4 lines
Shrug. AFAICT the mechanism hasn't changed since it was first introduced. If there's a change, then both Linux and Xen will have to change, and most likely the same CPU vendor engineer will provide a patch for both. Xen has a good record for tracking new CPU features.
I don't think that the same CPU vendor engineer will do that, believe me!
Many individual Intel and AMD engineers contribute code to both Xen and Linux. It's not a question of belief.
Thanks,
J
Sorry, missed your specific comments below.
Quoted 7 lines · Borislav Petkov
Quoted 2 lines
collect_cpu_infocan't work, because the domain doesn't have access to all the host's physical CPUs.Why would you need that? You can safely assume that the ucode patch level on all cores across the system are identical - I've yet to see a machine running with different patch levels for the same reason that mixed silicon systems is a large pain in the ass and you're better off buying yourself a completely new system.
I was thinking specifically about a large multi-chassis system with lots of nodes. In that case you could well imagine different nodes/chassis having variations of CPU steppings.
But even on a single board, I don't know what limitations there are on mixing steppings between sockets, but I suspect it is possible. Obviously I wouldn't expect there to be large variations between CPUs (different models or features) - that would be a problem.
Quoted 5 lines
Quoted 3 lines
However, even aside from that, it means exporting a pile of internal details from
microcode_amdand reusing them withinmicrocode_xen. And it requires that it be done again for each vendor.Why can't you load the appropriate, unmodified microcode_<vendor> module in dom0 and let it call the hypercall?
Well, if it can call hypercalls, then to some extent it is already "modified". Either I change the source to change the wrmsrs into hypercalls, or overwrite its operations structure to change the apply_microcode function pointer (which would require - at the very least - making it non-static) to a hyper-calling function.
Either way it implies that we're delving into the Intel/AMD drivers in order to implement Xen-specific functionality, which is clearly (to me, at least) much worse than just making the Xen-specific code completely self-contained.
Quoted 13 lines
Quoted 3 lines
But all that's really needed is a dead simple "request" that loads the entire file (with a vendor-specific name) and shoves it into Xen. There's no need for any vendor-specific code beyond the filename.
But that adds this funny chunk
(quoted diff) +3
-1- if (c->x86_vendor == X86_VENDOR_INTEL) + if (xen_pv_domain()) + microcode_ops = init_xen_microcode(); + else if (c->x86_vendor == X86_VENDOR_INTEL) microcode_ops = init_intel_microcode(); else if (c->x86_vendor == X86_VENDOR_AMD) microcode_ops = init_amd_microcode();
Quoted 2 lines
which can clearly be avoided. Now imagine the if-else thing contained a dozen or more virt solutions in there, bloat!
This is just the way microcode_core.c is structured. It doesn't have a dozen or more virt solutions, but if it did, we could change the way that microcode loaders are probed for to make it cleaner. Exactly the same issue arises if you say "there might be a dozen or more cpu vendors
- bloat!". True, there might be, and we should fix it in that case. Or conversely, there might be a new CPU vendor who decides to implement the Intel microcode loading interface, in which case the tests on a specific vendor become wrong.
Quoted 2 lines
Now imagine this not only
in the microcode driver but in a bunch of other places acrossarch/x86.
Why imagine that? We're not talking about the rest of arch/x86.
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
This is exactly what I'm talking about - why copy all that checking/filtering code from baremetal to Xen instead of simply reusing it? Especially if you'd need to update the copy from time to time when baremetal changes.
The code in the kernel is in the wrong place. It has to be done in Xen. When Xen is present, the code in the kernel is redundant, not the other way around.
Ok, this says it all. Let's remove the arch code and replace it with xen-friendly version - we won't need the baremetal one anyway. Jeez!
I didn't say anything about removing the code from Linux. But the simple fact is that a kernel running under a hypervisor isn't the ultimate owner of the machine's hardware - the hypervisor is, and it controls all access to the hardware. That's deeply fundamental to the way any hypervisor works, and Xen is just one example. And the consequence of that is that Linux has to use hypervisor facilities to perform certain operations rather than using its own code.
Quoted 5 lines
Quoted 4 lines
Shrug. AFAICT the mechanism hasn't changed since it was first introduced. If there's a change, then both Linux and Xen will have to change, and most likely the same CPU vendor engineer will provide a patch for both. Xen has a good record for tracking new CPU features.
I don't think that the same CPU vendor engineer will do that, believe me!
Many individual Intel and AMD engineers contribute code to both Xen and Linux. It's not a question of belief.
Thanks,
J
H. Peter Anvin
reviewerQuoted 4 lines · Jeremy Fitzhardinge
Quoted 3 lines · H. Peter Anvin
Note: Xen may not have devices, but it is already using multiboot to load multiple modules. It could load the microcode blob that way.
Quoted 9 lines
Quoted 2 lines
That would enable an earlier loading of microcode, which is a very good thing.
Yes, that is a thought, but it would require some distro support to make sure the firmware is copied into /boot, and grub updated appropriately.
The principle advantage of updating the microcode driver is that it Just Works regardless of whether the system is booting native or Xen.
As I mentioned on IRC...
1. Xen already uses Multiboot, so it's a fairly trivial thing to add another item to the list for the boot loader to get.
2. The only reason we don't currently install microcode from the boot loader is because of the considerable complexity in adding SMP support to boot loaders, as it requires handling the APIC system.
3. Arguably on native hardware we should still load the microcode into RAM in the boot loader, and install it on the very early CPU bringup path. That means locking down some (currently) 400K of RAM to handle different combinations of CPUs, or the additional complexity of jettisoning microcode which cannot be used while still be able to deal with hotplug. I think there is a strong case for this model, which would mean moving the microcode into /boot anyway.
-hpa
Quoted 9 lines · H. Peter Anvin
As I mentioned on IRC...
1. Xen already uses Multiboot, so it's a fairly trivial thing to add another item to the list for the boot loader to get.
2. The only reason we don't currently install microcode from the boot loader is because of the considerable complexity in adding SMP support to boot loaders, as it requires handling the APIC system.
Quoted 7 lines
3. Arguably on native hardware we should still load the microcode into RAM in the boot loader, and install it on the very early CPU bringup path. That means locking down some (currently) 400K of RAM to handle different combinations of CPUs, or the additional complexity of jettisoning microcode which cannot be used while still be able to deal with hotplug. I think there is a strong case for this model, which would mean moving the microcode into /boot anyway.
If we can come up with a scheme that works for both native and Xen (or at least v. similar) that we can get distros to support, then we can work with that. That principally means getting the microcode images into /boot in a pre-digested form (binary, not text, and maybe pack the Intel and AMD files together in some extensible way - or at least give them self-describing headers).
But in the meantime it would be nice to have microcode updates under Xen within the existing model (or failing that, a little patch to prevent the spew of spurious errors when the kernel tries and fails - but it would be strongly preferable to actually update microcode).
My main concern is that I want Xen to Just Work - ideally by not requiring users/admins to do anything.
J
Quoted 25 lines · Jeremy Fitzhardinge
Quoted 9 lines · H. Peter Anvin
As I mentioned on IRC...
1. Xen already uses Multiboot, so it's a fairly trivial thing to add another item to the list for the boot loader to get.
2. The only reason we don't currently install microcode from the boot loader is because of the considerable complexity in adding SMP support to boot loaders, as it requires handling the APIC system.
Quoted 7 lines
3. Arguably on native hardware we should still load the microcode into RAM in the boot loader, and install it on the very early CPU bringup path. That means locking down some (currently) 400K of RAM to handle different combinations of CPUs, or the additional complexity of jettisoning microcode which cannot be used while still be able to deal with hotplug. I think there is a strong case for this model, which would mean moving the microcode into /boot anyway.
If we can come up with a scheme that works for both native and Xen (or at least v. similar) that we can get distros to support, then we can work with that. That principally means getting the microcode images into /boot in a pre-digested form (binary, not text, and maybe pack the Intel and AMD files together in some extensible way - or at least give them self-describing headers).
Quoted 7 lines
But in the meantime it would be nice to have microcode updates under Xen within the existing model (or failing that, a little patch to prevent the spew of spurious errors when the kernel tries and fails - but it would be strongly preferable to actually update microcode).
My main concern is that I want Xen to Just Work - ideally by not requiring users/admins to do anything.
Well, that's a noble goal, but the reality is that Xen is not even close to the point where it Just Works. So instead of slapping some weird workaround into the kernel, we really should go for the correct solution right away.
Thanks,
tglx
Borislav Petkov
maintainerQuoted 7 lines · H. Peter Anvin
3. Arguably on native hardware we should still load the microcode into RAM in the boot loader, and install it on the very early CPU bringup path. That means locking down some (currently) 400K of RAM to handle different combinations of CPUs, or the additional complexity of jettisoning microcode which cannot be used while still be able to deal with hotplug. I think there is a strong case for this model, which would mean moving the microcode into /boot anyway.
/me like it, sounds very nifty. So how do we want to do that, we add a field to the real-mode kernel header that tells us where to find the microcode image and we take it and apply the ucode somewhere in do_boot_cpu() path?
Signature · 2 lines
Regards/Gruss,
Boris.H. Peter Anvin
reviewerQuoted 9 lines · Borislav Petkov
Quoted 7 lines · H. Peter Anvin
3. Arguably on native hardware we should still load the microcode into RAM in the boot loader, and install it on the very early CPU bringup path. That means locking down some (currently) 400K of RAM to handle different combinations of CPUs, or the additional complexity of jettisoning microcode which cannot be used while still be able to deal with hotplug. I think there is a strong case for this model, which would mean moving the microcode into /boot anyway.
Quoted 5 lines
/me like it, sounds very nifty. So how do we want to do that, we add a field to the real-mode kernel header that tells us where to find the microcode image and we take it and apply the ucode somewhere in
do_boot_cpu()path?
We already have a mechanism for passing arbitrary blobs -- the linked list -- so we don't have to add a new field at all.
-hpa
Quoted 4 lines · Jeremy Fitzhardinge
work with that. That principally means getting the microcode images into /boot in a pre-digested form (binary, not text, and maybe pack the Intel and AMD files together in some extensible way - or at least give them self-describing headers).
I can get you a tool to manipulate the Intel microcode packs, and output them in binary format. I was planning to eventually switch Debian to it anyway, as microcode.ctl is slow and unsuitable for initramfs use, and it is high time we junked it. The tool is somewhat messy, and I am pretty sure my messy C is not going to get any good taste awards, but it works.
I will just remove support for /lib/firmware/intel-ucode/* from it before making it public, because I want that hideously bad idea to DIE and it would be counter-productive to actually create users for it (AFAIK, nobody ever used /lib/firmware/intel-ucode/*, so we could still fix it).
But I'd really, really appreciate if someone from Intel [that actually cares for operating system support of microcode updates] could vouch that we're allowed to do that (convert their text packs to binary packs, merge microcodes from older packs with the new to have a single pack with all microcodes in their most up-to-date revision, and distribute the resulting binary packs) before I make the tool public.
It would not be much of a problem to add AMD support to it as well (or write a separate tool), just point me to a friendly AMD engineer that will supply the docs (or point me to them if they're already public), vouch for the fact that we're allowed to unpack/merge/strip/repack AMD microcode packs, and test the tool, because I have no AMD boxes at home or at work to test it.
Quoted 2 lines
My main concern is that I want Xen to Just Work - ideally by not requiring users/admins to do anything.
I have no experience with Xen. What do I get from cpuid(0) and cpuid(1) in
dom0 when the bare metal uses Intel CPUs? And AMD CPUs? I'd like to teach
the tool to not do anything idiotic under Xen...
Signature · 4 lines
"One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh
H. Peter Anvin
reviewerQuoted 8 lines · Henrique de Moraes Holschuh
But I'd really, really appreciate if someone from Intel [that actually cares for operating system support of microcode updates] could vouch that we're allowed to do that (convert their text packs to binary packs, merge microcodes from older packs with the new to have a single pack with all microcodes in their most up-to-date revision, and distribute the resulting binary packs) before I make the tool public.
I'm trying to figure this stuff out already.
The actual conversion isn't a problem, obviously.
-hpa
Quoted 5 lines · Henrique de Moraes Holschuh
It would not be much of a problem to add AMD support to it as well (or write a separate tool), just point me to a friendly AMD engineer that will supply the docs (or point me to them if they're already public), vouch for the fact that we're allowed to unpack/merge/strip/repack AMD microcode packs, and test the tool, because I have no AMD boxes at home or at work to test it.
We already have a single container file with all the ucode patches in
it: http://www.amd64.org/support/microcode.html and the microcode driver in the kernel can look at it and take out the patches it needs based on the CPU it is running on. Is that what you had in mind?
Quoted 6 lines
Quoted 2 lines
My main concern is that I want Xen to Just Work - ideally by not requiring users/admins to do anything.
I have no experience with Xen. What do I get from cpuid(0) and cpuid(1) in
dom0 when the bare metal uses Intel CPUs? And AMD CPUs? I'd like to teach
the tool to not do anything idiotic under Xen...
Actually, if the microcode image can be provided to the hypervisor early using multiboot, it should be easy for it to figure out on what hardware it is running and apply the correct microcode without the need for dom0 to know anything about microcode, IMHO.
Signature · 8 lines
Regards/Gruss, Boris. Advanced Micro Devices GmbH Einsteinring 24, 85609 Dornach General Managers: Alberto Bozzo, Andrew Bowd Registration: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632
Quoted 11 lines · Borislav Petkov
Quoted 5 lines · Henrique de Moraes Holschuh
It would not be much of a problem to add AMD support to it as well (or write a separate tool), just point me to a friendly AMD engineer that will supply the docs (or point me to them if they're already public), vouch for the fact that we're allowed to unpack/merge/strip/repack AMD microcode packs, and test the tool, because I have no AMD boxes at home or at work to test it.
We already have a single container file with all the ucode patches in
it: http://www.amd64.org/support/microcode.html and the microcode driver in the kernel can look at it and take out the patches it needs based on the CPU it is running on. Is that what you had in mind?
Validate the container file in userspace, let the user list available microcode updates, let the user merge multiple container files into a new one with just the most up-to-date microcodes for each CPU, optionally filtered for the CPUs currently online, or to the ones specificed in the command line.
I have a tool that does that for Intel, based on their documentation and also on the Linux driver.
However, since AMD has so few microcodes in that file and it is so small, that's probably not useful at all right now. Maybe in a few years :-)
Quoted 11 lines
Quoted 6 lines
Quoted 2 lines
My main concern is that I want Xen to Just Work - ideally by not requiring users/admins to do anything.
I have no experience with Xen. What do I get from cpuid(0) and cpuid(1) in
dom0 when the bare metal uses Intel CPUs? And AMD CPUs? I'd like to teach
the tool to not do anything idiotic under Xen...Actually, if the microcode image can be provided to the hypervisor early using multiboot, it should be easy for it to figure out on what hardware it is running and apply the correct microcode without the need for dom0 to know anything about microcode, IMHO.
I'd still appreciate that information.
Signature · 4 lines
"One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh
Borislav Petkov
maintainerQuoted 10 lines · H. Peter Anvin
Quoted 9 lines · Borislav Petkov
Quoted 7 lines · H. Peter Anvin
3. Arguably on native hardware we should still load the microcode into RAM in the boot loader, and install it on the very early CPU bringup path. That means locking down some (currently) 400K of RAM to handle different combinations of CPUs, or the additional complexity of jettisoning microcode which cannot be used while still be able to deal with hotplug. I think there is a strong case for this model, which would mean moving the microcode into /boot anyway.
Quoted 8 lines
Quoted 5 lines
/me like it, sounds very nifty. So how do we want to do that, we add a field to the real-mode kernel header that tells us where to find the microcode image and we take it and apply the ucode somewhere in
do_boot_cpu()path?We already have a mechanism for passing arbitrary blobs -- the linked list -- so we don't have to add a new field at all.
So, after staring at grub legacy sources a bit, we could load the microcode image using the grub module's mechanism:
kernel /...
module /boot/microcode.gz type=SETUP_MICROCODE # this is looked at by parse_setup_data()
and let grub write the pointer into setup_data passed through the kernel header.
This would mean that we need to add support to a bunch of boot loaders used currently, no? Or is there an even better way?
Signature · 2 lines
Regards/Gruss, Boris.
H. Peter Anvin
reviewerQuoted 7 lines · Borislav Petkov
Quoted 3 lines
We already have a mechanism for passing arbitrary blobs -- the linked list -- so we don't have to add a new field at all.
So, after staring at grub legacy sources a bit, we could load the microcode image using the grub module's mechanism:
Quoted 9 lines
kernel /...
module/boot/microcode.gztype=SETUP_MICROCODE # this is looked at byparse_setup_data()and let grub write the pointer into
setup_datapassed through the kernel header.This would mean that we need to add support to a bunch of boot loaders used currently, no? Or is there an even better way?
We would need to add boot loader support, yes.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.