Improve firmware loading times on AMD and Intel
Subsystems (2)
cover Improve firmware loading times on AMD and Intel
If no firmware is found on the system that matches the processor, the microcode module can take hours to load. For example on recent kernels when loading the microcode module on an Intel system:
[ 239.532116] microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x413 [ 299.693447] microcode: CPU1 sig=0x306e4, pf=0x1, revision=0x413 [ 359.821972] microcode: CPU2 sig=0x306e4, pf=0x1, revision=0x413 [ 419.960263] microcode: CPU3 sig=0x306e4, pf=0x1, revision=0x413 [ 480.090024] microcode: CPU4 sig=0x306e4, pf=0x1, revision=0x413 ... [ 2825.151364] microcode: CPU43 sig=0x306e4, pf=0x1, revision=0x413 [ 2885.280863] microcode: CPU44 sig=0x306e4, pf=0x1, revision=0x413 [ 2945.410719] microcode: CPU45 sig=0x306e4, pf=0x1, revision=0x413 [ 3005.540541] microcode: CPU46 sig=0x306e4, pf=0x1, revision=0x413 [ 3065.670405] microcode: CPU47 sig=0x306e4, pf=0x1, revision=0x413
...
etc.
Since this takes 1 minute per cpu, the microcode module requires two hours to load on a 120 cpu Intel box.
Similarly there is a 60 second "hang" when loading the AMD module, which isn't as bad as the Intel situation, but it is a noticeable delay in the system boot.
Using request_firmware_nowait() seems more appropriate here and then we can avoid these delays, resulting in very quick load times for the microcode.
Prarit Bhargava (2):
firmware, fix request_firmware_nowait() freeze with no uevent intel_microcode, Fix long microcode load time when firmware file is
missing
arch/x86/include/asm/microcode.h | 7 ++++ arch/x86/kernel/microcode_amd.c | 79 +++++++++++++++++++++++++++---------- arch/x86/kernel/microcode_core.c | 7 ++++ arch/x86/kernel/microcode_intel.c | 67 +++++++++++++++++++++++++------ drivers/base/firmware_class.c | 6 ++- 5 files changed, 132 insertions(+), 34 deletions(-)
Signature · 1 line
1.7.9.3
PATCH 1/2 firmware, fix request_firmware_nowait() freeze with no uevent
If request_firmware_nowait() is called with uevent == NULL, the firmware completion is never marked complete resulting in a hang in the process.
If uevent is undefined, that means we're not waiting on anything and the process should just clean up and complete. While we're at it, add a debug dev_dbg() to indicate that the FW has not been found.
--- drivers/base/firmware_class.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/base/firmware_class.c +5 -1
@@ -335,7 +335,8 @@ static bool fw_get_filesystem_firmware(struct device *device,set_bit (FW_STATUS_DONE , &buf->status);complete_all (&buf->completion);mutex_unlock (&fw_lock); - } + }else +dev_dbg (device,"firmware: %s not found\n" , buf->fw_id);return success; }
@@ -886,6 +887,9 @@ static int _request_firmware_load(struct firmware_priv *fw_priv, bool uevent,schedule_delayed_work (&fw_priv->timeout_work, timeout);kobject_uevent (&fw_priv->dev.kobj,KOBJ_ADD ); + }else { +/* if there is no uevent then just cleanup */ +schedule_delayed_work (&fw_priv->timeout_work,0 ); }wait_for_completion (&buf->completion);
Signature · 1 line
1.7.9.3
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 31 lines · Prarit Bhargava
If
request_firmware_nowait()is called with uevent == NULL, the firmware completion is never marked complete resulting in a hang in the process.If uevent is undefined, that means we're not waiting on anything and the process should just clean up and complete. While we're at it, add a debug
dev_dbg()to indicate that the FW has not been found.Signed-off-by Prarit Bhargava <[email protected]>--- drivers/base/firmware_class.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)drivers/base/firmware_class.c +2
-1@@ -335,7 +335,8 @@ static bool fw_get_filesystem_firmware(struct device *device,set_bit (FW_STATUS_DONE , &buf->status);complete_all (&buf->completion);mutex_unlock (&fw_lock); - } + }else +dev_dbg (device,"firmware: %s not found\n" , buf->fw_id);return success; }
drivers/base/firmware_class.c static int _request_firmware_load(struct firmware_priv *fw_priv, bool uevent, · line 893
Quoted 7 lines
drivers/base/firmware_class.c +3
-0@@ -886,6 +887,9 @@ static int _request_firmware_load(struct firmware_priv *fw_priv, bool uevent,schedule_delayed_work (&fw_priv->timeout_work, timeout);kobject_uevent (&fw_priv->dev.kobj,KOBJ_ADD ); + }else { +/* if there is no uevent then just cleanup */ +schedule_delayed_work (&fw_priv->timeout_work,0 ); }
This may not a good idea and might break current NOHOTPLUG users, and how can you make sure the user space application can complete the request during the timeout time?
Thanks,
--
Ming Lei
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 34 lines · Ming Lei
Quoted 33 lines · Prarit Bhargava
If
request_firmware_nowait()is called with uevent == NULL, the firmware completion is never marked complete resulting in a hang in the process.If uevent is undefined, that means we're not waiting on anything and the process should just clean up and complete. While we're at it, add a debug
dev_dbg()to indicate that the FW has not been found.Signed-off-by Prarit Bhargava <[email protected]>--- drivers/base/firmware_class.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)drivers/base/firmware_class.c +2
-1@@ -335,7 +335,8 @@ static bool fw_get_filesystem_firmware(struct device *device,set_bit (FW_STATUS_DONE , &buf->status);complete_all (&buf->completion);mutex_unlock (&fw_lock); - } + }else +dev_dbg (device,"firmware: %s not found\n" , buf->fw_id);return success; }@@ -886,6 +887,9 @@ static int _request_firmware_load(struct firmware_priv *fw_priv, bool uevent,schedule_delayed_work (&fw_priv->timeout_work, timeout);
Quoted 8 lines
drivers/base/firmware_class.c static int _request_firmware_load(struct firmware_priv *fw_priv, bool uevent, · line 893
Quoted 5 lines
drivers/base/firmware_class.c +3
-0kobject_uevent (&fw_priv->dev.kobj,KOBJ_ADD ); + }else { +/* if there is no uevent then just cleanup */ +schedule_delayed_work (&fw_priv->timeout_work,0 ); }This may not a good idea and might break current NOHOTPLUG users,
Ming,
The code is broken for all callers of request_firmware_nowait() with NOHOTPLUG and CONFIG_FW_LOADER_USER_HELPER=y. AFAICT with the two existing cases of this usage in the kernel, both are broken and both are attempting to do the same thing that I'm doing in the x86 microcode ATM.
This is the situation as I understand it and please correct me if I'm wrong about the execution path. If I call request_firmware_nowait() with NOHOTPLUG I am essentially saying that there is no uevent associated with this firmware load; that is uevent = 0. request_firmware_work_func() is called as scheduled task, which results in a call to _request_firmware(). _request_firmware() first calls _request_firmware_prepare() which eventually results in a call to __allocate_fw_buf() which does an init_completion(&buf->completion).
Returning back up the stack to _request_firmware() we eventually call fw_get_filesystem_firmware(). _If the firmware does not exist_ success is false and the if (success) loop is not executed, and it is important to note that the complete_all(&buf->completion) is _not_ called. fw_get_filesystem_firmware() returns an error so that fw_load_from_user_helper() is called from _request_firmware().
fw_load_from_user_helper() eventually calls _request_firmware_load() and this is where we get into a problem. fw_load_from_user_helper() calls all the file creation, etc., and then hits this chunk of code:
if (uevent) {
dev_set_uevent_suppress(f_dev, false);
dev_dbg(f_dev, "firmware: requesting %s\n", buf->fw_id);
if (timeout != MAX_SCHEDULE_TIMEOUT)
schedule_delayed_work(&fw_priv->timeout_work, timeout);
kobject_uevent(&fw_priv->dev.kobj, KOBJ_ADD);
}
wait_for_completion(&buf->completion);As I previously said, we've been called with NOHOTPLUG, ie) uevent = 0. That means we skip down to the wait_for_completion(&buf->completion) ... and we wait ... forever.
I can reproduce this by using a Dell PE 1850 & the dell_rbu module by doing the following:
insmod dell_rbu.ko
echo init > /sys/devices/platform/dell_rbu/image_type lsmod | grep dell_rbu
(after an hour)
[root@dell-pe1850-04 dell_rbu]# lsmod | grep dell_rbudell_rbu 14315 1
[root@dell-pe1850-04 dell_rbu]#
^^^ that use count is left because the thread is waiting with an existing module ref count. For kicks I put a printk in the dell_rbu code or instrument the _request_firmware() code and did a reboot. Since the completions are finished on system shutdown, I see the code continue to execute at the end of boot.
Quoted 2 lines
and how can you make sure the user space application can complete the request during the timeout time?
I see that your question really comes down to "are there additional synchronizations needed in the two drivers that already call the code this way?"
I realize that the answer to that is yes and I'll fix those up in a v2. It
should be trivial to make those changes AFAICT. I've introduced some additional synchronization via a completion in the x86 microcode and will likely have to do something similar in the other drivers ... although it may be easier to just have the firmware code do all the synchronization. I'll look into it.
Hope this explains things a bit better,
P.
Quoted 4 lines
Thanks,
--
Ming Lei
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 83 lines · Prarit Bhargava
Quoted 34 lines · Ming Lei
Quoted 33 lines · Prarit Bhargava
If
request_firmware_nowait()is called with uevent == NULL, the firmware completion is never marked complete resulting in a hang in the process.If uevent is undefined, that means we're not waiting on anything and the process should just clean up and complete. While we're at it, add a debug
dev_dbg()to indicate that the FW has not been found.Signed-off-by Prarit Bhargava <[email protected]>--- drivers/base/firmware_class.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)drivers/base/firmware_class.c +2
-1@@ -335,7 +335,8 @@ static bool fw_get_filesystem_firmware(struct device *device,set_bit (FW_STATUS_DONE , &buf->status);complete_all (&buf->completion);mutex_unlock (&fw_lock); - } + }else +dev_dbg (device,"firmware: %s not found\n" , buf->fw_id);return success; }@@ -886,6 +887,9 @@ static int _request_firmware_load(struct firmware_priv *fw_priv, bool uevent,schedule_delayed_work (&fw_priv->timeout_work, timeout);Quoted 8 lines
drivers/base/firmware_class.c static int _request_firmware_load(struct firmware_priv *fw_priv, bool uevent, · line 893
Quoted 5 lines
drivers/base/firmware_class.c +3
-0kobject_uevent (&fw_priv->dev.kobj,KOBJ_ADD ); + }else { +/* if there is no uevent then just cleanup */ +schedule_delayed_work (&fw_priv->timeout_work,0 ); }This may not a good idea and might break current NOHOTPLUG users,
Ming,
The code is broken for all callers of
request_firmware_nowait()with NOHOTPLUG andCONFIG_FW_LOADER_USER_HELPER=y. AFAICT with the two existing cases of this usage in the kernel, both are broken and both are attempting to do the same thing that I'm doing in the x86 microcode ATM.This is the situation as I understand it and please correct me if I'm wrong about the execution path. If I call
request_firmware_nowait()with NOHOTPLUG I am essentially saying that there is no uevent associated with this firmware load; that is uevent = 0.request_firmware_work_func()is called as scheduled task, which results in a call to_request_firmware()._request_firmware()first calls_request_firmware_prepare()which eventually results in a call to__allocate_fw_buf()which does aninit_completion(&buf->completion).Returning back up the stack to
_request_firmware()we eventually callfw_get_filesystem_firmware()._Ifthe firmware does notexist_success is false and the if (success) loop is not executed, and it is important to note that thecomplete_all(&buf->completion)is _not_ called.fw_get_filesystem_firmware()returns an error so thatfw_load_from_user_helper()is called from_request_firmware().
fw_load_from_user_helper()eventually calls_request_firmware_load()and this is where we get into a problem.fw_load_from_user_helper()calls all the file creation, etc., and then hits this chunk of code:if (uevent) { dev_set_uevent_suppress(f_dev, false); dev_dbg(f_dev, "firmware: requesting %s\n", buf->fw_id); if (timeout != MAX_SCHEDULE_TIMEOUT) schedule_delayed_work(&fw_priv->timeout_work, timeout); kobject_uevent(&fw_priv->dev.kobj, KOBJ_ADD); } wait_for_completion(&buf->completion);
Quoted 3 lines
As I previously said, we've been called with NOHOTPLUG, ie) uevent = 0. That means we skip down to the
wait_for_completion(&buf->completion)... and we wait ... forever.
Yes, it is exactly the previous design on NOHOTPLUG, because firmware loader has to wait for the handling from user space, and no one can predict when userspace comes because of no notification. For example, the userspace may be 'some inputting from shell by someone once he is free', :-) so it is difficult to set a timeout explicitly for the handling.
But the requests can be killed before suspend & shutdown, so it is still OK.
That is why NOHOTPLUG isn't encouraged to be taken, actually I don't suggest you to do that too, :-)
You need to make sure your approach won't break micro-code update application in current/previous distributions.
Quoted 10 lines
I can reproduce this by using a Dell PE 1850 & the
dell_rbumodule by doing the following:insmod dell_rbu.ko
echo init >/sys/devices/platform/dell_rbu/image_typelsmod | grepdell_rbu(after an hour)
Quoted 8 lines
[root@dell-pe1850-04 dell_rbu]# lsmod | grep
dell_rbudell_rbu14315 1
[root@dell-pe1850-04 dell_rbu]#^^^ that use count is left because the thread is waiting with an existing module ref count. For kicks I put a printk in the
dell_rbucode or instrument the_request_firmware()code and did a reboot. Since the completions are finished on system shutdown, I see the code continue to execute at the end of boot.
Right, so no obvious problem from user view, isn't it?
Quoted 12 lines
Quoted 3 lines
and how can you make sure the user space application can complete the request during the timeout time?
I see that your question really comes down to "are there additional synchronizations needed in the two drivers that already call the code this way?"
I realize that the answer to that is yes and I'll fix those up in a v2. It
should be trivial to make those changes AFAICT. I've introduced some additional synchronization via a completion in the x86 microcode and will likely have to do something similar in the other drivers ... although it may be easier to just have the firmware code do all the synchronization. I'll look into it.
Quoted 1 line
Hope this explains things a bit better,
As I said above, setting a timeout may be not ok, and may break current two drivers.
Thanks,
--
Ming Lei
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 95 lines · Ming Lei
Quoted 83 lines · Prarit Bhargava
Quoted 34 lines · Ming Lei
Quoted 33 lines · Prarit Bhargava
If
request_firmware_nowait()is called with uevent == NULL, the firmware completion is never marked complete resulting in a hang in the process.If uevent is undefined, that means we're not waiting on anything and the process should just clean up and complete. While we're at it, add a debug
dev_dbg()to indicate that the FW has not been found.Signed-off-by Prarit Bhargava <[email protected]>--- drivers/base/firmware_class.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)drivers/base/firmware_class.c +2
-1@@ -335,7 +335,8 @@ static bool fw_get_filesystem_firmware(struct device *device,set_bit (FW_STATUS_DONE , &buf->status);complete_all (&buf->completion);mutex_unlock (&fw_lock); - } + }else +dev_dbg (device,"firmware: %s not found\n" , buf->fw_id);return success; }@@ -886,6 +887,9 @@ static int _request_firmware_load(struct firmware_priv *fw_priv, bool uevent,schedule_delayed_work (&fw_priv->timeout_work, timeout);Quoted 8 lines
drivers/base/firmware_class.c static int _request_firmware_load(struct firmware_priv *fw_priv, bool uevent, · line 893
Quoted 5 lines
drivers/base/firmware_class.c +3
-0kobject_uevent (&fw_priv->dev.kobj,KOBJ_ADD ); + }else { +/* if there is no uevent then just cleanup */ +schedule_delayed_work (&fw_priv->timeout_work,0 ); }This may not a good idea and might break current NOHOTPLUG users,
Ming,
The code is broken for all callers of
request_firmware_nowait()with NOHOTPLUG andCONFIG_FW_LOADER_USER_HELPER=y. AFAICT with the two existing cases of this usage in the kernel, both are broken and both are attempting to do the same thing that I'm doing in the x86 microcode ATM.This is the situation as I understand it and please correct me if I'm wrong about the execution path. If I call
request_firmware_nowait()with NOHOTPLUG I am essentially saying that there is no uevent associated with this firmware load; that is uevent = 0.request_firmware_work_func()is called as scheduled task, which results in a call to_request_firmware()._request_firmware()first calls_request_firmware_prepare()which eventually results in a call to__allocate_fw_buf()which does aninit_completion(&buf->completion).Returning back up the stack to
_request_firmware()we eventually callfw_get_filesystem_firmware()._Ifthe firmware does notexist_success is false and the if (success) loop is not executed, and it is important to note that thecomplete_all(&buf->completion)is _not_ called.fw_get_filesystem_firmware()returns an error so thatfw_load_from_user_helper()is called from_request_firmware().
fw_load_from_user_helper()eventually calls_request_firmware_load()and this is where we get into a problem.fw_load_from_user_helper()calls all the file creation, etc., and then hits this chunk of code:if (uevent) { dev_set_uevent_suppress(f_dev, false); dev_dbg(f_dev, "firmware: requesting %s\n", buf->fw_id); if (timeout != MAX_SCHEDULE_TIMEOUT) schedule_delayed_work(&fw_priv->timeout_work, timeout); kobject_uevent(&fw_priv->dev.kobj, KOBJ_ADD); } wait_for_completion(&buf->completion);Quoted 3 lines
As I previously said, we've been called with NOHOTPLUG, ie) uevent = 0. That means we skip down to the
wait_for_completion(&buf->completion)... and we wait ... forever.Yes, it is exactly the previous design on NOHOTPLUG, because firmware loader has to wait for the handling from user space, and no one can predict when userspace comes because of no notification. For example, the userspace may be 'some inputting from shell by someone once he is free', :-) so it is difficult to set a timeout explicitly for the handling.
Quoted 5 lines
But the requests can be killed before suspend & shutdown, so it is still OK.
That is why NOHOTPLUG isn't encouraged to be taken, actually I don't suggest you to do that too, :-)
Okay ... I can certainly switch to HOTPLUG.
Quoted 3 lines
You need to make sure your approach won't break micro-code update application in current/previous distributions.
I've tested the following distributions today on a Dell PE 1850: Ubuntu, SuSe, Linux Mint, and of course Fedora. I do not see any issues with either the microcode update or the dell_rbu driver. Unfortunately I do not have access to a system that uses the lattice-ecp3-config, however, from code inspection it looks like the driver looks at a specific place for the FW update and then applies it via the call function in request_firmware_nowait() so it looks like it is solid too.
I think maybe this patchset should be split into two separate submits, one for the microcode and the second to figure out if the code really should wait indefinitely. AFAICT neither use case in the kernel expects an indefinite wait.
P.
Quoted 15 lines
Quoted 15 lines
I can reproduce this by using a Dell PE 1850 & the
dell_rbumodule by doing the following:insmod dell_rbu.ko
echo init >/sys/devices/platform/dell_rbu/image_typelsmod | grepdell_rbu(after an hour)
[root@dell-pe1850-04 dell_rbu]# lsmod | grep
dell_rbudell_rbu14315 1
[root@dell-pe1850-04 dell_rbu]#
Quoted 6 lines
Quoted 4 lines
^^^ that use count is left because the thread is waiting with an existing module ref count. For kicks I put a printk in the
dell_rbucode or instrument the_request_firmware()code and did a reboot. Since the completions are finished on system shutdown, I see the code continue to execute at the end of boot.Right, so no obvious problem from user view, isn't it?
Well, there is an issue that it is possible that the dell_rbu driver attempts to load the update BEFORE the update is available. I have written some additional code to fix that.
P.
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 5 lines · Prarit Bhargava
Quoted 3 lines · Ming Lei
That is why NOHOTPLUG isn't encouraged to be taken, actually I don't suggest you to do that too, :-)
Okay ... I can certainly switch to HOTPLUG.
OK, that should be the right approach.
Quoted 8 lines
Quoted 4 lines
You need to make sure your approach won't break micro-code update application in current/previous distributions.
I've tested the following distributions today on a Dell PE 1850: Ubuntu, SuSe, Linux Mint, and of course Fedora. I do not see any issues with either the microcode update or the
dell_rbudriver. Unfortunately I do not have access to
Actually I am wondering if your tests are enough because kernel can't break user-space, which means lots of previous old version distributions should surely be covered, :-)
If you keep HOTPLUG, only change to request_firmware_nowait(), that should be OK since the loading protocol between userspace and kernel won't change wrt. microcode updating.
Quoted 8 lines
a system that uses the lattice-ecp3-config, however, from code inspection it looks like the driver looks at a specific place for the FW update and then applies it via the call function in
request_firmware_nowait()so it looks like it is solid too.I think maybe this patchset should be split into two separate submits, one for the microcode and the second to figure out if the code really should wait indefinitely. AFAICT neither use case in the kernel expects an indefinite wait.
If you switch to HOTPLUG, you needn't worry about waiting indefinitely, need you?
Thanks,
--
Ming Lei
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 14 lines · Ming Lei
Quoted 5 lines · Prarit Bhargava
Quoted 3 lines · Ming Lei
That is why NOHOTPLUG isn't encouraged to be taken, actually I don't suggest you to do that too, :-)
Okay ... I can certainly switch to HOTPLUG.
OK, that should be the right approach.
Quoted 5 lines
Quoted 4 lines
You need to make sure your approach won't break micro-code update application in current/previous distributions.
Quoted 7 lines
Quoted 3 lines
I've tested the following distributions today on a Dell PE 1850: Ubuntu, SuSe, Linux Mint, and of course Fedora. I do not see any issues with either the microcode update or the
dell_rbudriver. Unfortunately I do not have access toActually I am wondering if your tests are enough because kernel can't break user-space, which means lots of previous old version distributions should surely be covered, :-)
I've tested an old version of Suse and a few older RHEL versions for kicks. No problems. I'm testing an older version of Ubuntu ATM and will update with details (it doesn't look like it does anything different FWIW so I'm not worried).
Quoted 10 lines
If you keep HOTPLUG, only change to
request_firmware_nowait(), that should be OK since the loading protocol between userspace and kernel won't change wrt. microcode updating.Quoted 5 lines
a system that uses the lattice-ecp3-config, however, from code inspection it looks like the driver looks at a specific place for the FW update and then applies it via the call function in
request_firmware_nowait()so it looks like it is solid too.
Quoted 6 lines
Quoted 3 lines
I think maybe this patchset should be split into two separate submits, one for the microcode and the second to figure out if the code really should wait indefinitely. AFAICT neither use case in the kernel expects an indefinite wait.
If you switch to HOTPLUG, you needn't worry about waiting indefinitely, need you?
Nope ... I'll modify the code and retest.
Thanks for the input Ming :)
P.
Quoted 4 lines
Thanks,
--
Ming Lei
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 39 lines · Prarit Bhargava
Quoted 14 lines · Ming Lei
Quoted 5 lines · Prarit Bhargava
Quoted 3 lines · Ming Lei
That is why NOHOTPLUG isn't encouraged to be taken, actually I don't suggest you to do that too, :-)
Okay ... I can certainly switch to HOTPLUG.
OK, that should be the right approach.
Quoted 5 lines
Quoted 4 lines
You need to make sure your approach won't break micro-code update application in current/previous distributions.
Quoted 7 lines
Quoted 3 lines
I've tested the following distributions today on a Dell PE 1850: Ubuntu, SuSe, Linux Mint, and of course Fedora. I do not see any issues with either the microcode update or the
dell_rbudriver. Unfortunately I do not have access toActually I am wondering if your tests are enough because kernel can't break user-space, which means lots of previous old version distributions should surely be covered, :-)
I've tested an old version of Suse and a few older RHEL versions for kicks. No problems. I'm testing an older version of Ubuntu ATM and will update with details (it doesn't look like it does anything different FWIW so I'm not worried).
Quoted 10 lines
If you keep HOTPLUG, only change to
request_firmware_nowait(), that should be OK since the loading protocol between userspace and kernel won't change wrt. microcode updating.Quoted 5 lines
a system that uses the lattice-ecp3-config, however, from code inspection it looks like the driver looks at a specific place for the FW update and then applies it via the call function in
request_firmware_nowait()so it looks like it is solid too.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I think maybe this patchset should be split into two separate submits, one for the microcode and the second to figure out if the code really should wait indefinitely. AFAICT neither use case in the kernel expects an indefinite wait.
If you switch to HOTPLUG, you needn't worry about waiting indefinitely, need you?
Nope ... I'll modify the code and retest.
After all this I completely forgot the problem I'm trying to solve here. The issue is that with HOTPLUG & request_microcode_nowait(), if the microcode image is not found (that is the file is not found on disk), then EACH cpu waits 1 minute and it takes 2 hours for a 120 cpu box to load the microcode module.
Which is terrible... so HOTPLUG doesn't work here.
Let me back up Ming and see if you have a better solution for me. I have a system that does not have the x86 microcode loaded on disk. I use the microcode module which calls request_firmware_nowait() to load the microcode image and I want it done as fast as possible. The microcode loader does not have a uevent so I'm not waiting on userspace for completion.
How do I avoid the 60 second delay/cpu introduced in the microcode path? I don't see one. If I use HOTPLUG I'm waiting 60 seconds. If I use NOHOTPLUG AFAICT the loading function never will return. AFAICT the same issue arises with the dell_rbu code -- it appears to never load the dell_rbu firmware.
P.
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 14 lines · Prarit Bhargava
After all this I completely forgot the problem I'm trying to solve here. The issue is that with HOTPLUG &
request_microcode_nowait(), if the microcode image is not found (that is the file is not found on disk), then EACH cpu waits 1 minute and it takes 2 hours for a 120 cpu box to load the microcode module.Which is terrible... so HOTPLUG doesn't work here.
Let me back up Ming and see if you have a better solution for me. I have a system that does not have the x86 microcode loaded on disk. I use the microcode module which calls
request_firmware_nowait()to load the microcode image and I want it done as fast as possible. The microcode loader does not have a uevent so I'm not waiting on userspace for completion.
Quoted 4 lines
How do I avoid the 60 second delay/cpu introduced in the microcode path? I don't see one. If I use HOTPLUG I'm waiting 60 seconds. If I use NOHOTPLUG AFAICT the loading function never will return. AFAICT the same issue arises with the
dell_rbucode -- it appears to never load thedell_rbufirmware.
As I said, the 60 second delay is from udev, so that is the root cause.
There are some workarounds for your reference:
- fix udev
- disable CONFIG_FW_LOADER_USER_HELPER if you are sure
the microcode image is under the default firmware search paths, which are defined in drivers/base/firmware_class.c
- fake a empty latest microcode image under your firmware path
- use request_firmware_nowait(HOTPLUG) to request firmware from user space, and apply the microcode at a appropriate time after the request is completed, and the approach is what we are discussing.
Thanks,
--
Ming Lei
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 16 lines · Ming Lei
Quoted 14 lines · Prarit Bhargava
After all this I completely forgot the problem I'm trying to solve here. The issue is that with HOTPLUG &
request_microcode_nowait(), if the microcode image is not found (that is the file is not found on disk), then EACH cpu waits 1 minute and it takes 2 hours for a 120 cpu box to load the microcode module.Which is terrible... so HOTPLUG doesn't work here.
Let me back up Ming and see if you have a better solution for me. I have a system that does not have the x86 microcode loaded on disk. I use the microcode module which calls
request_firmware_nowait()to load the microcode image and I want it done as fast as possible. The microcode loader does not have a uevent so I'm not waiting on userspace for completion.
Quoted 6 lines
Quoted 4 lines
How do I avoid the 60 second delay/cpu introduced in the microcode path? I don't see one. If I use HOTPLUG I'm waiting 60 seconds. If I use NOHOTPLUG AFAICT the loading function never will return. AFAICT the same issue arises with the
dell_rbucode -- it appears to never load thedell_rbufirmware.As I said, the 60 second delay is from udev, so that is the root cause.
Okay, so then my other option is to use NOHOTPLUG. I've correctly modified it to return and not wait for a NULL uevent. The synchronization between the cont function return and the actual application of the firmware is done in the driver (in my 2/2 patch) where I've used a completion struct. ... Am I still missing something at this point?
I also have to make that change to the dell_rbu code because it too, is broken the same way. That is, I can load the dell_rbu module and it just hangs without applying the firmware.
I'll submit a new version of these patches and we can continue from there.
P.
Quoted 3 lines
There are some workarounds for your reference:
These are workarounds for an issue that arises in the kernel.
P.
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 24 lines · Prarit Bhargava
Quoted 21 lines · Ming Lei
Quoted 18 lines · Prarit Bhargava
After all this I completely forgot the problem I'm trying to solve here. The issue is that with HOTPLUG &
request_microcode_nowait(), if the microcode image is not found (that is the file is not found on disk), then EACH cpu waits 1 minute and it takes 2 hours for a 120 cpu box to load the microcode module.Which is terrible... so HOTPLUG doesn't work here.
Let me back up Ming and see if you have a better solution for me. I have a system that does not have the x86 microcode loaded on disk. I use the microcode module which calls
request_firmware_nowait()to load the microcode image and I want it done as fast as possible. The microcode loader does not have a uevent so I'm not waiting on userspace for completion.How do I avoid the 60 second delay/cpu introduced in the microcode path? I don't see one. If I use HOTPLUG I'm waiting 60 seconds. If I use NOHOTPLUG AFAICT the loading function never will return. AFAICT the same issue arises with the
dell_rbucode -- it appears to never load thedell_rbufirmware.
Quoted 7 lines
Quoted 1 line
As I said, the 60 second delay is from udev, so that is the root cause.
Okay, so then my other option is to use NOHOTPLUG. I've correctly modified it to return and not wait for a NULL uevent. The synchronization between the cont function return and the actual application of the firmware is done in the driver (in my 2/2 patch) where I've used a completion struct. ... Am I still missing something at this point?
If you plan to use NOHOTPLUG to stop sending uevent to user space, you need to make sure all distributions(include old ones) do not require the notification previously, and you'd better to explain the microcode upgrading principle in a bit detail so that we can make sure it won't break the loading protocol between kernel and user space, at least current code is using request_fimrware() and the uevent is surely sent to userspace, right?
Quoted 4 lines
I also have to make that change to the
dell_rbucode because it too, is broken the same way. That is, I can load thedell_rbumodule and it just hangs without applying the firmware.
Because userspace does not handle the request and write fw data to driver, how can you expect the driver to apply firmware?
Anyway, you need Cc dell_rbu guys to make sure the change.
Quoted 8 lines
I'll submit a new version of these patches and we can continue from there.
P.
Quoted 3 lines
There are some workarounds for your reference:
These are workarounds for an issue that arises in the kernel.
Sorry, I don't understand, the root cause is surely from udev.
Thanks,
--
Ming Lei
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
In reply to Prarit BhargavaQuoted 4 lines · Prarit Bhargava
After all this I completely forgot the problem I'm trying to solve here. The issue is that with HOTPLUG &
request_microcode_nowait(), if the microcode image is not found (that is the file is not found on disk), then EACH cpu waits 1 minute and it takes 2 hours for a 120 cpu box to load the microcode module.
The proper fix seems to be teaching the concept of negative caching to the microcode core/drivers, as it was pointed out elsewhere in the thread. Negative caching should have a lifetime of "the current update-all-cores request".
This would fix the absurd compound timeout delays, as on most systems it will result in just one timeout (the first one).
That first timeout can be fixed by the user if they disable the userspace firmware loader helper. IMHO that might well be the best choice, as it is already the way forward.
Signature · 4 lines
"One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh
Re: [PATCH 1/2] firmware, fix request_firmware_nowait() freeze with no uevent
Quoted 11 lines · Henrique de Moraes Holschuh
Quoted 4 lines · Prarit Bhargava
After all this I completely forgot the problem I'm trying to solve here. The issue is that with HOTPLUG &
request_microcode_nowait(), if the microcode image is not found (that is the file is not found on disk), then EACH cpu waits 1 minute and it takes 2 hours for a 120 cpu box to load the microcode module.The proper fix seems to be teaching the concept of negative caching to the microcode core/drivers, as it was pointed out elsewhere in the thread. Negative caching should have a lifetime of "the current update-all-cores request".
Yes, I'm implementing v2 to do this already; caching the microcode is obvious. I was actually looking at the code to see if there was a reason that each processor needs to do a load request but cannot see one. I'm modifying the microcode driver to do this, as I said, in v2.
Quoted 6 lines
This would fix the absurd compound timeout delays, as on most systems it will result in just one timeout (the first one).
That first timeout can be fixed by the user if they disable the userspace firmware loader helper. IMHO that might well be the best choice, as it is already the way forward.
The problem with that is I may have a configuration which depends on having the userspace firmware loader helper for a device, but not the processors so IMO it isn't a complete solution.
I've also toyed with the idea that there should be a request_firmware_timeout() in which a timeout for HOTPLUG can be specified.
P.
PATCH 2/2 intel_microcode, Fix long microcode load time when firmware file is missing
In reply to Prarit BhargavaIf no firmware is found on the system that matches the processor, the microcode module can take hours to load. For example on recent kernels when loading the microcode module on an Intel system:
[ 239.532116] microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x413 [ 299.693447] microcode: CPU1 sig=0x306e4, pf=0x1, revision=0x413 [ 359.821972] microcode: CPU2 sig=0x306e4, pf=0x1, revision=0x413 [ 419.960263] microcode: CPU3 sig=0x306e4, pf=0x1, revision=0x413 [ 480.090024] microcode: CPU4 sig=0x306e4, pf=0x1, revision=0x413 ... [ 2825.151364] microcode: CPU43 sig=0x306e4, pf=0x1, revision=0x413 [ 2885.280863] microcode: CPU44 sig=0x306e4, pf=0x1, revision=0x413 [ 2945.410719] microcode: CPU45 sig=0x306e4, pf=0x1, revision=0x413 [ 3005.540541] microcode: CPU46 sig=0x306e4, pf=0x1, revision=0x413 [ 3065.670405] microcode: CPU47 sig=0x306e4, pf=0x1, revision=0x413
...
etc.
Similarly there is a 60 second "hang" when loading the AMD module, which isn't as bad as the Intel situation, but it is a noticeable delay in the system boot and can be improved upon.
Using request_firmware_nowait() seems more appropriate here and then we can avoid these delays, resulting in very quick load times for the microcode.
--- arch/x86/include/asm/microcode.h | 7 ++++ arch/x86/kernel/microcode_amd.c | 79 +++++++++++++++++++++++++++---------- arch/x86/kernel/microcode_core.c | 7 ++++ arch/x86/kernel/microcode_intel.c | 67 +++++++++++++++++++++++++------ 4 files changed, 127 insertions(+), 33 deletions(-)
arch/x86/include/asm/microcode.h +7 -0
@@ -11,6 +11,12 @@ struct device;enum ucode_state {UCODE_ERROR ,UCODE_OK ,UCODE_NFOUND }; +/* data struct for request_firmware_nowait callback */ +struct microcode_request_data { +unsigned long cpu; +char name[36 ]; +}; +struct microcode_ops {enum ucode_state (*request_microcode_user) (int cpu,const void __user *buf,size_t size);
@@ -34,6 +40,7 @@ struct ucode_cpu_info {struct cpu_signature cpu_sig;int valid;void *mc; +struct completion completion; };extern struct ucode_cpu_info ucode_cpu_info[];
arch/x86/kernel/microcode_amd.c +58 -21
106 lines not on this page · raw message
arch/x86/kernel/microcode_core.c +7 -0
@@ -171,8 +171,13 @@ static void apply_microcode_local(void *arg)static int apply_microcode_on_target (int cpu) {struct apply_microcode_ctx ctx = { .err =0 }; +struct ucode_cpu_info *uci = ucode_cpu_info + cpu;int ret; + ret =wait_for_completion_timeout (&uci->completion,HZ ); +if (!ret) +pr_warn ("microcode: cpu %d thread wait timed out\n" , cpu); + ret =smp_call_function_single (cpu, apply_microcode_local, &ctx,1 );if (!ret) ret = ctx.err;
@@ -285,6 +290,7 @@ static int reload_for_cpu(int cpu)if (!uci->valid)return err; +init_completion (&uci->completion); ustate = microcode_ops->request_microcode_fw (cpu, µcode_pdev->dev, true);if (ustate ==UCODE_OK )apply_microcode_on_target (cpu);
@@ -392,6 +398,7 @@ static enum ucode_state microcode_init_cpu(int cpu, bool refresh_fw)if (system_state !=SYSTEM_RUNNING )return UCODE_NFOUND ; +init_completion (&uci->completion); ustate = microcode_ops->request_microcode_fw (cpu, µcode_pdev->dev, refresh_fw);
arch/x86/kernel/microcode_intel.c +55 -12
89 lines not on this page · raw message
Signature · 1 line
1.7.9.3
Re: [PATCH 2/2] intel_microcode, Fix long microcode load time when firmware file is missing
Quoted 18 lines · Prarit Bhargava
If no firmware is found on the system that matches the processor, the microcode module can take hours to load. For example on recent kernels when loading the microcode module on an Intel system:
[ 239.532116] microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x413 [ 299.693447] microcode: CPU1 sig=0x306e4, pf=0x1, revision=0x413 [ 359.821972] microcode: CPU2 sig=0x306e4, pf=0x1, revision=0x413 [ 419.960263] microcode: CPU3 sig=0x306e4, pf=0x1, revision=0x413 [ 480.090024] microcode: CPU4 sig=0x306e4, pf=0x1, revision=0x413 ... [ 2825.151364] microcode: CPU43 sig=0x306e4, pf=0x1, revision=0x413 [ 2885.280863] microcode: CPU44 sig=0x306e4, pf=0x1, revision=0x413 [ 2945.410719] microcode: CPU45 sig=0x306e4, pf=0x1, revision=0x413 [ 3005.540541] microcode: CPU46 sig=0x306e4, pf=0x1, revision=0x413 [ 3065.670405] microcode: CPU47 sig=0x306e4, pf=0x1, revision=0x413...
etc.
Quoted 7 lines
Similarly there is a 60 second "hang" when loading the AMD module, which isn't as bad as the Intel situation, but it is a noticeable delay in the system boot and can be improved upon.
Using
request_firmware_nowait()seems more appropriate here and then we can avoid these delays, resulting in very quick load times for the microcode.
request_firmware_nowait() should be a good idea for the problem, but why do you pass FW_ACTION_NOHOTPLUG as uevent? It is used now by drivers which requires their own user-space applications to handle the request by polling the firmware device under sysfs.
So I am wondering if your microcode case belongs to the above case of FW_ACTION_NOHOTPLUG?
And why don't you pass FW_ACTION_HOTPLUG? and you are sure that udev isn't required to handle your microcode update request?
Thanks,
--
Ming Lei
Re: [PATCH 2/2] intel_microcode, Fix long microcode load time when firmware file is missing
Quoted 23 lines · Ming Lei
Quoted 22 lines · Prarit Bhargava
If no firmware is found on the system that matches the processor, the microcode module can take hours to load. For example on recent kernels when loading the microcode module on an Intel system:
[ 239.532116] microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x413 [ 299.693447] microcode: CPU1 sig=0x306e4, pf=0x1, revision=0x413 [ 359.821972] microcode: CPU2 sig=0x306e4, pf=0x1, revision=0x413 [ 419.960263] microcode: CPU3 sig=0x306e4, pf=0x1, revision=0x413 [ 480.090024] microcode: CPU4 sig=0x306e4, pf=0x1, revision=0x413 ... [ 2825.151364] microcode: CPU43 sig=0x306e4, pf=0x1, revision=0x413 [ 2885.280863] microcode: CPU44 sig=0x306e4, pf=0x1, revision=0x413 [ 2945.410719] microcode: CPU45 sig=0x306e4, pf=0x1, revision=0x413 [ 3005.540541] microcode: CPU46 sig=0x306e4, pf=0x1, revision=0x413 [ 3065.670405] microcode: CPU47 sig=0x306e4, pf=0x1, revision=0x413...
etc.Similarly there is a 60 second "hang" when loading the AMD module, which isn't as bad as the Intel situation, but it is a noticeable delay in the system boot and can be improved upon.
Quoted 8 lines
Quoted 3 lines
Using
request_firmware_nowait()seems more appropriate here and then we can avoid these delays, resulting in very quick load times for the microcode.
request_firmware_nowait()should be a good idea for the problem, but why do you passFW_ACTION_NOHOTPLUGas uevent? It is used now by drivers which requires their own user-space applications to handle the request by polling the firmware device under sysfs.
Hello Ming,
Oh, I see.
Quoted 3 lines
So I am wondering if your microcode case belongs to the above case of
FW_ACTION_NOHOTPLUG?
You're right. I can easily change that in v2.
Quoted 4 lines
And why don't you pass
FW_ACTION_HOTPLUG? and you are sure that udev isn't required to handle your microcode update request?
AFAICT in both cases, udev wasn't required to handle the cpu microcode update. Both drivers use CMH to load the firmware which removes the need for udev to do anything. Admittedly maybe I've missed some odd use case but I don't think it is necessary.
P.
Re: [PATCH 2/2] intel_microcode, Fix long microcode load time when firmware file is missing
Quoted 9 lines · Prarit Bhargava
Quoted 4 lines
And why don't you pass
FW_ACTION_HOTPLUG? and you are sure that udev isn't required to handle your microcode update request?AFAICT in both cases, udev wasn't required to handle the cpu microcode update. Both drivers use CMH to load the firmware which removes the need for udev to do anything. Admittedly maybe I've missed some odd use case but I don't think it is necessary.
OK, so I guess the CMH still need uevent to get notified, right?
If yes, you should take FW_ACTION_HOTPLUG.
Thanks,
--
Ming Lei
Re: [PATCH 2/2] intel_microcode, Fix long microcode load time when firmware file is missing
Quoted 6 lines · Ming Lei
Quoted 5 lines · Prarit Bhargava
Quoted 4 lines
And why don't you pass
FW_ACTION_HOTPLUG? and you are sure that udev isn't required to handle your microcode update request?
Quoted 6 lines
Quoted 4 lines
AFAICT in both cases, udev wasn't required to handle the cpu microcode update. Both drivers use CMH to load the firmware which removes the need for udev to do anything. Admittedly maybe I've missed some odd use case but I don't think it is necessary.
OK, so I guess the CMH still need uevent to get notified, right?
The code as it is _currently_ written does not use uevents to load the processor firmware. ie) call_usermodehelper does not need uevent to get notified, so I think FW_ACTION_NOHOTPLUG is correct.
P.
Re: [PATCH 2/2] intel_microcode, Fix long microcode load time when firmware file is missing
Quoted 14 lines · Prarit Bhargava
Quoted 11 lines · Ming Lei
Quoted 9 lines · Prarit Bhargava
Quoted 4 lines
And why don't you pass
FW_ACTION_HOTPLUG? and you are sure that udev isn't required to handle your microcode update request?AFAICT in both cases, udev wasn't required to handle the cpu microcode update. Both drivers use CMH to load the firmware which removes the need for udev to do anything. Admittedly maybe I've missed some odd use case but I don't think it is necessary.
Quoted 5 lines
Quoted 1 line
OK, so I guess the CMH still need uevent to get notified, right?
The code as it is _currently_ written does not use uevents to load the processor firmware. ie)
call_usermodehelperdoes not need uevent to get notified, so I thinkFW_ACTION_NOHOTPLUGis correct.
You need to make sure your patch won't break userspace in old distribution with your _currently_ code.
Actually if udev isn't used in your user space, the timeout issue won't be triggered because it is blocked by udev.
Thanks,
--
Ming Lei
Re: [PATCH 2/2] intel_microcode, Fix long microcode load time when firmware file is missing
Quoted 15 lines · Ming Lei
Quoted 14 lines · Prarit Bhargava
Quoted 11 lines · Ming Lei
Quoted 9 lines · Prarit Bhargava
Quoted 4 lines
And why don't you pass
FW_ACTION_HOTPLUG? and you are sure that udev isn't required to handle your microcode update request?AFAICT in both cases, udev wasn't required to handle the cpu microcode update. Both drivers use CMH to load the firmware which removes the need for udev to do anything. Admittedly maybe I've missed some odd use case but I don't think it is necessary.
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
OK, so I guess the CMH still need uevent to get notified, right?
The code as it is _currently_ written does not use uevents to load the processor firmware. ie)
call_usermodehelperdoes not need uevent to get notified, so I thinkFW_ACTION_NOHOTPLUGis correct.You need to make sure your patch won't break userspace in old distribution with your _currently_ code.
AFAICT, Suse, Ubuntu, Linux Mint and Fedora all work with my changes.
P.
Quoted 4 lines · Prarit Bhargava
Using
request_firmware_nowait()seems more appropriate here and then we can avoid these delays, resulting in very quick load times for the microcode.
It would be probably also good to remember if the loading failed for a given CPU and then don't try it on any other.
Just need to cache the exact stepping too to handle different steppings.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only