modules: Only return -EEXIST for modules that have finished loading
PATCH v2 1/1 modules: Only return -EEXIST for modules that have finished loading
Heiko, it would still be good to get a test of this patch from you. I tested this here at Red Hat on some System Z machines. Without the modification made here in v2, the systems failed to boot ~10% of the time. After the modification I do not see any boot failures. I also was able to reproduce the boot issue with the acpi_cpufreq driver on a very large & fast x86 system which had closer to 100% failure rate without the changes in v2. After the modification in v2 the system has rebooted all weekend without any issues.
P.
---8<---
Microsoft HyperV disables the X86_FEATURE_SMCA bit on AMD systems, and linux guests boot with repeated errors:
amd64_edac_mod: Unknown symbol amd_unregister_ecc_decoder (err -2)amd64_edac_mod: Unknown symbol amd_register_ecc_decoder (err -2)amd64_edac_mod: Unknown symbol amd_report_gart_errors (err -2)amd64_edac_mod: Unknown symbol amd_unregister_ecc_decoder (err -2)amd64_edac_mod: Unknown symbol amd_register_ecc_decoder (err -2)amd64_edac_mod: Unknown symbol amd_report_gart_errors (err -2)
The warnings occur because the module code erroneously returns -EEXIST for modules that have failed to load and are in the process of being removed from the module list.
module amd64_edac_mod has a dependency on module edac_mce_amd. Using modules.dep, systemd will load edac_mce_amd for every request of amd64_edac_mod. When the edac_mce_amd module loads, the module has state MODULE_STATE_UNFORMED and once the module load fails and the state becomes MODULE_STATE_GOING. Another request for edac_mce_amd module executes and add_unformed_module() will erroneously return -EEXIST even though the previous instance of edac_mce_amd has MODULE_STATE_GOING. Upon receiving -EEXIST, systemd attempts to load amd64_edac_mod, which fails because of unknown symbols from edac_mce_amd.
add_unformed_module() must wait to return for any case other than MODULE_STATE_LIVE to prevent a race between multiple loads of dependent modules.
v2: The initial (old->state != MODULE_STATE_LIVE) change exposed an additional issue in the code. wait_event_interruptible() puts each thread to sleep until the a module finishes loading an executes the module_wq workqueue. The result is a long delay during the boot. Switching to wait_event_interruptible_timeout() resolves the sleep problem.
--- kernel/module.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/module.c +4 -4
@@ -3568,12 +3568,12 @@ static int add_unformed_module(struct module *mod)mutex_lock (&module_mutex); old =find_module_all (mod->name,strlen (mod->name), true);if (old !=NULL ) { -if (old->state ==MODULE_STATE_COMING - || old->state ==MODULE_STATE_UNFORMED ) { +if (old->state !=MODULE_STATE_LIVE ) {/* Wait in case it fails to load. */ mutex_unlock (&module_mutex); - err =wait_event_interruptible (module_wq, -finished_loading (mod->name)); + err =wait_event_interruptible_timeout (module_wq, +finished_loading (mod->name), +HZ /1000 );if (err) goto out_unlocked; goto again;
Signature · 1 line
2.18.1
Hi Prarit,
Quoted 8 lines · Prarit Bhargava
Heiko, it would still be good to get a test of this patch from you. I tested this here at Red Hat on some System Z machines. Without the modification made here in v2, the systems failed to boot ~10% of the time. After the modification I do not see any boot failures. I also was able to reproduce the boot issue with the
acpi_cpufreqdriver on a very large & fast x86 system which had closer to 100% failure rate without the changes in v2. After the modification in v2 the system has rebooted all weekend without any issues.
I gave it a try on a machine that failed quite reliably with your first patch (the one that was in linux-next and caused problems on s390) - it now comes up without any issues. Also a machine with 80 CPUs (+SMT2 -> 160 CPUs) comes up without problems.
So everything seems to work. Thanks!
+++ Prarit Bhargava [07/05/19 10:54 -0400]:
Quoted 47 lines
Heiko, it would still be good to get a test of this patch from you. I tested this here at Red Hat on some System Z machines. Without the modification made here in v2, the systems failed to boot ~10% of the time. After the modification I do not see any boot failures. I also was able to reproduce the boot issue with the
acpi_cpufreqdriver on a very large & fast x86 system which had closer to 100% failure rate without the changes in v2. After the modification in v2 the system has rebooted all weekend without any issues.P.
---8<---
Microsoft HyperV disables the
X86_FEATURE_SMCAbit on AMD systems, and linux guests boot with repeated errors:
amd64_edac_mod: Unknown symbolamd_unregister_ecc_decoder(err -2)amd64_edac_mod: Unknown symbolamd_register_ecc_decoder(err -2)amd64_edac_mod: Unknown symbolamd_report_gart_errors(err -2)amd64_edac_mod: Unknown symbolamd_unregister_ecc_decoder(err -2)amd64_edac_mod: Unknown symbolamd_register_ecc_decoder(err -2)amd64_edac_mod: Unknown symbolamd_report_gart_errors(err -2)The warnings occur because the module code erroneously returns
-EEXISTfor modules that have failed to load and are in the process of being removed from the module list.module
amd64_edac_modhas a dependency on moduleedac_mce_amd. Using modules.dep, systemd will loadedac_mce_amdfor every request ofamd64_edac_mod. When theedac_mce_amdmodule loads, the module has stateMODULE_STATE_UNFORMEDand once the module load fails and the state becomesMODULE_STATE_GOING. Another request foredac_mce_amdmodule executes andadd_unformed_module()will erroneously return-EEXISTeven though the previous instance ofedac_mce_amdhasMODULE_STATE_GOING. Upon receiving-EEXIST, systemd attempts to loadamd64_edac_mod, which fails because of unknown symbols fromedac_mce_amd.
add_unformed_module()must wait to return for any case other thanMODULE_STATE_LIVEto prevent a race between multiple loads of dependent modules.v2: The initial (
old->state!=MODULE_STATE_LIVE) change exposed an additional issue in the code.wait_event_interruptible()puts each thread to sleep until the a module finishes loading an executes themodule_wqworkqueue. The result is a long delay during the boot. Switching towait_event_interruptible_timeout()resolves the sleep problem.
Quoted 4 lines
Signed-off-by Prarit Bhargava <[email protected]>Cc Jessica Yu <[email protected]>Cc Heiko Carstens <[email protected]>Cc David Arcari <[email protected]>
Hi Prarit,
Thanks a lot for the revised patch. I'll queue this up right after the merge window is over.
Thanks!
Jessica
Quoted 28 lines
---
kernel/module.c| 8 ++++----1 file changed, 4 insertions(+), 4 deletions(-)kernel/module.c +4
-4@@ -3568,12 +3568,12 @@ static int add_unformed_module(struct module *mod)mutex_lock (&module_mutex); old =find_module_all (mod->name,strlen (mod->name), true);if (old !=NULL ) { -if (old->state ==MODULE_STATE_COMING - || old->state ==MODULE_STATE_UNFORMED ) { +if (old->state !=MODULE_STATE_LIVE ) {/* Wait in case it fails to load. */ mutex_unlock (&module_mutex); - err =wait_event_interruptible (module_wq, -finished_loading (mod->name)); + err =wait_event_interruptible_timeout (module_wq, +finished_loading (mod->name), +HZ /1000 );if (err) goto out_unlocked; goto again;Signature · 1 line
2.18.1