Skip to contentSkip to conversation

modules: Only return -EEXIST for modules that have finished loading

PATCH Superseded Prarit Bhargava 3 participants 3 messages May 7, 2019 – May 9, 2019

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 modules: Only return -EEXIST for modules that have finished loading

Heiko, it would still be good to get a test of this patch from you. I tested this here at Red Hat on some System Z machines. Without the modification made here in v2, the systems failed to boot ~10% of the time. After the modification I do not see any boot failures. I also was able to reproduce the boot issue with the acpi_cpufreq driver on a very large & fast x86 system which had closer to 100% failure rate without the changes in v2. After the modification in v2 the system has rebooted all weekend without any issues.

P.

---8<---

Microsoft HyperV disables the X86_FEATURE_SMCA bit on AMD systems, and linux guests boot with repeated errors:

amd64_edac_mod: Unknown symbol amd_unregister_ecc_decoder (err -2)
amd64_edac_mod: Unknown symbol amd_register_ecc_decoder (err -2)
amd64_edac_mod: Unknown symbol amd_report_gart_errors (err -2)
amd64_edac_mod: Unknown symbol amd_unregister_ecc_decoder (err -2)
amd64_edac_mod: Unknown symbol amd_register_ecc_decoder (err -2)
amd64_edac_mod: Unknown symbol amd_report_gart_errors (err -2)

The warnings occur because the module code erroneously returns -EEXIST for modules that have failed to load and are in the process of being removed from the module list.

module amd64_edac_mod has a dependency on module edac_mce_amd. Using modules.dep, systemd will load edac_mce_amd for every request of amd64_edac_mod. When the edac_mce_amd module loads, the module has state MODULE_STATE_UNFORMED and once the module load fails and the state becomes MODULE_STATE_GOING. Another request for edac_mce_amd module executes and add_unformed_module() will erroneously return -EEXIST even though the previous instance of edac_mce_amd has MODULE_STATE_GOING. Upon receiving -EEXIST, systemd attempts to load amd64_edac_mod, which fails because of unknown symbols from edac_mce_amd.

add_unformed_module() must wait to return for any case other than MODULE_STATE_LIVE to prevent a race between multiple loads of dependent modules.

v2: The initial (old->state != MODULE_STATE_LIVE) change exposed an additional issue in the code. wait_event_interruptible() puts each thread to sleep until the a module finishes loading an executes the module_wq workqueue. The result is a long delay during the boot. Switching to wait_event_interruptible_timeout() resolves the sleep problem.

Cc Jessica Yu <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc David Arcari <[email protected]>
---
 kernel/module.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
kernel/module.c +4 -4
index 1c429d8d2d74..6c868aabaf37 100644
@@ -3568,12 +3568,12 @@ static int add_unformed_module(struct module *mod)
 	mutex_lock(&module_mutex);
 	old = find_module_all(mod->name, strlen(mod->name), true);
 	if (old != NULL) {
-		if (old->state == MODULE_STATE_COMING
-		    || old->state == MODULE_STATE_UNFORMED) {
+		if (old->state != MODULE_STATE_LIVE) {
 			/* Wait in case it fails to load. */
 			mutex_unlock(&module_mutex);
-			err = wait_event_interruptible(module_wq,
-					       finished_loading(mod->name));
+			err = wait_event_interruptible_timeout(module_wq,
+					       finished_loading(mod->name),
+					       HZ/1000);
 			if (err)
 				goto out_unlocked;
 			goto again;
Signature · 1 line
2.18.1
#1 lore raw

Hi Prarit,

Quoted 8 lines · Prarit Bhargava

Heiko, it would still be good to get a test of this patch from you. I tested this here at Red Hat on some System Z machines. Without the modification made here in v2, the systems failed to boot ~10% of the time. After the modification I do not see any boot failures. I also was able to reproduce the boot issue with the acpi_cpufreq driver on a very large & fast x86 system which had closer to 100% failure rate without the changes in v2. After the modification in v2 the system has rebooted all weekend without any issues.

I gave it a try on a machine that failed quite reliably with your first patch (the one that was in linux-next and caused problems on s390) - it now comes up without any issues. Also a machine with 80 CPUs (+SMT2 -> 160 CPUs) comes up without problems.

So everything seems to work. Thanks!

#2 lore raw
In reply to Prarit Bhargava

+++ Prarit Bhargava [07/05/19 10:54 -0400]:

Quoted 47 lines

Heiko, it would still be good to get a test of this patch from you. I tested this here at Red Hat on some System Z machines. Without the modification made here in v2, the systems failed to boot ~10% of the time. After the modification I do not see any boot failures. I also was able to reproduce the boot issue with the acpi_cpufreq driver on a very large & fast x86 system which had closer to 100% failure rate without the changes in v2. After the modification in v2 the system has rebooted all weekend without any issues.

P.

---8<---

Microsoft HyperV disables the X86_FEATURE_SMCA bit on AMD systems, and linux guests boot with repeated errors:

amd64_edac_mod: Unknown symbol amd_unregister_ecc_decoder (err -2)
amd64_edac_mod: Unknown symbol amd_register_ecc_decoder (err -2)
amd64_edac_mod: Unknown symbol amd_report_gart_errors (err -2)
amd64_edac_mod: Unknown symbol amd_unregister_ecc_decoder (err -2)
amd64_edac_mod: Unknown symbol amd_register_ecc_decoder (err -2)
amd64_edac_mod: Unknown symbol amd_report_gart_errors (err -2)

The warnings occur because the module code erroneously returns -EEXIST for modules that have failed to load and are in the process of being removed from the module list.

module amd64_edac_mod has a dependency on module edac_mce_amd. Using modules.dep, systemd will load edac_mce_amd for every request of amd64_edac_mod. When the edac_mce_amd module loads, the module has state MODULE_STATE_UNFORMED and once the module load fails and the state becomes MODULE_STATE_GOING. Another request for edac_mce_amd module executes and add_unformed_module() will erroneously return -EEXIST even though the previous instance of edac_mce_amd has MODULE_STATE_GOING. Upon receiving -EEXIST, systemd attempts to load amd64_edac_mod, which fails because of unknown symbols from edac_mce_amd.

add_unformed_module() must wait to return for any case other than MODULE_STATE_LIVE to prevent a race between multiple loads of dependent modules.

v2: The initial (old->state != MODULE_STATE_LIVE) change exposed an additional issue in the code. wait_event_interruptible() puts each thread to sleep until the a module finishes loading an executes the module_wq workqueue. The result is a long delay during the boot. Switching to wait_event_interruptible_timeout() resolves the sleep problem.

Quoted 4 lines
Cc Jessica Yu <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc David Arcari <[email protected]>

Hi Prarit,

Thanks a lot for the revised patch. I'll queue this up right after the merge window is over.

Thanks!

Jessica

Quoted 28 lines

---
kernel/module.c | 8 ++++----

1 file changed, 4 insertions(+), 4 deletions(-)
kernel/module.c +4 -4
index 1c429d8d2d74..6c868aabaf37 100644
@@ -3568,12 +3568,12 @@ static int add_unformed_module(struct module *mod)
 	mutex_lock(&module_mutex);
 	old = find_module_all(mod->name, strlen(mod->name), true);
 	if (old != NULL) {
-		if (old->state == MODULE_STATE_COMING
-		    || old->state == MODULE_STATE_UNFORMED) {
+		if (old->state != MODULE_STATE_LIVE) {
 			/* Wait in case it fails to load. */
 			mutex_unlock(&module_mutex);
-			err = wait_event_interruptible(module_wq,
-					       finished_loading(mod->name));
+			err = wait_event_interruptible_timeout(module_wq,
+					       finished_loading(mod->name),
+					       HZ/1000);
 			if (err)
 				goto out_unlocked;
 			goto again;
Signature · 1 line
2.18.1