Skip to contentSkip to conversation

lib/igt_kmod: Allow some leeway in igt_kmod_unload_r

PATCH In review Jonathan Cavitt 3 participants 4 messages Jan 18, 2023 Library code

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 lib/igt_kmod: Allow some leeway in igt_kmod_unload_r

kmod_module_remove_module occasionally returns EAGAIN for mei_gsc in the setup of some gem_lmem_swapping subtests. Just because EAGAIN is returned doesn't mean the module is lost and unable to be unloaded. Try again some number of times (currently 20) before giving up.

Retries will occur for -EBUSY and -EAGAIN, as these imply the system is waiting for the target module can simply be waited for. All other errors exit immediately, but as the context for each error informs their relative severity, no warnings will be issued.

CC Stuart Summers <[email protected]>
CC Sandeep Kumar Parupalli <[email protected]>
CC Chris Wilson <[email protected]>
---
 lib/igt_kmod.c | 35 ++++++++++++++++++++++++++++++++---
 1 file changed, 32 insertions(+), 3 deletions(-)
lib/igt_kmod.c +32 -3
index 17090110c..4aa5320d0 100644
@@ -253,8 +253,11 @@ out:
 
 static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags)
 {
+#define MAX_TRIES	20
+#define SLEEP_DURATION	500000
 	struct kmod_list *holders, *pos;
-	int err = 0;
+	int err, tries;
+	const char *mod_name = kmod_module_get_name(kmod);
 
 	holders = kmod_module_get_holders(kmod);
 	kmod_list_foreach(pos, holders) {
@@ -269,7 +272,6 @@ static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags)
 		return err;
 
 	if (igt_kmod_is_loading(kmod)) {
-		const char *mod_name = kmod_module_get_name(kmod);
 		igt_debug("%s still initializing\n", mod_name);
 		err = igt_wait(!igt_kmod_is_loading(kmod), 10000, 100);
 		if (err < 0) {
@@ -279,7 +281,34 @@ static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags)
 		}
 	}
 
-	return kmod_module_remove_module(kmod, flags);
+	for (tries = 0; tries < MAX_TRIES; tries++) {
+		err = kmod_module_remove_module(kmod, flags);
+
+		/* Only loop in the following cases */
+		if (err != -EBUSY && err != -EAGAIN)
+			break;
+
+		igt_debug("Module %s failed to unload with err: %d on attempt: %i\n",
+			  mod_name, err, tries + 1);
+
+		if (MAX_TRIES - tries - 1)
+			usleep(SLEEP_DURATION);
+	}
+
+	if (err == -ENOENT)
+		igt_debug("Module %s could not be found or does not exist. err: %d\n",
+			  mod_name, err);
+	else if (err == -ENOTSUP)
+		igt_debug("Module %s cannot be unloaded. err: %d\n",
+			  mod_name, err);
+	else if (err)
+		igt_debug("Module %s failed to unload with err: %d after ~%.1fms\n",
+			  mod_name, err, SLEEP_DURATION*tries/1000.);
+	else if (tries)
+		igt_debug("Module %s unload took ~%.1fms over %i attempts\n",
+			  mod_name, SLEEP_DURATION*tries/1000., tries + 1);
+
+	return err;
 }
 
 /**
Signature · 1 line
2.25.1
#1 lore raw

[igt-dev] ✗ Fi.CI.BAT: failure for lib/igt_kmod: Allow some leeway in igt_kmod_unload_r (rev3)

== Series Details ==

== Series Details ==

Series: lib/igt_kmod: Allow some leeway in igt_kmod_unload_r (rev3)
URL : https://patchwork.freedesktop.org/series/112566/
State : failure

== Summary ==

CI Bug Log - changes from IGT_7123 -> IGTPW_8367
====================================================

Summary
-------

**FAILURE**

Serious unknown changes coming with IGTPW_8367 absolutely need to be verified manually.

If you think the reported changes have nothing to do with the changes introduced in IGTPW_8367, please notify your bug team to allow them to document this new failure mode, which will reduce false positives in CI.

External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8367/index.html

Participating hosts (42 -> 42)
------------------------------

Additional (1): fi-pnv-d510
Missing (1): fi-snb-2520m

Possible new issues
-------------------

Here are the unknown changes that may have been introduced in IGTPW_8367:

### IGT changes ###

#### Possible regressions ####

* igt@core_hotunplug@unbind-rebind:
- fi-apl-guc: [PASS][1] -> [DMESG-WARN][2]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7123/fi-apl-guc/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8367/fi-apl-guc/igt@[email protected]

Known issues
------------

Here are the changes found in IGTPW_8367 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@kms_chamelium_hpd@common-hpd-after-suspend:
- bat-dg1-6: NOTRUN -> [SKIP][3] ([i915#7828])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8367/bat-dg1-6/igt@[email protected]

* igt@kms_psr@primary_page_flip:
- fi-pnv-d510: NOTRUN -> [SKIP][4] ([fdo#109271]) +44 similar issues
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8367/fi-pnv-d510/igt@kms_psr@primary_page_flip.html

* igt@runner@aborted:
- fi-apl-guc: NOTRUN -> [FAIL][5] ([i915#4312])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8367/fi-apl-guc/igt@[email protected]

#### Possible fixes ####

* igt@i915_selftest@live@gt_lrc:
- {bat-adln-1}: [INCOMPLETE][6] ([i915#4983] / [i915#7609]) -> [PASS][7]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7123/bat-adln-1/igt@i915_selftest@live@gt_lrc.html
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8367/bat-adln-1/igt@i915_selftest@live@gt_lrc.html

* igt@i915_selftest@live@reset:
- {bat-rpls-1}: [DMESG-FAIL][8] ([i915#4983]) -> [PASS][9]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7123/bat-rpls-1/igt@i915_selftest@[email protected]
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8367/bat-rpls-1/igt@i915_selftest@[email protected]

* igt@i915_selftest@live@workarounds:
- bat-dg1-6: [INCOMPLETE][10] ([i915#4983]) -> [PASS][11]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_7123/bat-dg1-6/igt@i915_selftest@[email protected]
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8367/bat-dg1-6/igt@i915_selftest@[email protected]

{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).

[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[i915#4312]: https://gitlab.freedesktop.org/drm/intel/issues/4312
[i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
[i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
[i915#6997]: https://gitlab.freedesktop.org/drm/intel/issues/6997
[i915#7609]: https://gitlab.freedesktop.org/drm/intel/issues/7609
[i915#7625]: https://gitlab.freedesktop.org/drm/intel/issues/7625
[i915#7828]: https://gitlab.freedesktop.org/drm/intel/issues/7828

Build changes
-------------

* CI: CI-20190529 -> None
* IGT: IGT_7123 -> IGTPW_8367

CI-20190529: 20190529
CI_DRM_12601: 5fee98189f91987f2b91fa666541de66e6c4a5b9 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_8367: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8367/index.html
IGT_7123: 2b29e8ac07fbcfadc48b9d60e4d736a6e3b289ab @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8367/index.html

#2 lore raw
In reply to Jonathan Cavitt

Hi Jonathan,

Quoted 51 lines · Jonathan Cavitt

kmod_module_remove_module occasionally returns EAGAIN for mei_gsc in the setup of some gem_lmem_swapping subtests. Just because EAGAIN is returned doesn't mean the module is lost and unable to be unloaded. Try again some number of times (currently 20) before giving up.

Retries will occur for -EBUSY and -EAGAIN, as these imply the system is waiting for the target module can simply be waited for. All other errors exit immediately, but as the context for each error informs their relative severity, no warnings will be issued.

CC Stuart Summers <[email protected]>
CC Sandeep Kumar Parupalli <[email protected]>
CC Chris Wilson <[email protected]>
---
 lib/igt_kmod.c | 35 ++++++++++++++++++++++++++++++++---
 1 file changed, 32 insertions(+), 3 deletions(-)
lib/igt_kmod.c +5 -3
index 17090110c..4aa5320d0 100644
@@ -253,8 +253,11 @@ out:
 
 static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags)
 {
+#define MAX_TRIES	20
+#define SLEEP_DURATION	500000
 	struct kmod_list *holders, *pos;
-	int err = 0;
+	int err, tries;
+	const char *mod_name = kmod_module_get_name(kmod);
 
 	holders = kmod_module_get_holders(kmod);
 	kmod_list_foreach(pos, holders) {
@@ -269,7 +272,6 @@ static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags)
 		return err;
 
 	if (igt_kmod_is_loading(kmod)) {
-		const char *mod_name = kmod_module_get_name(kmod);
 		igt_debug("%s still initializing\n", mod_name);
 		err = igt_wait(!igt_kmod_is_loading(kmod), 10000, 100);
 		if (err < 0) {
@@ -279,7 +281,34 @@ static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags)
 		}
 	}
 
-	return kmod_module_remove_module(kmod, flags);
+	for (tries = 0; tries < MAX_TRIES; tries++) {

lib/igt_kmod.c static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags) · line 294

Quoted 10 lines
lib/igt_kmod.c +10 -0
+		err = kmod_module_remove_module(kmod, flags);
+
+		/* Only loop in the following cases */
+		if (err != -EBUSY && err != -EAGAIN)
+			break;
+
+		igt_debug("Module %s failed to unload with err: %d on attempt: %i\n",
+			  mod_name, err, tries + 1);
+
+		if (MAX_TRIES - tries - 1)

------------------ ^
This should be tries < MAX_TRIES - 1

Quoted 5 lines
lib/igt_kmod.c +5 -0
+			usleep(SLEEP_DURATION);
+	}
+
+	if (err == -ENOENT)
+		igt_debug("Module %s could not be found or does not exist. err: %d\n",

lib/igt_kmod.c static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags) · line 309

Quoted 10 lines
lib/igt_kmod.c +10 -0
+			  mod_name, err);
+	else if (err == -ENOTSUP)
+		igt_debug("Module %s cannot be unloaded. err: %d\n",
+			  mod_name, err);
+	else if (err)
+		igt_debug("Module %s failed to unload with err: %d after ~%.1fms\n",
+			  mod_name, err, SLEEP_DURATION*tries/1000.);
+	else if (tries)
+		igt_debug("Module %s unload took ~%.1fms over %i attempts\n",
+			  mod_name, SLEEP_DURATION*tries/1000., tries + 1);

What about one final else with igt_debug about success at first try ?

Regards,
Kamil

Quoted 8 lines
lib/igt_kmod.c +2 -0
+
+	return err;
 }
 
 /**
Signature · 1 line
2.25.1
#3 lore raw

-----Original Message-----
From: Kamil Konieczny <[email protected]>
Sent: Wednesday, January 18, 2023 11:32 AM
To: [email protected]
Cc: [email protected]; Cavitt, Jonathan <[email protected]>; Parupalli, Sandeep Kumar <[email protected]>
Subject: Re: [igt-dev] [PATCH i-g-t] lib/igt_kmod: Allow some leeway in igt_kmod_unload_r

Quoted 53 lines

Hi Jonathan,

Quoted 49 lines · Jonathan Cavitt

kmod_module_remove_module occasionally returns EAGAIN for mei_gsc in the setup of some gem_lmem_swapping subtests. Just because EAGAIN is returned doesn't mean the module is lost and unable to be unloaded. Try again some number of times (currently 20) before giving up.

Retries will occur for -EBUSY and -EAGAIN, as these imply the system is waiting for the target module can simply be waited for. All other errors exit immediately, but as the context for each error informs their relative severity, no warnings will be issued.

CC Stuart Summers <[email protected]>
CC Sandeep Kumar Parupalli <[email protected]>
CC Chris Wilson <[email protected]>
---
 lib/igt_kmod.c | 35 ++++++++++++++++++++++++++++++++---
 1 file changed, 32 insertions(+), 3 deletions(-)
lib/igt_kmod.c +4 -2
index 17090110c..4aa5320d0 100644
@@ -253,8 +253,11 @@ out:
 
 static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags)
 {
+#define MAX_TRIES	20
+#define SLEEP_DURATION	500000
 	struct kmod_list *holders, *pos;
-	int err = 0;
+	int err, tries;
+	const char *mod_name = kmod_module_get_name(kmod);
 
 	holders = kmod_module_get_holders(kmod);
 	kmod_list_foreach(pos, holders) {
@@ -269,7 +272,6 @@ static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags)
 		return err;
 
 	if (igt_kmod_is_loading(kmod)) {
-		const char *mod_name = kmod_module_get_name(kmod);
 		igt_debug("%s still initializing\n", mod_name);
 		err = igt_wait(!igt_kmod_is_loading(kmod), 10000, 100);
 		if (err < 0) {
@@ -279,7 +281,34 @@ static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags)
 		}
 	}
 
Quoted 14 lines

lib/igt_kmod.c static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags) · line 294

Quoted 12 lines
lib/igt_kmod.c +11 -1
-	return kmod_module_remove_module(kmod, flags);
+	for (tries = 0; tries < MAX_TRIES; tries++) {
+		err = kmod_module_remove_module(kmod, flags);
+
+		/* Only loop in the following cases */
+		if (err != -EBUSY && err != -EAGAIN)
+			break;
+
+		igt_debug("Module %s failed to unload with err: %d on attempt: %i\n",
+			  mod_name, err, tries + 1);
+
+		if (MAX_TRIES - tries - 1)

------------------ ^
This should be tries < MAX_TRIES - 1

I'm using "tries != MAX_TRIES - 1", which should also work as long as the for-loop doesn't somehow overflow. Here, the two are logically equivalent:

tries != MAX_TRIES - 1
tries - tries != MAX_TRIES - 1 - tries
0 != MAX_TRIES - tries - 1

... I'll clean it up for clarity...

Quoted 10 lines
Quoted 10 lines
lib/igt_kmod.c +9 -0
+			usleep(SLEEP_DURATION);
+	}
+
+	if (err == -ENOENT)
+		igt_debug("Module %s could not be found or does not exist. err: %d\n",
+			  mod_name, err);
+	else if (err == -ENOTSUP)
+		igt_debug("Module %s cannot be unloaded. err: %d\n",
+			  mod_name, err);
Quoted 8 lines

lib/igt_kmod.c static int igt_kmod_unload_r(struct kmod_module *kmod, unsigned int flags) · line 309

Quoted 6 lines
lib/igt_kmod.c +6 -0
+	else if (err)
+		igt_debug("Module %s failed to unload with err: %d after ~%.1fms\n",
+			  mod_name, err, SLEEP_DURATION*tries/1000.);
+	else if (tries)
+		igt_debug("Module %s unload took ~%.1fms over %i attempts\n",
+			  mod_name, SLEEP_DURATION*tries/1000., tries + 1);

What about one final else with igt_debug about success at first try ?

Generally speaking, should we really be printing debug information when a function behaves as intended? I don't believe so, but if you want me to add that debug message there I'll do it.
-Jonathan Cavitt

Quoted 13 lines

Regards,
Kamil

Quoted 8 lines
lib/igt_kmod.c +2 -0
+
+	return err;
 }
 
 /**
Signature · 1 line
2.25.1