Skip to contentSkip to conversation

Regression/boot failure on 5.16.3

DISCUSSION Jason Self 2 participants 4 messages Feb 4, 2022 Regressions

lore
Share & tools
lore

Jason Self

author
#0 lore raw

Forwarding as requested.

Begin forwarded message:

Quoted 7 lines · On Fri, 4 Feb 2022 08:00:17 +0100 Greg KH <[email protected]> wrote:

Please cc: the authors of this commit, and the upstream wireless developers so they can help you out here as I think the same issue shows up in 5.17-rc2, right?

thanks,

greg k-h

Date: Thu, 3 Feb 2022 16:19:59 -0800
From: Jason Self <[email protected]>
To: [email protected]
Subject: Regression/boot failure on 5.16.3

The computer (amd64) fails to boot. The init was stuck at the synchronization of the time through the network. This began between 5.16.2 (good) and 5.16.3 (bad.) This continues on 5.16.4 and 5.16.5. Git bisect revealed the following. In this case the nonfree firmwre is not present on the system. Blacklisting the iwflwifi module works as a workaround for now.

6b5ad4bd0d78fef6bbe0ecdf96e09237c9c52cc1 is the first bad commit commit 6b5ad4bd0d78fef6bbe0ecdf96e09237c9c52cc1

Author: Johannes Berg <[email protected]>
Date:   Fri Dec 10 11:12:42 2021 +0200

iwlwifi: fix leaks/bad data after failed firmware load

[ Upstream commit ab07506b0454bea606095951e19e72c282bfbb42 ]

If firmware load fails after having loaded some parts of the firmware, e.g. the IML image, then this would leak. For the host command list we'd end up running into a WARN on the next attempt to load another firmware image.

Fix this by calling iwl_dealloc_ucode() on failures, and make that also clear the data so we start fresh on the next round.

Link:
https://lore.kernel.org/r/iwlwifi.20211210110539.1f742f0eb58a.I1315f22f6aa632d94ae2069f85e1bca5e734dce0@changeid

 drivers/net/wireless/intel/iwlwifi/iwl-drv.c | 8 ++++++++
 1 file changed, 8 insertions(+)
Quoted 112 lines · On Fri, 4 Feb 2022 09:48:24 +0100 Thorsten Leemhuis <[email protected]> wrote:

[TLDR: I'm adding this regression to regzbot, the Linux kernel regression tracking bot; most text you find below is compiled from a few templates paragraphs some of you might have seen already.]

Hi, this is your Linux kernel regression tracker speaking.

Adding the regression mailing list to the list of recipients, as it should be in the loop for all regressions, as explained here:
https://www.kernel.org/doc/html/latest/admin-guide/reporting-issues.html

Quoted 11 lines · Jason Self

The computer (amd64) fails to boot. The init was stuck at the synchronization of the time through the network. This began between 5.16.2 (good) and 5.16.3 (bad.) This continues on 5.16.4 and 5.16.5. Git bisect revealed the following. In this case the nonfree firmwre is not present on the system. Blacklisting the iwflwifi module works as a workaround for now.

6b5ad4bd0d78fef6bbe0ecdf96e09237c9c52cc1 is the first bad commit commit 6b5ad4bd0d78fef6bbe0ecdf96e09237c9c52cc1

Author: Johannes Berg <[email protected]>
Date:   Fri Dec 10 11:12:42 2021 +0200  

To be sure this issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, my Linux kernel regression tracking bot:

#regzbot ^introduced 6b5ad4bd0d78fef6bbe0ecdf96e09237c9c52cc1 #regzbot title net: iwlwifi: system fails to boot since 5.16.3 #regzbot ignore-activity

Reminder: when fixing the issue, please add a 'Link:' tag with the URL to the report (the parent of this mail) using the kernel.org redirector, as explained in 'Documentation/process/submitting-patches.rst'. Regzbot then will automatically mark the regression as resolved once the fix lands in the appropriate tree. For more details about regzbot see footer.

Sending this to everyone that got the initial report, to make all aware of the tracking. I also hope that messages like this motivate people to directly get at least the regression mailing list and ideally even regzbot involved when dealing with regressions, as messages like this wouldn't be needed then.

Don't worry, I'll send further messages wrt to this regression just to the lists (with a tag in the subject so people can filter them away), as long as they are intended just for regzbot. With a bit of luck no such messages will be needed anyway.

Ciao, Thorsten (wearing his 'Linux kernel regression tracker' hat)

P.S.: As a Linux kernel regression tracker I'm getting a lot of reports on my table. I can only look briefly into most of them. Unfortunately therefore I sometimes will get things wrong or miss something important. I hope that's not the case here; if you think it is, don't hesitate to tell me about it in a public reply, that's in everyone's interest.

BTW, I have no personal interest in this issue, which is tracked using regzbot, my Linux kernel regression tracking bot
(https://linux-regtracking.leemhuis.info/regzbot/). I'm only posting this mail to get things rolling again and hence don't need to be CC on all further activities wrt to this regression.

Quoted 14 lines

iwlwifi: fix leaks/bad data after failed firmware load

[ Upstream commit ab07506b0454bea606095951e19e72c282bfbb42 ]

If firmware load fails after having loaded some parts of the firmware, e.g. the IML image, then this would leak. For the host command list we'd end up running into a WARN on the next attempt to load another firmware image.

Fix this by calling iwl_dealloc_ucode() on failures, and make that also clear the data so we start fresh on the next round.

Quoted 7 lines

Link:
https://lore.kernel.org/r/iwlwifi.20211210110539.1f742f0eb58a.I1315f22f6aa632d94ae2069f85e1bca5e734dce0@changeid

 drivers/net/wireless/intel/iwlwifi/iwl-drv.c | 8 ++++++++
 1 file changed, 8 insertions(+)  

---
Additional information about regzbot:

If you want to know more about regzbot, check out its web-interface, the getting start guide, and/or the references documentation:

https://linux-regtracking.leemhuis.info/regzbot/
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/getting_started.md
https://gitlab.com/knurd42/regzbot/-/blob/main/docs/reference.md

The last two documents will explain how you can interact with regzbot yourself if your want to.

Hint for reporters: when reporting a regression it's in your interest to tell #regzbot about it in the report, as that will ensure the regression gets on the radar of regzbot and the regression tracker. That's in your interest, as they will make sure the report won't fall through the cracks unnoticed.

Hint for developers: you normally don't need to care about regzbot once it's involved. Fix the issue as you normally would, just remember to include a 'Link:' tag to the report in the commit message, as explained in Documentation/process/submitting-patches.rst That aspect was recently was made more explicit in commit 1f57bd42b77c: https://git.kernel.org/linus/1f57bd42b77c

#1 lore raw
Quoted 11 lines

The computer (amd64) fails to boot. The init was stuck at the synchronization of the time through the network. This began between 5.16.2 (good) and 5.16.3 (bad.) This continues on 5.16.4 and 5.16.5. Git bisect revealed the following. In this case the nonfree firmwre is not present on the system. Blacklisting the iwflwifi module works as a workaround for now.

6b5ad4bd0d78fef6bbe0ecdf96e09237c9c52cc1 is the first bad commit commit 6b5ad4bd0d78fef6bbe0ecdf96e09237c9c52cc1

I checked out this commit and ran a test with no firmware present, but couldn't reproduce any crash.

Could you take a picture when it crashes? Perhaps boot without 'quiet' on the command line, or boot into recovery mode or such where you actually see the output?

johannes

author
#2 lore raw
Quoted 8 lines · On Fri, 04 Feb 2022 15:34:33 +0100 Johannes Berg <[email protected]>…
Quoted 8 lines

The computer (amd64) fails to boot. The init was stuck at the synchronization of the time through the network. This began between 5.16.2 (good) and 5.16.3 (bad.) This continues on 5.16.4 and 5.16.5. Git bisect revealed the following. In this case the nonfree firmwre is not present on the system. Blacklisting the iwflwifi module works as a workaround for now.

Quoted 6 lines
Quoted 3 lines

6b5ad4bd0d78fef6bbe0ecdf96e09237c9c52cc1 is the first bad commit commit 6b5ad4bd0d78fef6bbe0ecdf96e09237c9c52cc1

I checked out this commit and ran a test with no firmware present, but couldn't reproduce any crash.

I can confirm it still happens on 5.17-rc2. For the environment, Debian 11 with only the Main repo enabled.

lspci says:
00:14.3 Network controller: Intel Corporation Comet Lake PCH-LP CNVi WiFi

There is also a USB WiFi attached (AR9271)

Quoted 3 lines

Could you take a picture when it crashes? Perhaps boot without 'quiet' on the command line, or boot into recovery mode or such where you actually see the output?

They are attached:
* Raise network interfaces takes several seconds;
* Network Time Synchronization stuck.

author
#3 lore raw
Quoted 13 lines · On Fri, 4 Feb 2022 12:06:18 -0800 Jason Self <[email protected]> wrote:

On Fri, 04 Feb 2022 15:34:33 +0100
Johannes Berg <[email protected]> wrote:

[...]
[...]

I can confirm it still happens on 5.17-rc2. For the environment, Debian 11 with only the Main repo enabled.

lspci says:
00:14.3 Network controller: Intel Corporation Comet Lake PCH-LP CNVi WiFi

Quoted 7 lines

There is also a USB WiFi attached (AR9271)

[...]

They are attached:
* Raise network interfaces takes several seconds;
* Network Time Synchronization stuck.

Also attaching the kernel config used for 5.17-rc2 in case that is helpful.