New: NVMe/writeback wb_workfn/blocked for more than 30 seconds
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): [email protected]/
Keith Busch
maintainerQuoted 35 lines · On Sun, Mar 13, 2022 at 08:13:53PM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=215679
Bug ID: 215679 Summary: NVMe/writeback wb_workfn/blocked for more than 30 seconds Product: IO/Storage Version: 2.5 Kernel Version: 5.17.0-rc7 Hardware: x86-64 OS: Linux Tree: Mainline Status: NEW Severity: high Priority: P1 Component: NVMe Assignee: [email protected] Reporter: [email protected] Regression: NoCreated attachment 300564
--> https://bugzilla.kernel.org/attachment.cgi?id=300564&action=edit
dmesg log after suspend resume, io stuckAfter system suspend/resume filesystem IO will stall, producing a 'kworker blocked for more than x sec" in dmesg, recovering after a long delay. See the attached dmesg-suspend-resume-nvme-stuck.txt. I also noticed the same issue happening right after booting or after runtime suspend transitions.
The same issue also happens on multiple SKL systems in the i915 team's CI farm, see:
https://gitlab.freedesktop.org/drm/intel/-/issues/4547
I bisected the problem to
commit4f5022453acd0f7b28012e20b7d048470f129894
Quoted 8 lines
Author: Jens Axboe <[email protected]> Date: Mon Oct 18 08:45:39 2021 -0600nvme: wire up completion batching for the IRQ path
By reverting it on top of 5.17.0-rc7, I can't reproduce the problem. Attached dmesg-suspend-resume-nvme-ok.txt with the revert, captured after a few suspend/resume.
Forwarding to linux-nvme for higher visibility.
Hi, this is your Linux kernel regression tracker. Top-posting for once, to make this easily accessible to everyone.
Jens, this regression is now a month old and the culprit was already identified when the issue was reported, nevertheless it seems no real progress was made to address this, at least afaics. What's up there? Should the culprit be reverted?
Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
P.S.: As the Linux kernel's regression tracker I deal with a lot of reports and sometimes miss something important when writing mails like this. If that's the case here, don't hesitate to tell me in a public reply, it's in everyone's interest to set the public record straight.
#regzbot poke
Quoted 47 lines · Keith Busch
Quoted 35 lines · On Sun, Mar 13, 2022 at 08:13:53PM +0000, [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=215679
Bug ID: 215679 Summary: NVMe/writeback wb_workfn/blocked for more than 30 seconds Product: IO/Storage Version: 2.5 Kernel Version: 5.17.0-rc7 Hardware: x86-64 OS: Linux Tree: Mainline Status: NEW Severity: high Priority: P1 Component: NVMe Assignee: [email protected] Reporter: [email protected] Regression: NoCreated attachment 300564
--> https://bugzilla.kernel.org/attachment.cgi?id=300564&action=edit
dmesg log after suspend resume, io stuckAfter system suspend/resume filesystem IO will stall, producing a 'kworker blocked for more than x sec" in dmesg, recovering after a long delay. See the attached dmesg-suspend-resume-nvme-stuck.txt. I also noticed the same issue happening right after booting or after runtime suspend transitions.
The same issue also happens on multiple SKL systems in the i915 team's CI farm, see:
https://gitlab.freedesktop.org/drm/intel/-/issues/4547
I bisected the problem to
commit4f5022453acd0f7b28012e20b7d048470f129894Quoted 8 lines
Author: Jens Axboe <[email protected]> Date: Mon Oct 18 08:45:39 2021 -0600nvme: wire up completion batching for the IRQ path
By reverting it on top of 5.17.0-rc7, I can't reproduce the problem. Attached dmesg-suspend-resume-nvme-ok.txt with the revert, captured after a few suspend/resume.
Forwarding to linux-nvme for higher visibility.