kernel: fork: set wake_q_sleeper.next=NULL again in dup_task_struct
Subsystems (3)
PATCH 1/1 kernel: fork: set wake_q_sleeper.next=NULL again in dup_task_struct
From: Steffen Dirkwinkel <[email protected]>
Without this we get system hangs within a couple of days. It's also reproducible in minutes with "stress-ng --exec 20".
Example error in dmesg:
INFO: task stress-ng:163916 blocked for more than 120 seconds.
Not tainted 5.10.168-rt83 #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:stress-ng state:D stack: 0 pid:163916 ppid: 72833 flags:0x00004000
Call Trace:
__schedule+0x2bd/0x940
preempt_schedule_lock+0x23/0x50
rt_spin_lock_slowlock_locked+0x117/0x2c0
rt_spin_lock_slowlock+0x51/0x80
rt_write_lock+0x1e/0x1c0
do_exit+0x3ac/0xb20
do_group_exit+0x39/0xb0
get_signal+0x145/0x960
? wake_up_new_task+0x21f/0x3c0
arch_do_signal_or_restart+0xf1/0x830
? __x64_sys_futex+0x146/0x1d0
exit_to_user_mode_prepare+0x116/0x1a0
syscall_exit_to_user_mode+0x28/0x190
entry_SYSCALL_64_after_hwframe+0x61/0xc6
RIP: 0033:0x7f738d9074a7
RSP: 002b:00007ffdafda3cb0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: fffffffffffffe00 RBX: 00000000000000ca RCX: 00007f738d9074a7
RDX: 0000000000028051 RSI: 0000000000000000 RDI: 00007f738be949d0
RBP: 00007ffdafda3d88 R08: 0000000000000000 R09: 00007f738be94700
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000028051
R13: 00007f738be949d0 R14: 00007ffdafda51e0 R15: 00007f738be94700--- kernel/fork.c | 1 + 1 file changed, 1 insertion(+)
kernel/fork.c +1 -0
@@ -949,6 +949,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) tsk->splice_pipe =NULL ; tsk->task_frag.page =NULL ; tsk->wake_q.next =NULL ; + tsk->wake_q_sleeper.next =NULL ; tsk->pf_io_worker =NULL ;account_kernel_stack (tsk,1 );
Signature · 1 line
2.40.0
Quoted 33 lines · On Mon, Mar 20, 2023 at 09:03:47AM +0100, [email protected] wrote:
From: Steffen Dirkwinkel <[email protected]>
Without this we get system hangs within a couple of days. It's also reproducible in minutes with "stress-ng --exec 20".
Example error in dmesg:
INFO: task stress-ng:163916 blocked for more than 120 seconds. Not tainted 5.10.168-rt83 #2 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:stress-ng state:D stack: 0 pid:163916 ppid: 72833 flags:0x00004000 Call Trace: __schedule+0x2bd/0x940 preempt_schedule_lock+0x23/0x50 rt_spin_lock_slowlock_locked+0x117/0x2c0 rt_spin_lock_slowlock+0x51/0x80 rt_write_lock+0x1e/0x1c0 do_exit+0x3ac/0xb20 do_group_exit+0x39/0xb0 get_signal+0x145/0x960 ? wake_up_new_task+0x21f/0x3c0 arch_do_signal_or_restart+0xf1/0x830 ? __x64_sys_futex+0x146/0x1d0 exit_to_user_mode_prepare+0x116/0x1a0 syscall_exit_to_user_mode+0x28/0x190 entry_SYSCALL_64_after_hwframe+0x61/0xc6 RIP: 0033:0x7f738d9074a7 RSP: 002b:00007ffdafda3cb0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: fffffffffffffe00 RBX: 00000000000000ca RCX: 00007f738d9074a7 RDX: 0000000000028051 RSI: 0000000000000000 RDI: 00007f738be949d0 RBP: 00007ffdafda3d88 R08: 0000000000000000 R09: 00007f738be94700 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000028051 R13: 00007f738be949d0 R14: 00007ffdafda51e0 R15: 00007f738be94700
Quoted 1 line
Fixes 1ba44dcf789d ("Merge tag 'v5.10.162' into v5.10-rt")
Thank you for spotting and investigating that!
I dropped that specific line while fixing a small merge conflict from
788d0824269b io_uring: import 5.15-stable io_uring
Interestingly enough, I didn't see that problem while running stress-ng. I may need to add a few more, different, systems to my test base.
Anyway, I will add this fix to the next build.
Luis
Quoted 20 lines
Signed-off-by Steffen Dirkwinkel <[email protected]>--- kernel/fork.c | 1 + 1 file changed, 1 insertion(+)kernel/fork.c +1
-0@@ -949,6 +949,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) tsk->splice_pipe =NULL ; tsk->task_frag.page =NULL ; tsk->wake_q.next =NULL ; + tsk->wake_q_sleeper.next =NULL ; tsk->pf_io_worker =NULL ;account_kernel_stack (tsk,1 );Signature · 1 line
2.40.0
---end quoted text---
I have just checked the 5.15-rt branch on stable-rt and it is also missing this line.
________________________________________
From: Luis Claudio R. Goncalves <[email protected]>
Sent: 20 March 2023 11:10
To: [email protected]
Cc: [email protected]; [email protected]; Steffen Dirkwinkel
Subject: Re: [PATCH v5.10-rt] kernel: fork: set wake_q_sleeper.next=NULL again in dup_task_struct
*External Message* - Use caution before opening links or attachments
Quoted 33 lines · On Mon, Mar 20, 2023 at 09:03:47AM +0100, [email protected] wrote:
From: Steffen Dirkwinkel <[email protected]>
Without this we get system hangs within a couple of days. It's also reproducible in minutes with "stress-ng --exec 20".
Example error in dmesg:
INFO: task stress-ng:163916 blocked for more than 120 seconds. Not tainted 5.10.168-rt83 #2 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:stress-ng state:D stack: 0 pid:163916 ppid: 72833 flags:0x00004000 Call Trace: __schedule+0x2bd/0x940 preempt_schedule_lock+0x23/0x50 rt_spin_lock_slowlock_locked+0x117/0x2c0 rt_spin_lock_slowlock+0x51/0x80 rt_write_lock+0x1e/0x1c0 do_exit+0x3ac/0xb20 do_group_exit+0x39/0xb0 get_signal+0x145/0x960 ? wake_up_new_task+0x21f/0x3c0 arch_do_signal_or_restart+0xf1/0x830 ? __x64_sys_futex+0x146/0x1d0 exit_to_user_mode_prepare+0x116/0x1a0 syscall_exit_to_user_mode+0x28/0x190 entry_SYSCALL_64_after_hwframe+0x61/0xc6 RIP: 0033:0x7f738d9074a7 RSP: 002b:00007ffdafda3cb0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: fffffffffffffe00 RBX: 00000000000000ca RCX: 00007f738d9074a7 RDX: 0000000000028051 RSI: 0000000000000000 RDI: 00007f738be949d0 RBP: 00007ffdafda3d88 R08: 0000000000000000 R09: 00007f738be94700 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000028051 R13: 00007f738be949d0 R14: 00007ffdafda51e0 R15: 00007f738be94700
Quoted 1 line
Fixes 1ba44dcf789d ("Merge tag 'v5.10.162' into v5.10-rt")
Thank you for spotting and investigating that!
I dropped that specific line while fixing a small merge conflict from
788d0824269b io_uring: import 5.15-stable io_uring
Interestingly enough, I didn't see that problem while running stress-ng. I may need to add a few more, different, systems to my test base.
Anyway, I will add this fix to the next build.
Luis
Quoted 18 lines
Signed-off-by Steffen Dirkwinkel <[email protected]>--- kernel/fork.c | 1 + 1 file changed, 1 insertion(+)kernel/fork.c +1
-1@@ -949,6 +949,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) tsk->splice_pipe =NULL ; tsk->task_frag.page =NULL ; tsk->wake_q.next =NULL ; + tsk->wake_q_sleeper.next =NULL ; tsk->pf_io_worker =NULL ;account_kernel_stack (tsk,1 ); --
Quoted 2 lines
2.40.0
---end quoted text---
Sorry - just ignore - I now see that 5.15 didn't need this change.
________________________________________
From: Leonard, Niall <[email protected]>
Sent: 23 March 2023 08:59
To: Luis Claudio R. Goncalves; [email protected]
Cc: [email protected]; [email protected]; Steffen Dirkwinkel
Subject: Re: [PATCH v5.10-rt] kernel: fork: set wake_q_sleeper.next=NULL again in dup_task_struct
*External Message* - Use caution before opening links or attachments
I have just checked the 5.15-rt branch on stable-rt and it is also missing this line.
________________________________________
From: Luis Claudio R. Goncalves <[email protected]>
Sent: 20 March 2023 11:10
To: [email protected]
Cc: [email protected]; [email protected]; Steffen Dirkwinkel
Subject: Re: [PATCH v5.10-rt] kernel: fork: set wake_q_sleeper.next=NULL again in dup_task_struct
*External Message* - Use caution before opening links or attachments
Quoted 33 lines · On Mon, Mar 20, 2023 at 09:03:47AM +0100, [email protected] wrote:
From: Steffen Dirkwinkel <[email protected]>
Without this we get system hangs within a couple of days. It's also reproducible in minutes with "stress-ng --exec 20".
Example error in dmesg:
INFO: task stress-ng:163916 blocked for more than 120 seconds. Not tainted 5.10.168-rt83 #2 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:stress-ng state:D stack: 0 pid:163916 ppid: 72833 flags:0x00004000 Call Trace: __schedule+0x2bd/0x940 preempt_schedule_lock+0x23/0x50 rt_spin_lock_slowlock_locked+0x117/0x2c0 rt_spin_lock_slowlock+0x51/0x80 rt_write_lock+0x1e/0x1c0 do_exit+0x3ac/0xb20 do_group_exit+0x39/0xb0 get_signal+0x145/0x960 ? wake_up_new_task+0x21f/0x3c0 arch_do_signal_or_restart+0xf1/0x830 ? __x64_sys_futex+0x146/0x1d0 exit_to_user_mode_prepare+0x116/0x1a0 syscall_exit_to_user_mode+0x28/0x190 entry_SYSCALL_64_after_hwframe+0x61/0xc6 RIP: 0033:0x7f738d9074a7 RSP: 002b:00007ffdafda3cb0 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: fffffffffffffe00 RBX: 00000000000000ca RCX: 00007f738d9074a7 RDX: 0000000000028051 RSI: 0000000000000000 RDI: 00007f738be949d0 RBP: 00007ffdafda3d88 R08: 0000000000000000 R09: 00007f738be94700 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000028051 R13: 00007f738be949d0 R14: 00007ffdafda51e0 R15: 00007f738be94700
Quoted 1 line
Fixes 1ba44dcf789d ("Merge tag 'v5.10.162' into v5.10-rt")
Thank you for spotting and investigating that!
I dropped that specific line while fixing a small merge conflict from
788d0824269b io_uring: import 5.15-stable io_uring
Interestingly enough, I didn't see that problem while running stress-ng. I may need to add a few more, different, systems to my test base.
Anyway, I will add this fix to the next build.
Luis
Quoted 18 lines
Signed-off-by Steffen Dirkwinkel <[email protected]>--- kernel/fork.c | 1 + 1 file changed, 1 insertion(+)kernel/fork.c +1
-1@@ -949,6 +949,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node) tsk->splice_pipe =NULL ; tsk->task_frag.page =NULL ; tsk->wake_q.next =NULL ; + tsk->wake_q_sleeper.next =NULL ; tsk->pf_io_worker =NULL ;account_kernel_stack (tsk,1 ); --
Quoted 2 lines
2.40.0
---end quoted text---