INFO: task hung in ext4_da_get_block_prep

Quoted 13 lines · syzbot

Hello,

syzbot found the following crash on:

HEAD commit:    f2b6e66e9885 Add linux-next specific files for 20180904
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=1735dc92400000
kernel config:  https://syzkaller.appspot.com/x/.config?x=15ad48400e39c1b3
dashboard link: https://syzkaller.appspot.com/bug?extid=f0fc7f62e88b1de99af3
compiler:       gcc (GCC) 8.0.1 20180413 (experimental)

CC: [[email protected] [email protected]
[email protected] [email protected]]

Quoted 7 lines

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:

Reported-by [email protected]

[ 7961] 0 7961 17585 8737 131072 0 0
syz-executor3

Hi Tetsuo,

Maybe you know what are these repeated lines with numbers? We started getting them on linux-next recently, also:
https://syzkaller.appspot.com/bug?extid=f8fa79b458bcae4d913d
They seem to cause various hangs/stalls.

Quoted 25 lines · Dmitry Vyukov

Quoted 13 lines · syzbot

Hello,

syzbot found the following crash on:

HEAD commit:    f2b6e66e9885 Add linux-next specific files for 20180904
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=1735dc92400000
kernel config:  https://syzkaller.appspot.com/x/.config?x=15ad48400e39c1b3
dashboard link: https://syzkaller.appspot.com/bug?extid=f0fc7f62e88b1de99af3
compiler:       gcc (GCC) 8.0.1 20180413 (experimental)

CC: [[email protected] [email protected]
[email protected] [email protected]]

Quoted 7 lines

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:

Reported-by [email protected]

[ 7961] 0 7961 17585 8737 131072 0 0
syz-executor3

Hi Tetsuo,

Quoted 4 lines

Maybe you know what are these repeated lines with numbers? We started getting them on linux-next recently, also:
https://syzkaller.appspot.com/bug?extid=f8fa79b458bcae4d913d
They seem to cause various hangs/stalls.

Yes, these lines are from the OOM killer. (Thus, if we can, I want to remove ext4 people before upstreaming this report.)

  dump_tasks mm/oom_kill.c:420 [inline]
  dump_header+0xf0d/0xf70 mm/oom_kill.c:450
  oom_kill_process.cold.28+0x10/0x95a mm/oom_kill.c:953
  out_of_memory+0xa88/0x1430 mm/oom_kill.c:1120

What is annoying is that one for_each_process() traversal with printk() is taking 52 seconds which is too long to do under RCU section. Under such situation, invoking the OOM killer for three times will exceed khungtaskd threshold 140 seconds. Was syzbot trying to test fork bomb situation?

Anyway, we might need to introduce rcu_lock_break() like check_hung_uninterruptible_tasks() does...

[  999.629589] [  16497]     0 16497    17585     8739   126976        0             0 syz-executor5
[ 1026.435955] [  32764]     0 32764    17585     8739   126976        0             0 syz-executor5
[ 1026.445027] [    311]     0   311    17585     8737   131072        0             0 syz-executor3
[ 1047.914324] [  10315]     0 10315    17585     8271   126976        0             0 syz-executor0
[ 1047.923384] Out of memory: Kill process 4670 (syz-fuzzer) score 53 or sacrifice child
[ 1047.931934] Killed process 5032 (syz-executor1) total-vm:70212kB, anon-rss:60kB, file-rss:0kB, shmem-rss:0kB
[ 1047.988138] syz-executor2 invoked oom-killer: gfp_mask=0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null), order=1, oom_score_adj=0
[ 1048.000015] syz-executor2 cpuset=syz2 mems_allowed=0
[ 1048.005199] CPU: 0 PID: 4700 Comm: syz-executor2 Not tainted 4.19.0-rc2-next-20180904+ #55
[ 1048.740679] [   2347]     0  2347      278      186    32768        0             0 none
[ 1051.319928] [  16497]     0 16497    17585     8739   126976        0             0 syz-executor5
[ 1096.740878] [   8841]     0  8841    17585     8232   126976        0             0 syz-executor5
[ 1078.140677] [  32764]     0 32764    17585     8739   126976        0             0 syz-executor5
[ 1078.149807] [    311]     0   311    17585     8737   131072        0             0 syz-executor3
[ 1096.740878] [   8841]     0  8841    17585     8232   126976        0             0 syz-executor5

Also, another notable thing is that the backtrace for some reason includes

[ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0

line. Was syzbot testing process freezing functionality?

Quoted 5 lines

Also, another notable thing is that the backtrace for some reason includes

[ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0

line. Was syzbot testing process freezing functionality?

What's the API for this?

Quoted 6 lines · Dmitry Vyukov

Quoted 5 lines

Also, another notable thing is that the backtrace for some reason includes

[ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0

line. Was syzbot testing process freezing functionality?

Quoted 2 lines

What's the API for this?

I'm not a user of suspend/hibernation. But it seems that usage of the API is to write one of words listed in /sys/power/state into /sys/power/state .

# echo suspend > /sys/power/state

Quoted 10 lines · Tetsuo Handa

Quoted 8 lines · Dmitry Vyukov

Quoted 5 lines

Also, another notable thing is that the backtrace for some reason includes

[ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0

line. Was syzbot testing process freezing functionality?

What's the API for this?

Quoted 4 lines

I'm not a user of suspend/hibernation. But it seems that usage of the API is to write one of words listed in /sys/power/state into /sys/power/state .

# echo suspend > /sys/power/state

syzkaller should not write to /sys/power/state. The only mention of "power" is in some selinux contexts.

Quoted 12 lines · Dmitry Vyukov

Quoted 10 lines · Tetsuo Handa

Quoted 8 lines · Dmitry Vyukov

Quoted 5 lines

Also, another notable thing is that the backtrace for some reason includes

[ 1048.211540]  ? oom_killer_disable+0x3a0/0x3a0

line. Was syzbot testing process freezing functionality?

What's the API for this?

Quoted 8 lines

Quoted 4 lines

I'm not a user of suspend/hibernation. But it seems that usage of the API is to write one of words listed in /sys/power/state into /sys/power/state .

# echo suspend > /sys/power/state

syzkaller should not write to /sys/power/state. The only mention of "power" is in some selinux contexts.

OK. Then, I have no idea.
Anyway, I think we can apply this patch.

From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001
From: Tetsuo Handa <[email protected]>
Date: Thu, 6 Sep 2018 19:53:18 +0900
Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

Since printk() is slow, printing one line takes nearly 0.01 second. As a result, syzbot is stalling for 52 seconds trying to dump 5600

tasks at for_each_process() under RCU. Since such situation is almost inflight fork bomb attack (the OOM killer will print similar tasks for so many times), it makes little sense to print all candidate tasks. Thus, this patch introduces 3 seconds limit for printing.

Signed-off-by Tetsuo Handa <[email protected]>

Cc Dmitry Vyukov <[email protected]>

---
 mm/oom_kill.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

mm/oom_kill.c +10 -1

index f10aa53..48e5bf6 100644
@@ -399,14 +399,22 @@ static void dump_tasks(struct mem_cgroup *memcg, const nodemask_t *nodemask)
 {
        struct task_struct *p;
        struct task_struct *task;
+       unsigned long start;
+       unsigned int skipped = 0;
 
        pr_info("Tasks state (memory values in pages):\n");
        pr_info("[  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name\n");
        rcu_read_lock();
+       start = jiffies;
        for_each_process(p) {
                if (oom_unkillable_task(p, memcg, nodemask))
                        continue;
 
+               if (time_after(jiffies, start + 3 * HZ)) {
+                       skipped++;
+                       continue;
+               }
+
                task = find_lock_task_mm(p);
                if (!task) {
                        /*

@@ -426,6 +434,8 @@ static void dump_tasks(struct mem_cgroup *memcg, const nodemask_t *nodemask)
                task_unlock(task);
        }
        rcu_read_unlock();
+       if (skipped)
+               pr_info("Printing %u tasks omitted.\n", skipped);
 }
 
 static void dump_header(struct oom_control *oc, struct task_struct *p)
--

1.8.3.1

Quoted 2 lines

Since printk() is slow, printing one line takes nearly 0.01 second. As a result, syzbot is stalling for 52 seconds trying to dump 5600

I wonder why there are so many of them?
We have at most 8 test processes (each having no more than 16 threads if that matters).
No more than 1 instance of syz-executor1 at a time. But we see output like the one below. It has lots of instances of syz-executor1 with different pid's. So does it print all tasks that ever existed (kernel does not store that info, right)? Or it livelocks picking up new and new tasks as they are created slower than they are created? Or we have tons of zombies?

...

[ 8037] 0 8037 17618 8738 131072 0 0 syz-executor1

Quoted 1 line

From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001

From: Tetsuo Handa <[email protected]>
Date: Thu, 6 Sep 2018 19:53:18 +0900
Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

Since printk() is slow, printing one line takes nearly 0.01 second. As a result, syzbot is stalling for 52 seconds trying to dump 5600 tasks at for_each_process() under RCU. Since such situation is almost inflight fork bomb attack (the OOM killer will print similar tasks for so many times), it makes little sense to print all candidate tasks. Thus, this patch introduces 3 seconds limit for printing.

Signed-off-by Tetsuo Handa <[email protected]>

Cc Dmitry Vyukov <[email protected]>

On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
[...]

Quoted 12 lines

Quoted 1 line

From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001

From: Tetsuo Handa <[email protected]>
Date: Thu, 6 Sep 2018 19:53:18 +0900
Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

Since printk() is slow, printing one line takes nearly 0.01 second. As a result, syzbot is stalling for 52 seconds trying to dump 5600 tasks at for_each_process() under RCU. Since such situation is almost inflight fork bomb attack (the OOM killer will print similar tasks for so many times), it makes little sense to print all candidate tasks. Thus, this patch introduces 3 seconds limit for printing.

Quoted 2 lines

Signed-off-by Tetsuo Handa <[email protected]>

Cc Dmitry Vyukov <[email protected]>

You really love timeout based solutions with randomly chosen timeouts, don't you. This is just ugly as hell. We already have means to disable tasks dumping (see /proc/sys/vm/oom_dump_tasks).

Quoted 17 lines · Michal Hocko

On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
[...]

Quoted 14 lines

Quoted 1 line

From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001

From: Tetsuo Handa <[email protected]>
Date: Thu, 6 Sep 2018 19:53:18 +0900
Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

Since printk() is slow, printing one line takes nearly 0.01 second. As a result, syzbot is stalling for 52 seconds trying to dump 5600 tasks at for_each_process() under RCU. Since such situation is almost inflight fork bomb attack (the OOM killer will print similar tasks for so many times), it makes little sense to print all candidate tasks. Thus, this patch introduces 3 seconds limit for printing.

Signed-off-by Tetsuo Handa <[email protected]>

Cc Dmitry Vyukov <[email protected]>

Quoted 3 lines

You really love timeout based solutions with randomly chosen timeouts, don't you. This is just ugly as hell. We already have means to disable tasks dumping (see /proc/sys/vm/oom_dump_tasks).

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Quoted 18 lines · Tetsuo Handa

Quoted 17 lines · Michal Hocko

On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
[...]

Quoted 14 lines

Quoted 1 line

From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001

From: Tetsuo Handa <[email protected]>
Date: Thu, 6 Sep 2018 19:53:18 +0900
Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

Since printk() is slow, printing one line takes nearly 0.01 second. As a result, syzbot is stalling for 52 seconds trying to dump 5600 tasks at for_each_process() under RCU. Since such situation is almost inflight fork bomb attack (the OOM killer will print similar tasks for so many times), it makes little sense to print all candidate tasks. Thus, this patch introduces 3 seconds limit for printing.

Signed-off-by Tetsuo Handa <[email protected]>

Cc Dmitry Vyukov <[email protected]>

Quoted 6 lines

Quoted 3 lines

You really love timeout based solutions with randomly chosen timeouts, don't you. This is just ugly as hell. We already have means to disable tasks dumping (see /proc/sys/vm/oom_dump_tasks).

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of the listing is to double check the list to understand the oom victim selection. If you have a partial list you simply cannot do that.

If the iteration takes too long and I can imagine it does with zillions of tasks then the proper way around it is either release the lock periodically after N tasks is processed or outright skip the whole thing if there are too many tasks. The first option is obviously tricky to prevent from duplicate entries or other artifacts.

So does anybody know if it can live lock picking up new tasks all the time? That's what it looks like at first glance. I also don't remember seeing anything similar in the past.

If it's a live lock and we resolve it, then we don't need to solve the problem of too many tasks here.

Ccing Oleg.

So a question for Oleg I guess. Is it possible that for_each_process live locks (or stalls for way too long/unbounded amount of time) under heavy fork/exit loads?

Quoted 4 lines · Michal Hocko

So a question for Oleg I guess. Is it possible that for_each_process live locks (or stalls for way too long/unbounded amount of time) under heavy fork/exit loads?

Oh yes, it can... plus other problems.

I even sent the initial patches which introduce for_each_process_break/continue a long ago... I'll try to find them tommorrow and resend.

Quoted 22 lines · Tetsuo Handa

Quoted 20 lines · Michal Hocko

On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
[...]

Quoted 12 lines

Quoted 1 line

From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001

From: Tetsuo Handa <[email protected]>
Date: Thu, 6 Sep 2018 19:53:18 +0900
Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

Since printk() is slow, printing one line takes nearly 0.01 second. As a result, syzbot is stalling for 52 seconds trying to dump 5600 tasks at for_each_process() under RCU. Since such situation is almost inflight fork bomb attack (the OOM killer will print similar tasks for so many times), it makes little sense to print all candidate tasks. Thus, this patch introduces 3 seconds limit for printing.

Quoted 2 lines

Signed-off-by Tetsuo Handa <[email protected]>

Cc Dmitry Vyukov <[email protected]>

You really love timeout based solutions with randomly chosen timeouts, don't you. This is just ugly as hell. We already have means to disable tasks dumping (see /proc/sys/vm/oom_dump_tasks).

Quoted 2 lines

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of the listing is to double check the list to understand the oom victim selection. If you have a partial list you simply cannot do that.

If the iteration takes too long and I can imagine it does with zillions of tasks then the proper way around it is either release the lock periodically after N tasks is processed or outright skip the whole thing if there are too many tasks. The first option is obviously tricky to prevent from duplicate entries or other artifacts.

Quoted 23 lines · Michal Hocko

Quoted 22 lines · Tetsuo Handa

Quoted 20 lines · Michal Hocko

On Thu 06-09-18 19:58:25, Tetsuo Handa wrote:
[...]

Quoted 12 lines

Quoted 1 line

From 18876f287dd69a7c33f65c91cfcda3564233f55e Mon Sep 17 00:00:00 2001

From: Tetsuo Handa <[email protected]>
Date: Thu, 6 Sep 2018 19:53:18 +0900
Subject: [PATCH] mm, oom: Introduce time limit for dump_tasks duration.

Since printk() is slow, printing one line takes nearly 0.01 second. As a result, syzbot is stalling for 52 seconds trying to dump 5600 tasks at for_each_process() under RCU. Since such situation is almost inflight fork bomb attack (the OOM killer will print similar tasks for so many times), it makes little sense to print all candidate tasks. Thus, this patch introduces 3 seconds limit for printing.

Quoted 2 lines

Signed-off-by Tetsuo Handa <[email protected]>

Cc Dmitry Vyukov <[email protected]>

You really love timeout based solutions with randomly chosen timeouts, don't you. This is just ugly as hell. We already have means to disable tasks dumping (see /proc/sys/vm/oom_dump_tasks).

Quoted 6 lines

Quoted 2 lines

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of the listing is to double check the list to understand the oom victim selection. If you have a partial list you simply cannot do that.

It serves as a safeguard for avoiding RCU stall warnings.

Quoted 1 line

Quoted 6 lines

If the iteration takes too long and I can imagine it does with zillions of tasks then the proper way around it is either release the lock periodically after N tasks is processed or outright skip the whole thing if there are too many tasks. The first option is obviously tricky to prevent from duplicate entries or other artifacts.

Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?

Quoted 17 lines

Quoted 6 lines

Quoted 2 lines

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of the listing is to double check the list to understand the oom victim selection. If you have a partial list you simply cannot do that.

It serves as a safeguard for avoiding RCU stall warnings.

Quoted 7 lines

If the iteration takes too long and I can imagine it does with zillions of tasks then the proper way around it is either release the lock periodically after N tasks is processed or outright skip the whole thing if there are too many tasks. The first option is obviously tricky to prevent from duplicate entries or other artifacts.

Quoted 1 line

Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?

This would be a better variant of your timeout based approach. But it can still produce an incomplete task list so it still consumes a lot of resources to print a long list of tasks potentially while that list is not useful for any evaluation. Maybe that is good enough. I don't know. I would generally recommend to disable the whole thing with workloads with many tasks though.

Quoted 17 lines · Michal Hocko

Quoted 17 lines

Quoted 6 lines

Quoted 2 lines

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of the listing is to double check the list to understand the oom victim selection. If you have a partial list you simply cannot do that.

It serves as a safeguard for avoiding RCU stall warnings.

Quoted 7 lines

If the iteration takes too long and I can imagine it does with zillions of tasks then the proper way around it is either release the lock periodically after N tasks is processed or outright skip the whole thing if there are too many tasks. The first option is obviously tricky to prevent from duplicate entries or other artifacts.

Quoted 9 lines

Quoted 1 line

Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?

This would be a better variant of your timeout based approach. But it can still produce an incomplete task list so it still consumes a lot of resources to print a long list of tasks potentially while that list is not useful for any evaluation. Maybe that is good enough. I don't know. I would generally recommend to disable the whole thing with workloads with many tasks though.

The "safeguard" is useful when there are _unexpectedly_ many tasks (like syzbot in this case). Why not to allow those who want to avoid lockup to avoid lockup rather than forcing them to disable the whole thing?

Quoted 28 lines · Tetsuo Handa

Quoted 26 lines · Michal Hocko

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of the listing is to double check the list to understand the oom victim selection. If you have a partial list you simply cannot do that.

It serves as a safeguard for avoiding RCU stall warnings.

Quoted 1 line

Quoted 8 lines

Quoted 6 lines

If the iteration takes too long and I can imagine it does with zillions of tasks then the proper way around it is either release the lock periodically after N tasks is processed or outright skip the whole thing if there are too many tasks. The first option is obviously tricky to prevent from duplicate entries or other artifacts.

Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?

This would be a better variant of your timeout based approach. But it can still produce an incomplete task list so it still consumes a lot of resources to print a long list of tasks potentially while that list is not useful for any evaluation. Maybe that is good enough. I don't know. I would generally recommend to disable the whole thing with workloads with many tasks though.

Quoted 3 lines

The "safeguard" is useful when there are _unexpectedly_ many tasks (like syzbot in this case). Why not to allow those who want to avoid lockup to avoid lockup rather than forcing them to disable the whole thing?

So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall then this should be recoverable thing (assuming we cannot really livelock as described by Dmitry).

Quoted 29 lines · Michal Hocko

Quoted 28 lines · Tetsuo Handa

Quoted 26 lines · Michal Hocko

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of the listing is to double check the list to understand the oom victim selection. If you have a partial list you simply cannot do that.

It serves as a safeguard for avoiding RCU stall warnings.

Quoted 1 line

Quoted 8 lines

Quoted 6 lines

If the iteration takes too long and I can imagine it does with zillions of tasks then the proper way around it is either release the lock periodically after N tasks is processed or outright skip the whole thing if there are too many tasks. The first option is obviously tricky to prevent from duplicate entries or other artifacts.

Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?

This would be a better variant of your timeout based approach. But it can still produce an incomplete task list so it still consumes a lot of resources to print a long list of tasks potentially while that list is not useful for any evaluation. Maybe that is good enough. I don't know. I would generally recommend to disable the whole thing with workloads with many tasks though.

Quoted 7 lines

Quoted 3 lines

The "safeguard" is useful when there are _unexpectedly_ many tasks (like syzbot in this case). Why not to allow those who want to avoid lockup to avoid lockup rather than forcing them to disable the whole thing?

So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall then this should be recoverable thing (assuming we cannot really livelock as described by Dmitry).

Should I add "vm.oom_dump_tasks = 0" to /etc/sysctl.conf on syzbot? It looks like it will make things faster, not pollute console output, prevent these stalls and that output does not seem to be too useful for debugging.

But I am still concerned as to what has changed recently. Potentially this happens only on linux-next, at least that's where I saw all existing reports.
New tasks seem to be added to the tail of the tasks list, but this part does not seem to be changed recently in linux-next..

Quoted 36 lines

Quoted 27 lines

Quoted 27 lines

Quoted 26 lines

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of the listing is to double check the list to understand the oom victim selection. If you have a partial list you simply cannot do that.

It serves as a safeguard for avoiding RCU stall warnings.

Quoted 1 line

Quoted 8 lines

Quoted 6 lines

If the iteration takes too long and I can imagine it does with zillions of tasks then the proper way around it is either release the lock periodically after N tasks is processed or outright skip the whole thing if there are too many tasks. The first option is obviously tricky to prevent from duplicate entries or other artifacts.

Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?

This would be a better variant of your timeout based approach. But it can still produce an incomplete task list so it still consumes a lot of resources to print a long list of tasks potentially while that list is not useful for any evaluation. Maybe that is good enough. I don't know. I would generally recommend to disable the whole thing with workloads with many tasks though.

Quoted 7 lines

Quoted 3 lines

The "safeguard" is useful when there are _unexpectedly_ many tasks (like syzbot in this case). Why not to allow those who want to avoid lockup to avoid lockup rather than forcing them to disable the whole thing?

So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall then this should be recoverable thing (assuming we cannot really livelock as described by Dmitry).

Quoted 4 lines

Should I add "vm.oom_dump_tasks = 0" to /etc/sysctl.conf on syzbot? It looks like it will make things faster, not pollute console output, prevent these stalls and that output does not seem to be too useful for debugging.

I think that oom_dump_tasks has only very limited usefulness for your testing.

Quoted 5 lines

But I am still concerned as to what has changed recently. Potentially this happens only on linux-next, at least that's where I saw all existing reports.
New tasks seem to be added to the tail of the tasks list, but this part does not seem to be changed recently in linux-next..

Yes, that would be interesting to find out.

Quoted 44 lines · Michal Hocko

Quoted 36 lines

Quoted 27 lines

Quoted 27 lines

Quoted 26 lines

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of the listing is to double check the list to understand the oom victim selection. If you have a partial list you simply cannot do that.

It serves as a safeguard for avoiding RCU stall warnings.

Quoted 1 line

Quoted 8 lines

Quoted 6 lines

If the iteration takes too long and I can imagine it does with zillions of tasks then the proper way around it is either release the lock periodically after N tasks is processed or outright skip the whole thing if there are too many tasks. The first option is obviously tricky to prevent from duplicate entries or other artifacts.

Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?

This would be a better variant of your timeout based approach. But it can still produce an incomplete task list so it still consumes a lot of resources to print a long list of tasks potentially while that list is not useful for any evaluation. Maybe that is good enough. I don't know. I would generally recommend to disable the whole thing with workloads with many tasks though.

Quoted 7 lines

Quoted 3 lines

The "safeguard" is useful when there are _unexpectedly_ many tasks (like syzbot in this case). Why not to allow those who want to avoid lockup to avoid lockup rather than forcing them to disable the whole thing?

So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall then this should be recoverable thing (assuming we cannot really livelock as described by Dmitry).

Quoted 4 lines

Should I add "vm.oom_dump_tasks = 0" to /etc/sysctl.conf on syzbot? It looks like it will make things faster, not pollute console output, prevent these stalls and that output does not seem to be too useful for debugging.

I think that oom_dump_tasks has only very limited usefulness for your testing.

Quoted 7 lines

Quoted 5 lines

But I am still concerned as to what has changed recently. Potentially this happens only on linux-next, at least that's where I saw all existing reports.
New tasks seem to be added to the tail of the tasks list, but this part does not seem to be changed recently in linux-next..

Yes, that would be interesting to find out.

Looking at another similar report:
https://syzkaller.appspot.com/bug?extid=0d867757fdc016c0157e
It looks like it can be just syzkaller learning how to do fork bombs after all (same binary multiplied infinite amount of times). Probably required some creativity because test programs do not contain loops per se and clone syscall does not accept start function pc. I will set vm.oom_dump_tasks = 0 and try to additionally restrict it with cgroups.

But I am still concerned as to what has changed recently. Potentially this happens only on linux-next, at least that's where I saw all existing reports.
New tasks seem to be added to the tail of the tasks list, but this part does not seem to be changed recently in linux-next..

Quoted 28 lines · Tetsuo Handa

Quoted 26 lines · Michal Hocko

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

I know /proc/sys/vm/oom_dump_tasks . Showing some entries while not always printing all entries might be helpful.

Not really. It could be more confusing than helpful. The main purpose of the listing is to double check the list to understand the oom victim selection. If you have a partial list you simply cannot do that.

It serves as a safeguard for avoiding RCU stall warnings.

Quoted 1 line

Quoted 8 lines

Quoted 6 lines

If the iteration takes too long and I can imagine it does with zillions of tasks then the proper way around it is either release the lock periodically after N tasks is processed or outright skip the whole thing if there are too many tasks. The first option is obviously tricky to prevent from duplicate entries or other artifacts.

Can we add rcu_lock_break() like check_hung_uninterruptible_tasks() does?

This would be a better variant of your timeout based approach. But it can still produce an incomplete task list so it still consumes a lot of resources to print a long list of tasks potentially while that list is not useful for any evaluation. Maybe that is good enough. I don't know. I would generally recommend to disable the whole thing with workloads with many tasks though.

Quoted 3 lines

The "safeguard" is useful when there are _unexpectedly_ many tasks (like syzbot in this case). Why not to allow those who want to avoid lockup to avoid lockup rather than forcing them to disable the whole thing?

So you get an rcu lockup splat and what? Unless you have panic_on_rcu_stall then this should be recoverable thing (assuming we cannot really livelock as described by Dmitry).