Skip to contentSkip to conversation

WARNING: CPU: 3 PID: 1568 at kernel/sched/core.c:7738 __might_sleep+0x69/0x7e

DISCUSSION Michal Hocko 4 participants 7 messages Dec 9, 2016 – Dec 21, 2016

lore
Share & tools
lore
#0 lore raw

Hi Tejun,
I am seeing the following warning during boot in my kvm testing environment:

[    3.549649] WARNING: CPU: 3 PID: 1568 at kernel/sched/core.c:7738 __might_sleep+0x69/0x7e
[    3.551279] do not call blocking ops when !TASK_RUNNING; state=2 set at [<ffffffff81098621>] prepare_to_wait+0x65/0xac
[    3.553354] Modules linked in:
[    3.554032] CPU: 3 PID: 1568 Comm: modprobe Not tainted 4.9.0-rc8-nofstest5-next-20161209-00013-g3134766e941a #1018
[    3.556099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014
[    3.557330] Call Trace:
[    3.557330]  dump_stack+0x68/0x92
[    3.557330]  __warn+0xc2/0xdd
[    3.557330]  warn_slowpath_fmt+0x4b/0x53
[    3.557330]  ? prepare_to_wait+0x65/0xac
[    3.557330]  ? prepare_to_wait+0x65/0xac
[    3.557330]  __might_sleep+0x69/0x7e
[    3.557330]  do_ide_request+0x47/0x540
[    3.557330]  ? __lock_is_held+0x47/0x7a
[    3.557330]  ? hrtimer_try_to_cancel+0x15f/0x1c6
[    3.557330]  ? _raw_spin_unlock_irqrestore+0x4c/0x61
[    3.557330]  __blk_run_queue+0x31/0x3a
[    3.557330]  cfq_insert_request+0x331/0x429
[    3.557330]  __elv_add_request+0x2bb/0x346
[    3.557330]  blk_flush_plug_list+0x1da/0x21f
[    3.557330]  ? bit_wait+0x50/0x50
[    3.557330]  io_schedule_timeout+0x47/0x102
[    3.557330]  bit_wait_io+0x1b/0x5a
[    3.557330]  __wait_on_bit+0x4d/0x7b
[    3.557330]  out_of_line_wait_on_bit+0x72/0x7d
[    3.557330]  ? bit_wait+0x50/0x50
[    3.557330]  ? autoremove_wake_function+0x4e/0x4e
[    3.557330]  __wait_on_buffer+0x40/0x43
[    3.557330]  bh_submit_read+0x60/0x72
[    3.557330]  ext4_get_branch+0x91/0x115
[    3.557330]  ext4_ind_map_blocks+0x181/0xa16
[    3.557330]  ? __lock_is_held+0x47/0x7a
[    3.557330]  ? lock_acquire+0x133/0x1c7
[    3.557330]  ext4_map_blocks+0x120/0x4ca
[    3.557330]  ext4_mpage_readpages+0x2a7/0x70a
[    3.557330]  ext4_readpages+0x33/0x35
[    3.557330]  __do_page_cache_readahead+0x239/0x32a
[    3.557330]  filemap_fault+0x180/0x602
[    3.557330]  ? filemap_fault+0x180/0x602
[    3.557330]  ext4_filemap_fault+0x37/0x4b
[    3.557330]  __do_fault+0x1f/0x94
[    3.557330]  handle_mm_fault+0x946/0xfca
[    3.557330]  __do_page_fault+0x361/0x518
[    3.557330]  do_page_fault+0x42/0x5f
[    3.557330]  page_fault+0x22/0x30

At first I thought this would be related to you recent work
http://lkml.kernel.org/r/[email protected]
but that doesn't seem to be in the linux-next I am using as a base. io_schedule_timeout is on the stack though so you might have an idea what is going on here. That might_sleep check has been added a long ago 02e7cf8f8488 ("ide-cd,atapi: use bio for internal commands") in 2.6.31 so I am wondering what has changed since than.

Thanks!

Signature · 2 lines
Michal Hocko
SUSE Labs
#1 lore raw

Hello, Michal.

Quoted 49 lines · Michal Hocko

Hi Tejun,
I am seeing the following warning during boot in my kvm testing environment:

[    3.549649] WARNING: CPU: 3 PID: 1568 at kernel/sched/core.c:7738 __might_sleep+0x69/0x7e
[    3.551279] do not call blocking ops when !TASK_RUNNING; state=2 set at [<ffffffff81098621>] prepare_to_wait+0x65/0xac
[    3.553354] Modules linked in:
[    3.554032] CPU: 3 PID: 1568 Comm: modprobe Not tainted 4.9.0-rc8-nofstest5-next-20161209-00013-g3134766e941a #1018
[    3.556099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014
[    3.557330] Call Trace:
[    3.557330]  dump_stack+0x68/0x92
[    3.557330]  __warn+0xc2/0xdd
[    3.557330]  warn_slowpath_fmt+0x4b/0x53
[    3.557330]  ? prepare_to_wait+0x65/0xac
[    3.557330]  ? prepare_to_wait+0x65/0xac
[    3.557330]  __might_sleep+0x69/0x7e
[    3.557330]  do_ide_request+0x47/0x540
[    3.557330]  ? __lock_is_held+0x47/0x7a
[    3.557330]  ? hrtimer_try_to_cancel+0x15f/0x1c6
[    3.557330]  ? _raw_spin_unlock_irqrestore+0x4c/0x61
[    3.557330]  __blk_run_queue+0x31/0x3a
[    3.557330]  cfq_insert_request+0x331/0x429
[    3.557330]  __elv_add_request+0x2bb/0x346
[    3.557330]  blk_flush_plug_list+0x1da/0x21f
[    3.557330]  ? bit_wait+0x50/0x50
[    3.557330]  io_schedule_timeout+0x47/0x102
[    3.557330]  bit_wait_io+0x1b/0x5a
[    3.557330]  __wait_on_bit+0x4d/0x7b
[    3.557330]  out_of_line_wait_on_bit+0x72/0x7d
[    3.557330]  ? bit_wait+0x50/0x50
[    3.557330]  ? autoremove_wake_function+0x4e/0x4e
[    3.557330]  __wait_on_buffer+0x40/0x43
[    3.557330]  bh_submit_read+0x60/0x72
[    3.557330]  ext4_get_branch+0x91/0x115
[    3.557330]  ext4_ind_map_blocks+0x181/0xa16
[    3.557330]  ? __lock_is_held+0x47/0x7a
[    3.557330]  ? lock_acquire+0x133/0x1c7
[    3.557330]  ext4_map_blocks+0x120/0x4ca
[    3.557330]  ext4_mpage_readpages+0x2a7/0x70a
[    3.557330]  ext4_readpages+0x33/0x35
[    3.557330]  __do_page_cache_readahead+0x239/0x32a
[    3.557330]  filemap_fault+0x180/0x602
[    3.557330]  ? filemap_fault+0x180/0x602
[    3.557330]  ext4_filemap_fault+0x37/0x4b
[    3.557330]  __do_fault+0x1f/0x94
[    3.557330]  handle_mm_fault+0x946/0xfca
[    3.557330]  __do_page_fault+0x361/0x518
[    3.557330]  do_page_fault+0x42/0x5f
[    3.557330]  page_fault+0x22/0x30
Quoted 7 lines

At first I thought this would be related to you recent work
http://lkml.kernel.org/r/[email protected]
but that doesn't seem to be in the linux-next I am using as a base. io_schedule_timeout is on the stack though so you might have an idea what is going on here. That might_sleep check has been added a long ago 02e7cf8f8488 ("ide-cd,atapi: use bio for internal commands") in 2.6.31 so I am wondering what has changed since than.

10d784eae2b4 ("sched: always use blk_schedule_flush_plug in io_schedule_out") seems related. might_sleep is complaining that it's being called while !TASK_RUNNING. Shaohua, any ideas?

Thanks.

Signature · 1 line
tejun
#2 lore raw
Quoted 11 lines · Tejun Heo

Hello, Michal.

Quoted 7 lines · Michal Hocko

At first I thought this would be related to you recent work
http://lkml.kernel.org/r/[email protected]
but that doesn't seem to be in the linux-next I am using as a base. io_schedule_timeout is on the stack though so you might have an idea what is going on here. That might_sleep check has been added a long ago 02e7cf8f8488 ("ide-cd,atapi: use bio for internal commands") in 2.6.31 so I am wondering what has changed since than.

Quoted 3 lines

10d784eae2b4 ("sched: always use blk_schedule_flush_plug in io_schedule_out") seems related. might_sleep is complaining that it's being called while !TASK_RUNNING. Shaohua, any ideas?

Yeah, it is pretty much similar to the one I fixed for networking:
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=d9dc8b0f8b4ec8cdc48ad5a20a3105387138be82

But in your case, it is complicated because of the wait bits API.

#3 lore raw
In reply to Tejun Heo
Quoted 60 lines · Tejun Heo

Hello, Michal.

Quoted 49 lines · Michal Hocko

Hi Tejun,
I am seeing the following warning during boot in my kvm testing environment:

[    3.549649] WARNING: CPU: 3 PID: 1568 at kernel/sched/core.c:7738 __might_sleep+0x69/0x7e
[    3.551279] do not call blocking ops when !TASK_RUNNING; state=2 set at [<ffffffff81098621>] prepare_to_wait+0x65/0xac
[    3.553354] Modules linked in:
[    3.554032] CPU: 3 PID: 1568 Comm: modprobe Not tainted 4.9.0-rc8-nofstest5-next-20161209-00013-g3134766e941a #1018
[    3.556099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014
[    3.557330] Call Trace:
[    3.557330]  dump_stack+0x68/0x92
[    3.557330]  __warn+0xc2/0xdd
[    3.557330]  warn_slowpath_fmt+0x4b/0x53
[    3.557330]  ? prepare_to_wait+0x65/0xac
[    3.557330]  ? prepare_to_wait+0x65/0xac
[    3.557330]  __might_sleep+0x69/0x7e
[    3.557330]  do_ide_request+0x47/0x540
[    3.557330]  ? __lock_is_held+0x47/0x7a
[    3.557330]  ? hrtimer_try_to_cancel+0x15f/0x1c6
[    3.557330]  ? _raw_spin_unlock_irqrestore+0x4c/0x61
[    3.557330]  __blk_run_queue+0x31/0x3a
[    3.557330]  cfq_insert_request+0x331/0x429
[    3.557330]  __elv_add_request+0x2bb/0x346
[    3.557330]  blk_flush_plug_list+0x1da/0x21f
[    3.557330]  ? bit_wait+0x50/0x50
[    3.557330]  io_schedule_timeout+0x47/0x102
[    3.557330]  bit_wait_io+0x1b/0x5a
[    3.557330]  __wait_on_bit+0x4d/0x7b
[    3.557330]  out_of_line_wait_on_bit+0x72/0x7d
[    3.557330]  ? bit_wait+0x50/0x50
[    3.557330]  ? autoremove_wake_function+0x4e/0x4e
[    3.557330]  __wait_on_buffer+0x40/0x43
[    3.557330]  bh_submit_read+0x60/0x72
[    3.557330]  ext4_get_branch+0x91/0x115
[    3.557330]  ext4_ind_map_blocks+0x181/0xa16
[    3.557330]  ? __lock_is_held+0x47/0x7a
[    3.557330]  ? lock_acquire+0x133/0x1c7
[    3.557330]  ext4_map_blocks+0x120/0x4ca
[    3.557330]  ext4_mpage_readpages+0x2a7/0x70a
[    3.557330]  ext4_readpages+0x33/0x35
[    3.557330]  __do_page_cache_readahead+0x239/0x32a
[    3.557330]  filemap_fault+0x180/0x602
[    3.557330]  ? filemap_fault+0x180/0x602
[    3.557330]  ext4_filemap_fault+0x37/0x4b
[    3.557330]  __do_fault+0x1f/0x94
[    3.557330]  handle_mm_fault+0x946/0xfca
[    3.557330]  __do_page_fault+0x361/0x518
[    3.557330]  do_page_fault+0x42/0x5f
[    3.557330]  page_fault+0x22/0x30
Quoted 7 lines

At first I thought this would be related to you recent work
http://lkml.kernel.org/r/[email protected]
but that doesn't seem to be in the linux-next I am using as a base. io_schedule_timeout is on the stack though so you might have an idea what is going on here. That might_sleep check has been added a long ago 02e7cf8f8488 ("ide-cd,atapi: use bio for internal commands") in 2.6.31 so I am wondering what has changed since than.

Quoted 3 lines

10d784eae2b4 ("sched: always use blk_schedule_flush_plug in io_schedule_out") seems related. might_sleep is complaining that it's being called while !TASK_RUNNING. Shaohua, any ideas?

I can't see how the patch is related. Both blk_flush_plug and
blk_schedule_flush_plug can sleep. The patch should only change behavior for plug which accepts from_schedule, which IIRC only raid code uses it.

Thanks,
Shaohua

#4 lore raw
Quoted 61 lines · Shaohua Li
Quoted 60 lines · Tejun Heo

Hello, Michal.

Quoted 56 lines · Michal Hocko

Hi Tejun,
I am seeing the following warning during boot in my kvm testing environment:

[    3.549649] WARNING: CPU: 3 PID: 1568 at kernel/sched/core.c:7738 __might_sleep+0x69/0x7e
[    3.551279] do not call blocking ops when !TASK_RUNNING; state=2 set at [<ffffffff81098621>] prepare_to_wait+0x65/0xac
[    3.553354] Modules linked in:
[    3.554032] CPU: 3 PID: 1568 Comm: modprobe Not tainted 4.9.0-rc8-nofstest5-next-20161209-00013-g3134766e941a #1018
[    3.556099] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3-20161025_171302-gandalf 04/01/2014
[    3.557330] Call Trace:
[    3.557330]  dump_stack+0x68/0x92
[    3.557330]  __warn+0xc2/0xdd
[    3.557330]  warn_slowpath_fmt+0x4b/0x53
[    3.557330]  ? prepare_to_wait+0x65/0xac
[    3.557330]  ? prepare_to_wait+0x65/0xac
[    3.557330]  __might_sleep+0x69/0x7e
[    3.557330]  do_ide_request+0x47/0x540
[    3.557330]  ? __lock_is_held+0x47/0x7a
[    3.557330]  ? hrtimer_try_to_cancel+0x15f/0x1c6
[    3.557330]  ? _raw_spin_unlock_irqrestore+0x4c/0x61
[    3.557330]  __blk_run_queue+0x31/0x3a
[    3.557330]  cfq_insert_request+0x331/0x429
[    3.557330]  __elv_add_request+0x2bb/0x346
[    3.557330]  blk_flush_plug_list+0x1da/0x21f
[    3.557330]  ? bit_wait+0x50/0x50
[    3.557330]  io_schedule_timeout+0x47/0x102
[    3.557330]  bit_wait_io+0x1b/0x5a
[    3.557330]  __wait_on_bit+0x4d/0x7b
[    3.557330]  out_of_line_wait_on_bit+0x72/0x7d
[    3.557330]  ? bit_wait+0x50/0x50
[    3.557330]  ? autoremove_wake_function+0x4e/0x4e
[    3.557330]  __wait_on_buffer+0x40/0x43
[    3.557330]  bh_submit_read+0x60/0x72
[    3.557330]  ext4_get_branch+0x91/0x115
[    3.557330]  ext4_ind_map_blocks+0x181/0xa16
[    3.557330]  ? __lock_is_held+0x47/0x7a
[    3.557330]  ? lock_acquire+0x133/0x1c7
[    3.557330]  ext4_map_blocks+0x120/0x4ca
[    3.557330]  ext4_mpage_readpages+0x2a7/0x70a
[    3.557330]  ext4_readpages+0x33/0x35
[    3.557330]  __do_page_cache_readahead+0x239/0x32a
[    3.557330]  filemap_fault+0x180/0x602
[    3.557330]  ? filemap_fault+0x180/0x602
[    3.557330]  ext4_filemap_fault+0x37/0x4b
[    3.557330]  __do_fault+0x1f/0x94
[    3.557330]  handle_mm_fault+0x946/0xfca
[    3.557330]  __do_page_fault+0x361/0x518
[    3.557330]  do_page_fault+0x42/0x5f
[    3.557330]  page_fault+0x22/0x30

At first I thought this would be related to you recent work
http://lkml.kernel.org/r/[email protected]
but that doesn't seem to be in the linux-next I am using as a base. io_schedule_timeout is on the stack though so you might have an idea what is going on here. That might_sleep check has been added a long ago 02e7cf8f8488 ("ide-cd,atapi: use bio for internal commands") in 2.6.31 so I am wondering what has changed since than.

Quoted 8 lines
Quoted 3 lines

10d784eae2b4 ("sched: always use blk_schedule_flush_plug in io_schedule_out") seems related. might_sleep is complaining that it's being called while !TASK_RUNNING. Shaohua, any ideas?

I can't see how the patch is related. Both blk_flush_plug and blk_schedule_flush_plug can sleep. The patch should only change behavior for plug which accepts from_schedule, which IIRC only raid code uses it.

Is there anything I could test?

Signature · 2 lines
Michal Hocko
SUSE Labs
#5 lore raw
Quoted 2 lines · Michal Hocko

Is there anything I could test?

I am working on a patchset, will send them for you to test. Meanwhile, if you could provide me a reproducer, that would help a lot.

Thanks!

#6 lore raw
Quoted 6 lines · Cong Wang
Quoted 2 lines · Michal Hocko

Is there anything I could test?

I am working on a patchset, will send them for you to test. Meanwhile, if you could provide me a reproducer, that would help a lot.

I just boot my kvm image. Nothing really special here.

Signature · 2 lines
Michal Hocko
SUSE Labs