Skip to contentSkip to conversation

btrfs: sleeping function called from invalid context

DISCUSSION Theodore Y. Ts'o 3 participants 7 messages Feb 23, 2020 – Feb 25, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hi, I noticed this when I was doing some build tests; is this a known issue?

[   74.030154] BUG: sleeping function called from invalid context at mm/slab.h:565
[   74.037763] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 14425, name: dd
[   74.046849] 4 locks held by dd/14425:
[   74.052024]  #0: ffff91960e34f498 (sb_writers#14){.+.+}, at: mnt_want_write+0x20/0x50
[   74.060008]  #1: ffff9195fe328708 (&sb->s_type->i_mutex_key#18){+.+.}, at: do_truncate+0x69/0xd0
[   74.070311]  #2: ffff91960e34f6f8 (sb_internal#2){.+.+}, at: start_transaction+0x3cf/0x550
[   74.078721]  #3: ffff919602d6c740 (btrfs-tree-00){++++}, at: btrfs_tree_lock+0x77/0x270
[   74.088791] CPU: 0 PID: 14425 Comm: dd Not tainted 5.6.0-rc2-xfstests-00009-g9db176bceb5c #1518
[   74.097618] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[   74.106966] Call Trace:
[   74.110926]  dump_stack+0x71/0xa0
[   74.114369]  ___might_sleep.cold+0xa6/0xb6
[   74.118586]  kmem_cache_alloc+0x26f/0x350
[   74.122724]  alloc_extent_state+0x20/0x1a0
[   74.126989]  __clear_extent_bit+0x2f4/0x5c0
[   74.131290]  ? _raw_spin_unlock_irq+0x24/0x30
[   74.135771]  btrfs_truncate_inode_items+0x33e/0xd90
[   74.140769]  ? do_raw_spin_unlock+0x4b/0xc0
[   74.145076]  btrfs_setsize.isra.0+0x203/0x4d0
[   74.149554]  btrfs_setattr+0x5e/0xe0
[   74.153254]  notify_change+0x357/0x4e0
[   74.157120]  do_truncate+0x76/0xd0
[   74.160654]  do_last+0x404/0x860
[   74.164004]  path_openat+0x8f/0x240
[   74.167617]  do_filp_open+0x91/0x100
[   74.171327]  ? do_raw_spin_unlock+0x4b/0xc0
[   74.175650]  ? _raw_spin_unlock+0x1f/0x30
[   74.179778]  do_sys_openat2+0x1fc/0x2a0
[   74.183742]  do_sys_open+0x44/0x80
[   74.187323]  do_syscall_64+0x50/0x1f0
[   74.192500]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[   74.197676] RIP: 0033:0x7fd0d67be1ae
[   74.201369] Code: 25 00 00 41 00 3d 00 00 41 00 74 48 48 8d 05 59 65 0d 00 8b 00 85 c0 75 69 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a6 00 00 00 48 8b 4c 24 28 64 48 33 0c 25
[   74.221644] RSP: 002b:00007fff579e3000 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[   74.229331] RAX: ffffffffffffffda RBX: 000055cbdb1853e0 RCX: 00007fd0d67be1ae
[   74.236586] RDX: 0000000000000241 RSI: 00007fff579e4e2c RDI: 00000000ffffff9c
[   74.243835] RBP: 0000000000000001 R08: 000055cbdb17f2e6 R09: 0000000000000000
[   74.251092] R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000000241
[   74.258452] R13: 00007fff579e4e2c R14: 0000000000000001 R15: 00007fff579e32f8

It's causing very large number of xfstests failures:

TESTRUNID: tytso-20200222194637
KERNEL:    kernel 5.6.0-rc2-xfstests-00009-g9db176bceb5c #1518 SMP Fri Feb 21 19:32:23 EST 2020 x86_64
CMDLINE:   -c btrfs -g auto
CPUS:      2
MEM:       7680

btrfs/default: 988 tests, 300 failures, 203 skipped, 8951 seconds
Failures: btrfs/002 btrfs/004 btrfs/005 btrfs/007 btrfs/013

    btrfs/014 btrfs/020 btrfs/022 btrfs/024 btrfs/026 btrfs/028 
    btrfs/034 btrfs/037 btrfs/041 btrfs/055 btrfs/056 btrfs/057 
    btrfs/076 btrfs/078 btrfs/080 btrfs/081 btrfs/094 btrfs/095 
    btrfs/096 btrfs/098 btrfs/108 btrfs/109 btrfs/119 btrfs/137 
    btrfs/138 btrfs/139 btrfs/153 btrfs/156 btrfs/159 btrfs/169 
    btrfs/173 btrfs/174 btrfs/177 btrfs/179 btrfs/183 btrfs/188 
    btrfs/191 btrfs/199 btrfs/200 btrfs/201 btrfs/204 generic/001 
    generic/005 generic/006 generic/007 generic/008 generic/009 
    generic/011 generic/013 generic/014 generic/018 generic/027 
    generic/030 generic/032 generic/033 generic/039 generic/040 
    generic/041 generic/056 generic/057 generic/062 generic/066 
    generic/068 generic/070 generic/074 generic/075 generic/076 
    generic/077 generic/078 generic/079 generic/080 generic/083 
    generic/086 generic/087 generic/089 generic/090 generic/091 
    generic/092 generic/093 generic/094 generic/095 generic/096 
    generic/097 generic/099 generic/101 generic/103 generic/104 
    generic/105 generic/107 generic/109 generic/110 generic/112 
    generic/113 generic/114 generic/116 generic/117 generic/118 
    generic/119 generic/121 generic/122 generic/123 generic/124 
    generic/126 generic/127 generic/129 generic/130 generic/131 
    generic/133 generic/134 generic/136 generic/137 generic/138 
    generic/139 generic/140 generic/142 generic/143 generic/144 
    generic/146 generic/149 generic/150 generic/151 generic/152 
    generic/154 generic/155 generic/157 generic/158 generic/159 
    generic/169 generic/175 generic/176 generic/177 generic/178 
    generic/179 generic/180 generic/181 generic/182 generic/193 
    generic/198 generic/199 generic/200 generic/204 generic/209 
    generic/213 generic/225 generic/239 generic/240 generic/241 
    generic/245 generic/248 generic/249 generic/254 generic/255 
    generic/256 generic/257 generic/259 generic/260 generic/263 
    generic/269 generic/277 generic/285 generic/286 generic/297 
    generic/298 generic/299 generic/300 generic/301 generic/302 
    generic/303 generic/304 generic/306 generic/307 generic/308 
    generic/309 generic/310 generic/311 generic/313 generic/315 
    generic/316 generic/318 generic/319 generic/320 generic/321 
    generic/322 generic/324 generic/325 generic/329 generic/336 
    generic/337 generic/339 generic/342 generic/343 generic/347 
    generic/348 generic/352 generic/353 generic/356 generic/357 
    generic/361 generic/375 generic/376 generic/377 generic/389 
    generic/390 generic/391 generic/393 generic/407 generic/408 
    generic/409 generic/410 generic/411 generic/414 generic/415 
    generic/416 generic/418 generic/420 generic/423 generic/426 
    generic/427 generic/434 generic/436 generic/438 generic/439 
    generic/444 generic/446 generic/447 generic/448 generic/450 
    generic/451 generic/454 generic/463 generic/464 generic/465 
    generic/467 generic/469 generic/472 generic/475 generic/476 
    generic/477 generic/479 generic/480 generic/481 generic/483 
    generic/488 generic/489 generic/490 generic/493 generic/494 
    generic/495 generic/496 generic/498 generic/501 generic/502 
    generic/509 generic/510 generic/511 generic/512 generic/515 
    generic/516 generic/520 generic/523 generic/524 generic/526 
    generic/527 generic/528 generic/531 generic/532 generic/534 
    generic/535 generic/538 generic/539 generic/541 generic/543 
    generic/545 generic/546 generic/547 generic/551 generic/552 
    generic/553 generic/555 generic/557 generic/560 generic/561 
    generic/562 generic/564 generic/567 generic/569 generic/571 
    generic/578 generic/585 generic/586 generic/588 generic/589 
    generic/591 shared/002 shared/298 

Totals: 785 tests, 203 skipped, 300 failures, 0 errors, 8909s

FSTESTIMG: gce-xfstests/xfstests-202002211357
FSTESTPRJ: gce-xfstests
FSTESTVER: blktests f7b47c5 (Tue, 11 Feb 2020 14:22:21 -0800)
FSTESTVER: e2fsprogs v1.45.4-15-g4b4f7b35 (Wed, 9 Oct 2019 20:25:01 -0400)
FSTESTVER: fio  fio-3.18 (Wed, 5 Feb 2020 07:59:58 -0700)
FSTESTVER: fsverity v1.0 (Wed, 6 Nov 2019 10:35:02 -0800)
FSTESTVER: ima-evm-utils v1.2 (Fri, 26 Jul 2019 07:42:17 -0400)
FSTESTVER: nvme-cli v1.10.1 (Tue, 7 Jan 2020 13:55:21 -0700)
FSTESTVER: quota  9a001cc (Tue, 5 Nov 2019 16:12:59 +0100)
FSTESTVER: util-linux v2.35 (Tue, 21 Jan 2020 11:15:21 +0100)
FSTESTVER: xfsprogs v5.4.0 (Fri, 20 Dec 2019 16:47:12 -0500)
FSTESTVER: xfstests-bld a7ae9ff (Tue, 18 Feb 2020 14:22:36 -0500)
FSTESTVER: xfstests linux-v3.8-2692-g3fe2fd0d (Fri, 21 Feb 2020 13:42:43 -0500)
FSTESTCFG: btrfs
FSTESTSET: -g auto
FSTESTOPT: aex
GCE ID:    8335279961640039838
#1 lore raw
Quoted 2 lines · Theodore Y. Ts'o

Hi, I noticed this when I was doing some build tests; is this a known issue?

So this is fallout from 28553fa992cb28be6a65566681aac6cafabb4f2d because it's being called while we have locked extent buffers (before calling btrfs_free_Path which is holding a rwlock (a variant of spinlock). And actually unlocking btrfs' extent requires allocating structures to reflect the new state. This allocation is currently done with GFP_NOFS which implies DIRECT_RECLAIM hence the maybe sleep from slab allocator is triggered.

Filipe, can the unlock be done _after_ freeing the path or even better - reduce the critical section altogether in btrfs_truncate_inode_items?

I don't think '[PATCH] Btrfs: fix deadlock during fast fsync when logging prealloc extents beyond eof' actually fixes the problem since the unlock can happen under the path again.

#2 lore raw
Quoted 8 lines · Nikolay Borisov

So this is fallout from 28553fa992cb28be6a65566681aac6cafabb4f2d because it's being called while we have locked extent buffers (before calling btrfs_free_Path which is holding a rwlock (a variant of spinlock). And actually unlocking btrfs' extent requires allocating structures to reflect the new state. This allocation is currently done with GFP_NOFS which implies DIRECT_RECLAIM hence the maybe sleep from slab allocator is triggered.

Quoted 6 lines

Filipe, can the unlock be done _after_ freeing the path or even better - reduce the critical section altogether in btrfs_truncate_inode_items?

I don't think '[PATCH] Btrfs: fix deadlock during fast fsync when logging prealloc extents beyond eof' actually fixes the problem since the unlock can happen under the path again.

Hmm... I don't know if the problem has been *completely* fixed, but I can say that with -rc3 (which has the "fix deadlock during fast fsync when..." commit), I'm no longer seeing the kernel complaints when I run "gce-xfstests -c btrfs -g auto". Here are the test results:

TESTRUNID: tytso-20200223230308
KERNEL:    kernel 5.6.0-rc3-xfstests #1522 SMP Sun Feb 23 23:01:10 EST 2020 x86_64
CMDLINE:   -c btrfs -g auto
CPUS:      2
MEM:       7680

btrfs/default: 988 tests, 8 failures, 203 skipped, 8739 seconds
Failures: btrfs/056 btrfs/153 btrfs/204 generic/065 generic/260

    generic/475 generic/562 shared/298 

Totals: 785 tests, 203 skipped, 8 failures, 0 errors, 8678s

FSTESTIMG: gce-xfstests/xfstests-202002211357
FSTESTPRJ: gce-xfstests
FSTESTVER: blktests f7b47c5 (Tue, 11 Feb 2020 14:22:21 -0800)
FSTESTVER: e2fsprogs v1.45.4-15-g4b4f7b35 (Wed, 9 Oct 2019 20:25:01 -0400)
FSTESTVER: fio  fio-3.18 (Wed, 5 Feb 2020 07:59:58 -0700)
FSTESTVER: fsverity v1.0 (Wed, 6 Nov 2019 10:35:02 -0800)
FSTESTVER: ima-evm-utils v1.2 (Fri, 26 Jul 2019 07:42:17 -0400)
FSTESTVER: nvme-cli v1.10.1 (Tue, 7 Jan 2020 13:55:21 -0700)
FSTESTVER: quota  9a001cc (Tue, 5 Nov 2019 16:12:59 +0100)
FSTESTVER: util-linux v2.35 (Tue, 21 Jan 2020 11:15:21 +0100)
FSTESTVER: xfsprogs v5.4.0 (Fri, 20 Dec 2019 16:47:12 -0500)
FSTESTVER: xfstests-bld a7ae9ff (Tue, 18 Feb 2020 14:22:36 -0500)
FSTESTVER: xfstests linux-v3.8-2692-g3fe2fd0d (Fri, 21 Feb 2020 13:42:43 -0500)
FSTESTCFG: btrfs
FSTESTSET: -g auto
FSTESTOPT: aex
GCE ID:    9110223165715314154

If you want the full test artifacts for this run, in case you want to dig into the test failures, let me know; I'm happy to send them. The compressed tarfile is is 1952k, so it's a bit too large for the vger mailing list.

Cheers,

- Ted

#6 lore raw
In reply to Nikolay Borisov
Quoted 15 lines · Nikolay Borisov
Quoted 2 lines · Theodore Y. Ts'o

Hi, I noticed this when I was doing some build tests; is this a known issue?

So this is fallout from 28553fa992cb28be6a65566681aac6cafabb4f2d because it's being called while we have locked extent buffers (before calling btrfs_free_Path which is holding a rwlock (a variant of spinlock). And actually unlocking btrfs' extent requires allocating structures to reflect the new state. This allocation is currently done with GFP_NOFS which implies DIRECT_RECLAIM hence the maybe sleep from slab allocator is triggered.

Quoted 6 lines

Filipe, can the unlock be done _after_ freeing the path or even better - reduce the critical section altogether in btrfs_truncate_inode_items?

I don't think '[PATCH] Btrfs: fix deadlock during fast fsync when logging prealloc extents beyond eof' actually fixes the problem since the unlock can happen under the path again.

The problem is that that patch was made against the integration branch (misc-next) where truncate doesn't use btree search paths with spinning locks anymore, so it didn't trigger that problem during development.
The patch was then picked into 5.6-rc2, where we still use spinning locks, whence people reporting the issue (before Ted, Dave Jones and a few others had reported it as well).

The solution was to pick a patch from the integration branch into 5.6-rc3:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=52e29e331070cd7d52a64cbf1b0958212a340e28

That solves the sleep in invalid context warnings.

Quoted 1 line
Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#3 lore raw
In reply to Theodore Y. Ts'o
Quoted 51 lines · Theodore Y. Ts'o
Quoted 8 lines · Nikolay Borisov

So this is fallout from 28553fa992cb28be6a65566681aac6cafabb4f2d because it's being called while we have locked extent buffers (before calling btrfs_free_Path which is holding a rwlock (a variant of spinlock). And actually unlocking btrfs' extent requires allocating structures to reflect the new state. This allocation is currently done with GFP_NOFS which implies DIRECT_RECLAIM hence the maybe sleep from slab allocator is triggered.

Quoted 6 lines

Filipe, can the unlock be done _after_ freeing the path or even better - reduce the critical section altogether in btrfs_truncate_inode_items?

I don't think '[PATCH] Btrfs: fix deadlock during fast fsync when logging prealloc extents beyond eof' actually fixes the problem since the unlock can happen under the path again.

Hmm... I don't know if the problem has been *completely* fixed, but I can say that with -rc3 (which has the "fix deadlock during fast fsync when..." commit), I'm no longer seeing the kernel complaints when I run "gce-xfstests -c btrfs -g auto". Here are the test results:

TESTRUNID: tytso-20200223230308
KERNEL:    kernel 5.6.0-rc3-xfstests #1522 SMP Sun Feb 23 23:01:10 EST 2020 x86_64
CMDLINE:   -c btrfs -g auto
CPUS:      2
MEM:       7680

btrfs/default: 988 tests, 8 failures, 203 skipped, 8739 seconds
Failures: btrfs/056 btrfs/153 btrfs/204 generic/065 generic/260

    generic/475 generic/562 shared/298

Totals: 785 tests, 203 skipped, 8 failures, 0 errors, 8678s

FSTESTIMG: gce-xfstests/xfstests-202002211357
FSTESTPRJ: gce-xfstests
FSTESTVER: blktests f7b47c5 (Tue, 11 Feb 2020 14:22:21 -0800)
FSTESTVER: e2fsprogs v1.45.4-15-g4b4f7b35 (Wed, 9 Oct 2019 20:25:01 -0400)
FSTESTVER: fio  fio-3.18 (Wed, 5 Feb 2020 07:59:58 -0700)
FSTESTVER: fsverity v1.0 (Wed, 6 Nov 2019 10:35:02 -0800)
FSTESTVER: ima-evm-utils v1.2 (Fri, 26 Jul 2019 07:42:17 -0400)
FSTESTVER: nvme-cli v1.10.1 (Tue, 7 Jan 2020 13:55:21 -0700)
FSTESTVER: quota  9a001cc (Tue, 5 Nov 2019 16:12:59 +0100)
FSTESTVER: util-linux v2.35 (Tue, 21 Jan 2020 11:15:21 +0100)
FSTESTVER: xfsprogs v5.4.0 (Fri, 20 Dec 2019 16:47:12 -0500)
FSTESTVER: xfstests-bld a7ae9ff (Tue, 18 Feb 2020 14:22:36 -0500)
FSTESTVER: xfstests linux-v3.8-2692-g3fe2fd0d (Fri, 21 Feb 2020 13:42:43 -0500)
FSTESTCFG: btrfs
FSTESTSET: -g auto
FSTESTOPT: aex
GCE ID:    9110223165715314154
Quoted 4 lines

If you want the full test artifacts for this run, in case you want to dig into the test failures, let me know; I'm happy to send them. The compressed tarfile is is 1952k, so it's a bit too large for the vger mailing list.

We do have some tests that fail in any kernel release so far. Some because the corresponding fixes are not yet merged or some fail often due to known problems.
Looking at your list of failure, I see some that shouldn't be failing like btrfs/053.

If you want you can send me the test logs to my gmail address. Thanks.

Quoted 4 lines

Cheers,

- Ted

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#4 lore raw
Quoted 5 lines · Filipe Manana

We do have some tests that fail in any kernel release so far. Some because the corresponding fixes are not yet merged or some fail often due to known problems.
Looking at your list of failure, I see some that shouldn't be failing like btrfs/053.

I've sent you the compressed tarfile with the test artifacts under separate cover. The files that you'll probably want to look at first are ./runtests.log and ./syslog. The xfstests results artifacts will be in ./btrfs/results-default/.

If you have a wiki page or some other pointer of what tests that you expect to fail, I can put them into a btrfs-specific or file system configuration specific exclude file. For example, see [1] and [2].

[1] https://github.com/tytso/xfstests-bld/blob/master/kvm-xfstests/test-appliance/files/root/fs/ext4/exclude
[2] https://github.com/tytso/xfstests-bld/blob/master/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/bigalloc.exclude

I'm planning on running btrfs and xfs tests more frequently to support some $WORK initiatives. So if there are tests which are known failures that would be good for me to suppress, and if there are some file system configurations that would be useful for me to run, please let me know and I'm happy to set them so that gce-xfstests and kvm-xfstests can better test btrfs.

Also, I assume you do have some btrfs developers who are regularly running xfstests, so I don't know how helpful this would be to you, but given that I'm going to be running the tests *anyway*, if it would be helpful for me to forward test results to you, or to only send you a note when test regressions show up, I'm happy to do that.

Cheers,

- Ted

#5 lore raw
Quoted 8 lines · Theodore Y. Ts'o
Quoted 5 lines · Filipe Manana

We do have some tests that fail in any kernel release so far. Some because the corresponding fixes are not yet merged or some fail often due to known problems.
Looking at your list of failure, I see some that shouldn't be failing like btrfs/053.

Quoted 8 lines

I've sent you the compressed tarfile with the test artifacts under separate cover. The files that you'll probably want to look at first are ./runtests.log and ./syslog. The xfstests results artifacts will be in ./btrfs/results-default/.

If you have a wiki page or some other pointer of what tests that you expect to fail, I can put them into a btrfs-specific or file system configuration specific exclude file. For example, see [1] and [2].

I don't think we do. Usually people keep their own list. It's also very frequent to have tests fail because the corresponding patches that fix them were not yet merged, specially for recently added tests.

Quoted 11 lines

[1] https://github.com/tytso/xfstests-bld/blob/master/kvm-xfstests/test-appliance/files/root/fs/ext4/exclude
[2] https://github.com/tytso/xfstests-bld/blob/master/kvm-xfstests/test-appliance/files/root/fs/ext4/cfg/bigalloc.exclude

I'm planning on running btrfs and xfs tests more frequently to support some $WORK initiatives. So if there are tests which are known failures that would be good for me to suppress, and if there are some file system configurations that would be useful for me to run, please let me know and I'm happy to set them so that gce-xfstests and kvm-xfstests can better test btrfs.

Quoted 2 lines

Also, I assume you do have some btrfs developers who are regularly running xfstests,

I run them daily. And I suppose other developers run them frequently as well.

Quoted 4 lines

so I don't know how helpful this would be to you,
but given that I'm going to be running the tests *anyway*, if it would be helpful for me to forward test results to you, or to only send you a note when test regressions show up, I'm happy to do that.

Regarding the failures you got:

btrfs/056 - failure to setup dmflakey, this happens sporadically to me and others as well, for any test that uses dmflakey and it seems to happen more often when running dmflakey tests in sequence

btrfs/153 - has been failing for a long time, known issue, just ignore it for now

btrfs/204 - expected to fail, there's a fix in the mailing list but it wasn't merged yet (https://patchwork.kernel.org/patch/11357415/)

generic/065 - another failure to create the dmflakey device

generic/260 - expected to fail on btrfs, due to the way space is allocated and managed in btrfs

generic/475 - this one can fail often for several different reasons. Some reasons why it can fail are fixed by patches sent to rc3 for example, other problems like the one you hit (missing file extent holes) are addressed by a patchset that is in the integration branch and will likely go into the next merge window

generic/562 - fails with ENOSPC. Doesn't fail for me here, needs to be investigated.

shared/298 - The test fails with error trying to open a file that doesn't exists apparently (/root/xfstests/src/parse-extent-tree.awk). Never seen this failure in my vms, the test always passes for me. Needs investigation.

Thanks for the report and logs.

Quoted 4 lines

Cheers,

- Ted

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”