Skip to contentSkip to conversation

dcache_readdir NULL inode oops

DISCUSSION Jan Glauber 6 participants 14 messages Nov 9, 2018 – Apr 30, 2019 VFS

lore
Share & tools
lore
#0 lore raw

Hi Al,

I'm seeing the following oops reproducible with upstream kernel on arm64 (ThunderX2):

[ 5428.795719] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
[ 5428.813838] Mem abort info:
[ 5428.820721]   ESR = 0x96000006
[ 5428.828476]   Exception class = DABT (current EL), IL = 32 bits
[ 5428.841590]   SET = 0, FnV = 0
[ 5428.848939]   EA = 0, S1PTW = 0
[ 5428.855941] Data abort info:
[ 5428.862422]   ISV = 0, ISS = 0x00000006
[ 5428.870787]   CM = 0, WnR = 0
[ 5428.877359] user pgtable: 4k pages, 48-bit VAs, pgdp = 0000000052f9e034
[ 5428.891098] [0000000000000040] pgd=0000007ebb0d6003, pud=0000007ed3073003, pmd=0000000000000000
[ 5428.909251] Internal error: Oops: 96000006 [#1] SMP
[ 5428.919122] Modules linked in: xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 xt_tcpudp bridge stp llc ip6table_filter ip6_tables iptable_filter ipmi_ssif ip_tables x_tables ipv6 crc32_ce bnx2x crct10dif_ce igb nvme nvme_core i2c_algo_bit mdio gpio_xlp i2c_xlp9xx
[ 5428.972724] CPU: 45 PID: 220018 Comm: stress-ng-dev Not tainted 4.19.0-jang+ #45
[ 5428.987664] Hardware name: To be filled by O.E.M. Saber/To be filled by O.E.M., BIOS 0ACKL018 03/30/2018
[ 5429.006819] pstate: 60400009 (nZCv daif +PAN -UAO)
[ 5429.016567] pc : dcache_readdir+0xfc/0x1a8
[ 5429.024903] lr : dcache_readdir+0x134/0x1a8
[ 5429.033376] sp : ffff00002d553d70
[ 5429.040101] x29: ffff00002d553d70 x28: ffff807db4988000
[ 5429.050892] x27: 0000000000000000 x26: 0000000000000000
[ 5429.061679] x25: 0000000056000000 x24: ffff8024577106c0
[ 5429.072457] x23: 0000000000000000 x22: ffff80267b92a480
[ 5429.083248] x21: ffff80267b92a520 x20: ffff8024575e5e00
[ 5429.094029] x19: ffff00002d553e40 x18: 0000000000000000
[ 5429.104805] x17: 0000000000000000 x16: 0000000000000000
[ 5429.115553] x15: 0000000000000000 x14: 0000000000000000
[ 5429.126332] x13: 0000000000000000 x12: 0000000000000000
[ 5429.137096] x11: 0000000000000000 x10: ffff80266b398228
[ 5429.147849] x9 : ffff80266b398000 x8 : 0000000000007e4e
[ 5429.158580] x7 : 0000000000000000 x6 : ffff00000830d190
[ 5429.169362] x5 : 0000000000000000 x4 : ffff00000d7506a8
[ 5429.180123] x3 : 0000000000000002 x2 : 0000000000000002
[ 5429.190890] x1 : ffff8024575e5e38 x0 : ffff00002d553e40
[ 5429.201715] Process stress-ng-dev (pid: 220018, stack limit = 0x000000009437ac28)
[ 5429.216828] Call trace:
[ 5429.221855]  dcache_readdir+0xfc/0x1a8
[ 5429.229459]  iterate_dir+0x8c/0x1a0
[ 5429.236561]  ksys_getdents64+0xa4/0x188
[ 5429.244357]  __arm64_sys_getdents64+0x28/0x38
[ 5429.253201]  el0_svc_handler+0x7c/0x100
[ 5429.260989]  el0_svc+0x8/0xc
[ 5429.266878] Code: a9429681 aa1303e0 b9402682 a9400e66 (f94020a4)
[ 5429.279192] ---[ end trace 5c1e28c07cf016c5 ]---

It happens after 1-3 hours of running 'stress-ng --dev 128'. This testcase does a scandir of /dev and then calls random stuff like ioctl, lseek, open/close etc. on the entries. I assume no files are deleted under /dev during the testcase.

The NULL pointer is the inode pointer of next. The next dentry->d_flags is DCACHE_RCUACCESS when this happens.

Any hints on how to further debug this?

--Jan

#1 lore raw
Quoted 2 lines · Jan Glauber

I'm seeing the following oops reproducible with upstream kernel on arm64 (ThunderX2):

[...]

Quoted 9 lines

It happens after 1-3 hours of running 'stress-ng --dev 128'. This testcase does a scandir of /dev and then calls random stuff like ioctl, lseek, open/close etc. on the entries. I assume no files are deleted under /dev during the testcase.

The NULL pointer is the inode pointer of next. The next dentry->d_flags is DCACHE_RCUACCESS when this happens.

Any hints on how to further debug this?

Can you reproduce the issue with vanilla -rc1 and do you have a "known good" kernel?

Will

#2 lore raw
Quoted 11 lines · Will Deacon
Quoted 2 lines · Jan Glauber

I'm seeing the following oops reproducible with upstream kernel on arm64 (ThunderX2):

[...]

Quoted 5 lines

It happens after 1-3 hours of running 'stress-ng --dev 128'. This testcase does a scandir of /dev and then calls random stuff like ioctl, lseek, open/close etc. on the entries. I assume no files are deleted under /dev during the testcase.

Quoted 7 lines
Quoted 4 lines

The NULL pointer is the inode pointer of next. The next dentry->d_flags is DCACHE_RCUACCESS when this happens.

Any hints on how to further debug this?

Can you reproduce the issue with vanilla -rc1 and do you have a "known good" kernel?

I can try out -rc1, but IIRC this wasn't bisectible as the bug was present at least back to 4.14. I need to double check that as there were other issues that are resolved now so I may confuse things here. I've defintely seen the same bug with 4.18.

Unfortunately I lost access to the machine as our data center seems to be moving currently so it might take some days until I can try -rc1.

thanks,
Jan

#3 lore raw
Quoted 20 lines · Jan Glauber
Quoted 11 lines · Will Deacon
Quoted 2 lines · Jan Glauber

I'm seeing the following oops reproducible with upstream kernel on arm64 (ThunderX2):

[...]

Quoted 5 lines

It happens after 1-3 hours of running 'stress-ng --dev 128'. This testcase does a scandir of /dev and then calls random stuff like ioctl, lseek, open/close etc. on the entries. I assume no files are deleted under /dev during the testcase.

Quoted 7 lines
Quoted 4 lines

The NULL pointer is the inode pointer of next. The next dentry->d_flags is DCACHE_RCUACCESS when this happens.

Any hints on how to further debug this?

Can you reproduce the issue with vanilla -rc1 and do you have a "known good" kernel?

Quoted 7 lines

I can try out -rc1, but IIRC this wasn't bisectible as the bug was present at least back to 4.14. I need to double check that as there were other issues that are resolved now so I may confuse things here. I've defintely seen the same bug with 4.18.

Unfortunately I lost access to the machine as our data center seems to be moving currently so it might take some days until I can try -rc1.

Ok, I've just managed to reproduce this in a KVM guest running v4.20-rc3 on both the host and the guest, so if anybody has any ideas of things to try then I'm happy to give them a shot. In the meantime, I'll try again with a bunch of debug checks enabled.

Interestingly, I see many CPUs crashing one after the other in the same place with *0x40, which indicates that the underlying data structure is corrupted somehow. The final crash was in a different place with *0x10, which I've also included below.

Will

--->8

[  353.086276] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
[  353.088334] Mem abort info:
[  353.088501]   ESR = 0x96000004
[  353.123277]   Exception class = DABT (current EL), IL = 32 bits
[  353.126126]   SET = 0, FnV = 0
[  353.127064]   EA = 0, S1PTW = 0
[  353.127917] Data abort info:
[  353.130869]   ISV = 0, ISS = 0x00000004
[  353.131793]   CM = 0, WnR = 0
[  353.133998] user pgtable: 4k pages, 48-bit VAs, pgdp = 00000000344077db
[  353.135410] [0000000000000040] pgd=0000000000000000
[  353.137903] Internal error: Oops: 96000004 [#1] PREEMPT SMP
[  353.139146] Modules linked in:
[  353.140232] CPU: 41 PID: 2514 Comm: stress-ng-dev Not tainted 4.20.0-rc3-00012-g40b114779944 #1
[  353.140367] Hardware name: linux,dummy-virt (DT)
[  353.190775] pstate: 40400005 (nZcv daif +PAN -UAO)
[  353.191833] pc : dcache_readdir+0xd0/0x170
[  353.193058] lr : dcache_readdir+0x108/0x170
[  353.194075] sp : ffff00000e17bd70
[  353.195027] x29: ffff00000e17bd70 x28: ffff8003cbe60000 
[  353.196232] x27: 0000000000000000 x26: 0000000000000000 
[  353.196334] x25: 0000000056000000 x24: ffff80037e3a9200 
[  353.255951] x23: 0000000000000000 x22: ffff8003d692ae40 
[  353.257708] x21: ffff8003d692aee0 x20: ffff00000e17be40 
[  353.259044] x19: ffff80037d875b00 x18: 0000000000000000 
[  353.259210] x17: 0000000000000000 x16: 0000000000000000 
[  353.259354] x15: 0000000000000000 x14: 0000000000000000 
[  353.259469] x13: 0000000000000000 x12: 0000000000000000 
[  353.259610] x11: 0000000000000000 x10: 0000000000000000 
[  353.259746] x9 : 0000ffffffffffff x8 : 0000ffffffffffff 
[  353.422637] x7 : 0000000000000005 x6 : ffff000008245768 
[  353.422639] x5 : 0000000000000000 x4 : 0000000000002000 
[  353.422640] x3 : 0000000000000002 x2 : 0000000000000001 
[  353.422642] x1 : ffff80037d875b38 x0 : ffff00000e17be40 
[  353.422646] Process stress-ng-dev (pid: 2514, stack limit = 0x000000006721788f)
[  353.422647] Call trace:
[  353.422654]  dcache_readdir+0xd0/0x170
[  353.422664]  iterate_dir+0x13c/0x190
[  353.429254]  ksys_getdents64+0x88/0x168
[  353.429256]  __arm64_sys_getdents64+0x1c/0x28
[  353.429260]  el0_svc_common+0x84/0xd8
[  353.429261]  el0_svc_handler+0x2c/0x80
[  353.429264]  el0_svc+0x8/0xc
[  353.429267] Code: a9429661 aa1403e0 a9400e86 b9402662 (f94020a4) 
[  353.429272] ---[ end trace 7bc53f0d6caaf0d1 ]---
[ 1770.346163] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000010
[ 1770.364229] Mem abort info:
[ 1770.364411]   ESR = 0x96000004
[ 1770.364419]   Exception class = DABT (current EL), IL = 32 bits
[ 1770.364434]   SET = 0, FnV = 0
[ 1770.364441]   EA = 0, S1PTW = 0
[ 1770.364442] Data abort info:
[ 1770.364443]   ISV = 0, ISS = 0x00000004
[ 1770.364444]   CM = 0, WnR = 0
[ 1770.364480] user pgtable: 4k pages, 48-bit VAs, pgdp = 00000000d05dfa48
[ 1770.364491] [0000000000000010] pgd=0000000000000000
[ 1770.364537] Internal error: Oops: 96000004 [#34] PREEMPT SMP
[ 1770.364586] Modules linked in:
[ 1770.364592] CPU: 2 PID: 2491 Comm: stress-ng-dev Tainted: G      D           4.20.0-rc3-00012-g40b114779944 #1
[ 1770.364594] Hardware name: linux,dummy-virt (DT)
[ 1770.364596] pstate: 60400005 (nZCv daif +PAN -UAO)
[ 1770.364665] pc : n_tty_ioctl+0x128/0x1a0
[ 1770.364668] lr : n_tty_ioctl+0xac/0x1a0
[ 1770.364669] sp : ffff00000e723ca0
[ 1770.364691] x29: ffff00000e723ca0 x28: ffff8003d2a94f80 
[ 1770.485270] x27: 0000000000000000 x26: 0000000000000000 
[ 1770.485343] x25: ffff8003955a9780 x24: 0000fffff3c025f0 
[ 1770.485346] x23: ffff80038ad46100 x22: ffff800394c1c0c0 
[ 1770.496821] x21: 0000000000000000 x20: ffff800394c1c000 
[ 1770.496824] x19: 0000fffff3c025f0 x18: 0000000000000000 
[ 1770.496825] x17: 0000000000000000 x16: 0000000000000000 
[ 1770.496827] x15: 0000000000000000 x14: 0000000000000000 
[ 1770.496828] x13: 0000000000000000 x12: 0000000000000000 
[ 1770.496829] x11: 0000000000000000 x10: 0000000000000000 
[ 1770.496830] x9 : 0000000000000000 x8 : 0000000000000000 
[ 1770.496831] x7 : 0000000000000000 x6 : 0000000000000000 
[ 1770.496833] x5 : 000000000000541b x4 : ffff0000085b4780 
[ 1770.496834] x3 : 0000fffff3c025f0 x2 : 000000000000541b 
[ 1770.496835] x1 : ffffffff00000001 x0 : 0000000000000002 
[ 1770.496839] Process stress-ng-dev (pid: 2491, stack limit = 0x000000001177919b)
[ 1770.496840] Call trace:
[ 1770.496845]  n_tty_ioctl+0x128/0x1a0
[ 1770.496847]  tty_ioctl+0x2fc/0xb70
[ 1770.496851]  do_vfs_ioctl+0xb8/0x890
[ 1770.496853]  ksys_ioctl+0x78/0xa8
[ 1770.496854]  __arm64_sys_ioctl+0x1c/0x28
[ 1770.496858]  el0_svc_common+0x84/0xd8
[ 1770.496860]  el0_svc_handler+0x2c/0x80
[ 1770.496863]  el0_svc+0x8/0xc
[ 1770.496865] Code: a94153f3 a9425bf5 a8c37bfd d65f03c0 (f9400aa4) 
[ 1770.496869] ---[ end trace 7bc53f0d6caaf0f2 ]---
#4 lore raw
Quoted 26 lines · Will Deacon
Quoted 25 lines · Jan Glauber
Quoted 11 lines · Will Deacon
Quoted 2 lines · Jan Glauber

I'm seeing the following oops reproducible with upstream kernel on arm64 (ThunderX2):

[...]

Quoted 5 lines

It happens after 1-3 hours of running 'stress-ng --dev 128'. This testcase does a scandir of /dev and then calls random stuff like ioctl, lseek, open/close etc. on the entries. I assume no files are deleted under /dev during the testcase.

Quoted 7 lines
Quoted 4 lines

The NULL pointer is the inode pointer of next. The next dentry->d_flags is DCACHE_RCUACCESS when this happens.

Any hints on how to further debug this?

Can you reproduce the issue with vanilla -rc1 and do you have a "known good" kernel?

I can try out -rc1, but IIRC this wasn't bisectible as the bug was present at least back to 4.14. I need to double check that as there were other issues that are resolved now so I may confuse things here. I've defintely seen the same bug with 4.18.

Quoted 7 lines
Quoted 2 lines

Unfortunately I lost access to the machine as our data center seems to be moving currently so it might take some days until I can try -rc1.

Ok, I've just managed to reproduce this in a KVM guest running v4.20-rc3 on both the host and the guest, so if anybody has any ideas of things to try then I'm happy to give them a shot. In the meantime, I'll try again with a bunch of debug checks enabled.

Weee, I eventually hit a use-after-free from KASAN. See below.

Will

--->8

[  615.973367] ==================================================================
[  615.974675] BUG: KASAN: use-after-free in next_positive.isra.2+0x188/0x1a0
[  615.975574] Read of size 8 at addr ffff8002fb33c190 by task stress-ng-dev/3145
[  615.977348] 
[  615.977692] CPU: 16 PID: 3145 Comm: stress-ng-dev Tainted: G      D           4.20.0-rc3-00012-g40b114779944 #2
[  615.980171] Hardware name: linux,dummy-virt (DT)
[  615.981325] Call trace:
[  615.981765]  dump_backtrace+0x0/0x280
[  615.982386]  show_stack+0x14/0x20
[  615.983125]  dump_stack+0xc4/0xec
[  615.983141]  print_address_description+0x60/0x25c
[  615.985226]  kasan_report+0x1a8/0x358
[  615.986161]  __asan_report_load8_noabort+0x18/0x20
[  615.986978]  next_positive.isra.2+0x188/0x1a0
[  615.987767]  dcache_readdir+0x2cc/0x488
[  615.988428]  iterate_dir+0x168/0x448
[  615.989342]  ksys_getdents64+0xe8/0x248
[  615.990334]  __arm64_sys_getdents64+0x68/0x98
[  615.990341]  el0_svc_common+0x104/0x210
[  615.990345]  el0_svc_handler+0x48/0xb0
[  615.990349]  el0_svc+0x8/0xc
[  615.990356] 
[  615.994175] Allocated by task 2720:
[  615.994184]  kasan_kmalloc.part.1+0x40/0x108
[  615.994188]  kasan_kmalloc+0xb4/0xc8
[  615.994192]  kasan_slab_alloc+0x14/0x20
[  615.994195]  kmem_cache_alloc+0x130/0x1f8
[  615.994203]  __d_alloc+0x30/0x848
[  615.994215]  d_alloc+0x30/0x1d0
[  616.000554]  d_alloc_name+0x84/0xb0
[  616.000562]  devpts_pty_new+0x2e0/0x5e8
[  616.000568]  ptmx_open+0x14c/0x288
[  616.000576]  chrdev_open+0x194/0x408
[  616.000586]  do_dentry_open+0x2e8/0xac8
[  616.004282]  vfs_open+0x8c/0xc0
[  616.004286]  path_openat+0x694/0x33e8
[  616.004288]  do_filp_open+0x13c/0x200
[  616.004296]  do_sys_open+0x1dc/0x2e0
[  616.006865]  __arm64_sys_openat+0x88/0xc8
[  616.006872]  el0_svc_common+0x104/0x210
[  616.006876]  el0_svc_handler+0x48/0xb0
[  616.006880]  el0_svc+0x8/0xc
[  616.006881] 
[  616.006883] Freed by task 0:
[  616.006889]  __kasan_slab_free+0x114/0x228
[  616.006897]  kasan_slab_free+0x10/0x18
[  616.012068]  kmem_cache_free+0x60/0x1e8
[  616.012071]  __d_free+0x18/0x20
[  616.012081]  rcu_process_callbacks+0x46c/0x940
[  616.012086]  __do_softirq+0x28c/0x6cc
[  616.012087] 
[  616.012100] The buggy address belongs to the object at ffff8002fb33c100
[  616.012100]  which belongs to the cache dentry of size 192
[  616.017462] The buggy address is located 144 bytes inside of
[  616.017462]  192-byte region [ffff8002fb33c100, ffff8002fb33c1c0)
[  616.017465] The buggy address belongs to the page:
[  616.017470] page:ffff7e000beccf00 count:1 mapcount:0 mapping:ffff800358c13400 index:0x0 compound_mapcount: 0
[  616.017477] flags: 0x1ffff00000010200(slab|head)
[  616.017488] raw: 1ffff00000010200 dead000000000100 dead000000000200 ffff800358c13400
[  616.024873] raw: 0000000000000000 0000000080400040 00000001ffffffff 0000000000000000
[  616.024875] page dumped because: kasan: bad access detected
[  616.024876] 
[  616.024877] Memory state around the buggy address:
[  616.024882]  ffff8002fb33c080: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
[  616.024885]  ffff8002fb33c100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
[  616.024887] >ffff8002fb33c180: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
[  616.024889]                          ^
[  616.024891]  ffff8002fb33c200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[  616.024893]  ffff8002fb33c280: 00 00 00 00 00 00 00 00 fc fc fc fc fc fc fc fc
[  616.024894] ==================================================================
#5 lore raw
Quoted 27 lines · Will Deacon
Quoted 26 lines · Will Deacon
Quoted 25 lines · Jan Glauber
Quoted 11 lines · Will Deacon
Quoted 2 lines · Jan Glauber

I'm seeing the following oops reproducible with upstream kernel on arm64 (ThunderX2):

[...]

Quoted 5 lines

It happens after 1-3 hours of running 'stress-ng --dev 128'. This testcase does a scandir of /dev and then calls random stuff like ioctl, lseek, open/close etc. on the entries. I assume no files are deleted under /dev during the testcase.

Quoted 7 lines
Quoted 4 lines

The NULL pointer is the inode pointer of next. The next dentry->d_flags is DCACHE_RCUACCESS when this happens.

Any hints on how to further debug this?

Can you reproduce the issue with vanilla -rc1 and do you have a "known good" kernel?

I can try out -rc1, but IIRC this wasn't bisectible as the bug was present at least back to 4.14. I need to double check that as there were other issues that are resolved now so I may confuse things here. I've defintely seen the same bug with 4.18.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Unfortunately I lost access to the machine as our data center seems to be moving currently so it might take some days until I can try -rc1.

Ok, I've just managed to reproduce this in a KVM guest running v4.20-rc3 on both the host and the guest, so if anybody has any ideas of things to try then I'm happy to give them a shot. In the meantime, I'll try again with a bunch of debug checks enabled.

Hi Will,

good that you can reproduce the issue. I've verified that the issue is indeed reproducible with 4.14.

Quoted 2 lines

Weee, I eventually hit a use-after-free from KASAN. See below.

I ran KASAN (and all the other debug stuff) but didn't trigger anything in the host.

--Jan

#6 lore raw

Hi all,

I've now managed to reproduce this on x86 (log below) but I'm out of my depth with this one. Looping in Greg and Jiri because I fear this is specific to the pty code. Rest of the thread is here:

http://lkml.kernel.org/r/20181109143744.GA12128@hc

Quoted 36 lines · Jan Glauber
Quoted 27 lines · Will Deacon
Quoted 26 lines · Will Deacon
Quoted 25 lines · Jan Glauber
Quoted 11 lines · Will Deacon
Quoted 2 lines · Jan Glauber

I'm seeing the following oops reproducible with upstream kernel on arm64 (ThunderX2):

[...]

Quoted 5 lines

It happens after 1-3 hours of running 'stress-ng --dev 128'. This testcase does a scandir of /dev and then calls random stuff like ioctl, lseek, open/close etc. on the entries. I assume no files are deleted under /dev during the testcase.

Quoted 7 lines
Quoted 4 lines

The NULL pointer is the inode pointer of next. The next dentry->d_flags is DCACHE_RCUACCESS when this happens.

Any hints on how to further debug this?

Can you reproduce the issue with vanilla -rc1 and do you have a "known good" kernel?

I can try out -rc1, but IIRC this wasn't bisectible as the bug was present at least back to 4.14. I need to double check that as there were other issues that are resolved now so I may confuse things here. I've defintely seen the same bug with 4.18.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Unfortunately I lost access to the machine as our data center seems to be moving currently so it might take some days until I can try -rc1.

Ok, I've just managed to reproduce this in a KVM guest running v4.20-rc3 on both the host and the guest, so if anybody has any ideas of things to try then I'm happy to give them a shot. In the meantime, I'll try again with a bunch of debug checks enabled.

Quoted 8 lines

good that you can reproduce the issue. I've verified that the issue is indeed reproducible with 4.14.

Quoted 2 lines

Weee, I eventually hit a use-after-free from KASAN. See below.

I ran KASAN (and all the other debug stuff) but didn't trigger anything in the host.

Doing some more debugging, it looks like the usual failure case is where one CPU clears the inode field in the dentry via:

	devpts_pty_kill()
		-> d_delete()	// dentry->d_lockref.count == 1
			-> dentry_unlink_inode()

whilst another CPU gets a pointer to the dentry via:

	sys_getdents64()
		-> iterate_dir()
			-> dcache_readdir()
				-> next_positive()

and explodes on the subsequent inode dereference when trying to pass the inode number to dir_emit():

	if (!dir_emit(..., d_inode(next)->i_ino, ...))

Indeed, the hack below triggers a warning, indicating that the inode is being cleared concurrently.

I can't work out whether the getdents64() path should hold a refcount to stop d_delete() in its tracks, or whether devpts_pty_kill() shouldn't be calling d_delete() like this at all.

Any help debugging this would be really appreciated.

Will

--->8

fs/libfs.c +1 -0
index 0fb590d79f30..fe3f72c5cf72 100644
@@ -117,6 +117,7 @@ static struct dentry *next_positive(struct dentry *parent,
                if (unlikely(*seq != n))
                        goto retry;
        }
+       WARN_ON(res && !d_inode(res));
        return res;
 }
 

--->8

[  410.560885] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[  410.561458] PGD 0 P4D 0 
[  410.561458] Oops: 0000 [#1] PREEMPT SMP PTI
[  410.561458] CPU: 1 PID: 983 Comm: stress-ng-dev Not tainted 4.20.0-rc3 #1
[  410.561458] RIP: 0010:dcache_readdir+0xd6/0x160
[  410.561458] Code: 7d 48 c7 43 08 02 00 00 00 4d 8d a5 a0 00 00 00 45 31 f6 eb 41 48 8b 45 30 48 8b 4b 08 48 89 df 8b 55 24 48 8b 75 28 4c 8b 13 <44> 0f b7 08 4c 8b 40 40 66 41 c1 e9 0c 41 83 e1 0f e8 24 5e c1 00
[  410.565432] RSP: 0018:ffffb0a2004e7e48 EFLAGS: 00010286
[  410.565432] RAX: 0000000000000000 RBX: ffffb0a2004e7ec0 RCX: 0000000000000002
[  410.565432] RDX: 0000000000000001 RSI: ffff9ccef8396938 RDI: ffffb0a2004e7ec0
[  410.565432] RBP: ffff9ccef8396900 R08: 0000000000000001 R09: 0000000000000000
[  410.565432] R10: ffffffff947d7070 R11: 0000000000000000 R12: ffff9ccefcc23e20
[  410.565432] R13: ffff9ccefcc23d80 R14: 0000000000000000 R15: ffff9ccef7c1c0c0
[  410.565432] FS:  00007f1883901740(0000) GS:ffff9ccefd700000(0000) knlGS:0000000000000000
[  410.565432] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  410.565432] CR2: 0000000000000000 CR3: 000000007b840004 CR4: 0000000000360ea0
[  410.565432] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  410.573446] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  410.573446] Call Trace:
[  410.573446]  iterate_dir+0x90/0x180
[  410.573446]  __x64_sys_getdents+0xa6/0x130
[  410.573446]  ? __ia32_compat_sys_getdents+0x130/0x130
[  410.573446]  ? do_syscall_64+0x43/0xf0
[  410.573446]  ? __ia32_sys_getdents+0x130/0x130
[  410.573446]  do_syscall_64+0x43/0xf0
[  410.573446]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  410.577435] RIP: 0033:0x7f1881d8a0a8
[  410.577435] Code: e9 0d d8 fb ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 41 57 41 56 b8 4e 00 00 00 41 55 41 54 55 53 48 89 f3 48 83 ec 08 0f 05 <48> 3d 00 f0 ff ff 77 58 4c 8d 2c 06 49 89 c4 4c 39 ee 73 36 0f 1f
[  410.577435] RSP: 002b:00007ffd64877580 EFLAGS: 00000202 ORIG_RAX: 000000000000004e
[  410.577435] RAX: ffffffffffffffda RBX: 00005586c709eb30 RCX: 00007f1881d8a0a8
[  410.577435] RDX: 0000000000008000 RSI: 00005586c709eb30 RDI: 0000000000000006
[  410.577435] RBP: 00005586c709eb30 R08: 0000000000000078 R09: 0000000000000000
[  410.577435] R10: 00005586c709c010 R11: 0000000000000202 R12: ffffffffffffff78
[  410.585529] R13: 0000000000000000 R14: ffffffffffffff78 R15: 00005586c709eb00
[  410.585529] Modules linked in:
[  410.585529] CR2: 0000000000000000
[  410.585529] ---[ end trace aece97fc2fce9ace ]---
#7 lore raw

I spent some more time looking at this today...

Quoted 17 lines · Will Deacon

Doing some more debugging, it looks like the usual failure case is where one CPU clears the inode field in the dentry via:

	devpts_pty_kill()
		-> d_delete()	// dentry->d_lockref.count == 1
			-> dentry_unlink_inode()

whilst another CPU gets a pointer to the dentry via:

	sys_getdents64()
		-> iterate_dir()
			-> dcache_readdir()
				-> next_positive()

and explodes on the subsequent inode dereference when trying to pass the inode number to dir_emit():

Quoted 8 lines
	if (!dir_emit(..., d_inode(next)->i_ino, ...))

Indeed, the hack below triggers a warning, indicating that the inode is being cleared concurrently.

I can't work out whether the getdents64() path should hold a refcount to stop d_delete() in its tracks, or whether devpts_pty_kill() shouldn't be calling d_delete() like this at all.

So the issue is that opening /dev/pts/ptmx creates a new pty in /dev/pts, which disappears when you close /dev/pts/ptmx. Consequently, when we tear down the dentry for the magic new file, we have to take the i_node rwsem of the *parent* so that concurrent path walkers don't trip over it whilst its being freed. I wrote a simple concurrent program to getdents(/dev/pts/) in one thread, whilst another opens and closes /dev/pts/ptmx: it crashes the kernel in seconds.

Patch below, but I'd still like somebody else to look at this, please.

Will

--->8

fs/devpts/inode.c +7 -1
index c53814539070..50ddb95ff84c 100644
@@ -619,11 +619,17 @@ void *devpts_get_priv(struct dentry *dentry)
  */
 void devpts_pty_kill(struct dentry *dentry)
 {
-	WARN_ON_ONCE(dentry->d_sb->s_magic != DEVPTS_SUPER_MAGIC);
+	struct super_block *sb = dentry->d_sb;
+	struct dentry *parent = sb->s_root;
 
+	WARN_ON_ONCE(sb->s_magic != DEVPTS_SUPER_MAGIC);
+
+	inode_lock(parent->d_inode);
 	dentry->d_fsdata = NULL;
 	drop_nlink(dentry->d_inode);
 	d_delete(dentry);
+	inode_unlock(parent->d_inode);
+
 	dput(dentry);	/* d_alloc_name() in devpts_pty_new() */
 }
 
#8 lore raw
Quoted 29 lines · Will Deacon

I spent some more time looking at this today...

Quoted 17 lines · Will Deacon

Doing some more debugging, it looks like the usual failure case is where one CPU clears the inode field in the dentry via:

      devpts_pty_kill()
              -> d_delete()   // dentry->d_lockref.count == 1
                      -> dentry_unlink_inode()

whilst another CPU gets a pointer to the dentry via:

      sys_getdents64()
              -> iterate_dir()
                      -> dcache_readdir()
                              -> next_positive()

and explodes on the subsequent inode dereference when trying to pass the inode number to dir_emit():

Quoted 8 lines
      if (!dir_emit(..., d_inode(next)->i_ino, ...))

Indeed, the hack below triggers a warning, indicating that the inode is being cleared concurrently.

I can't work out whether the getdents64() path should hold a refcount to stop d_delete() in its tracks, or whether devpts_pty_kill() shouldn't be calling d_delete() like this at all.

Quoted 7 lines

So the issue is that opening /dev/pts/ptmx creates a new pty in /dev/pts, which disappears when you close /dev/pts/ptmx. Consequently, when we tear down the dentry for the magic new file, we have to take the i_node rwsem of the *parent* so that concurrent path walkers don't trip over it whilst its being freed. I wrote a simple concurrent program to getdents(/dev/pts/) in one thread, whilst another opens and closes /dev/pts/ptmx: it crashes the kernel in seconds.

I also made a testcase and verified that your fix is fine. I also tried replacing open-close on /dev/ptmx with mkdir-rmdir but that does not trigger the error.

Quoted 1 line

Patch below, but I'd still like somebody else to look at this, please.

I wonder why no inode_lock on parent is needed for devpts_pty_new(), but I'm obviously not a VFS expert... So your patch looks good to me and clearly solves the issue.

thanks,
Jan

Quoted 26 lines

Will

--->8

fs/devpts/inode.c +7 -1
index c53814539070..50ddb95ff84c 100644
@@ -619,11 +619,17 @@ void *devpts_get_priv(struct dentry *dentry)
  */
 void devpts_pty_kill(struct dentry *dentry)
 {
-       WARN_ON_ONCE(dentry->d_sb->s_magic != DEVPTS_SUPER_MAGIC);
+       struct super_block *sb = dentry->d_sb;
+       struct dentry *parent = sb->s_root;
 
+       WARN_ON_ONCE(sb->s_magic != DEVPTS_SUPER_MAGIC);
+
+       inode_lock(parent->d_inode);
        dentry->d_fsdata = NULL;
        drop_nlink(dentry->d_inode);
        d_delete(dentry);
+       inode_unlock(parent->d_inode);
+
        dput(dentry);   /* d_alloc_name() in devpts_pty_new() */
 }
#9 lore raw

Adding Eric as he touched this code last :)

Quoted 67 lines · Jan Glauber
Quoted 29 lines · Will Deacon

I spent some more time looking at this today...

Quoted 17 lines · Will Deacon

Doing some more debugging, it looks like the usual failure case is where one CPU clears the inode field in the dentry via:

      devpts_pty_kill()
              -> d_delete()   // dentry->d_lockref.count == 1
                      -> dentry_unlink_inode()

whilst another CPU gets a pointer to the dentry via:

      sys_getdents64()
              -> iterate_dir()
                      -> dcache_readdir()
                              -> next_positive()

and explodes on the subsequent inode dereference when trying to pass the inode number to dir_emit():

Quoted 8 lines
      if (!dir_emit(..., d_inode(next)->i_ino, ...))

Indeed, the hack below triggers a warning, indicating that the inode is being cleared concurrently.

I can't work out whether the getdents64() path should hold a refcount to stop d_delete() in its tracks, or whether devpts_pty_kill() shouldn't be calling d_delete() like this at all.

Quoted 7 lines

So the issue is that opening /dev/pts/ptmx creates a new pty in /dev/pts, which disappears when you close /dev/pts/ptmx. Consequently, when we tear down the dentry for the magic new file, we have to take the i_node rwsem of the *parent* so that concurrent path walkers don't trip over it whilst its being freed. I wrote a simple concurrent program to getdents(/dev/pts/) in one thread, whilst another opens and closes /dev/pts/ptmx: it crashes the kernel in seconds.

I also made a testcase and verified that your fix is fine. I also tried replacing open-close on /dev/ptmx with mkdir-rmdir but that does not trigger the error.

Quoted 1 line

Patch below, but I'd still like somebody else to look at this, please.

I wonder why no inode_lock on parent is needed for devpts_pty_new(), but I'm obviously not a VFS expert... So your patch looks good to me and clearly solves the issue.

thanks,
Jan

Quoted 16 lines

Will

--->8

fs/devpts/inode.c +2 -1
index c53814539070..50ddb95ff84c 100644
@@ -619,11 +619,17 @@ void *devpts_get_priv(struct dentry *dentry)
  */
 void devpts_pty_kill(struct dentry *dentry)
 {
-       WARN_ON_ONCE(dentry->d_sb->s_magic != DEVPTS_SUPER_MAGIC);
+       struct super_block *sb = dentry->d_sb;
+       struct dentry *parent = sb->s_root;
 
Quoted 1 line

fs/devpts/inode.c void *devpts_get_priv(struct dentry *dentry) · line 625

Quoted 1 line
fs/devpts/inode.c +1 -0
+       WARN_ON_ONCE(sb->s_magic != DEVPTS_SUPER_MAGIC);

Side note, I wonder if this is even needed anymore...

Quoted 9 lines

fs/devpts/inode.c void *devpts_get_priv(struct dentry *dentry) · line 634

Quoted 9 lines
fs/devpts/inode.c +4 -0
+
+       inode_lock(parent->d_inode);
        dentry->d_fsdata = NULL;
        drop_nlink(dentry->d_inode);
        d_delete(dentry);
+       inode_unlock(parent->d_inode);
+
        dput(dentry);   /* d_alloc_name() in devpts_pty_new() */
 }

This feels right but getting some feedback from others would be good.

thanks,

greg k-h

#10 lore raw
Quoted 74 lines · "[email protected]" <[email protected]> writes:

Adding Eric as he touched this code last :)

Quoted 67 lines · Jan Glauber
Quoted 29 lines · Will Deacon

I spent some more time looking at this today...

Quoted 17 lines · Will Deacon

Doing some more debugging, it looks like the usual failure case is where one CPU clears the inode field in the dentry via:

      devpts_pty_kill()
              -> d_delete()   // dentry->d_lockref.count == 1
                      -> dentry_unlink_inode()

whilst another CPU gets a pointer to the dentry via:

      sys_getdents64()
              -> iterate_dir()
                      -> dcache_readdir()
                              -> next_positive()

and explodes on the subsequent inode dereference when trying to pass the inode number to dir_emit():

Quoted 8 lines
      if (!dir_emit(..., d_inode(next)->i_ino, ...))

Indeed, the hack below triggers a warning, indicating that the inode is being cleared concurrently.

I can't work out whether the getdents64() path should hold a refcount to stop d_delete() in its tracks, or whether devpts_pty_kill() shouldn't be calling d_delete() like this at all.

Quoted 7 lines

So the issue is that opening /dev/pts/ptmx creates a new pty in /dev/pts, which disappears when you close /dev/pts/ptmx. Consequently, when we tear down the dentry for the magic new file, we have to take the i_node rwsem of the *parent* so that concurrent path walkers don't trip over it whilst its being freed. I wrote a simple concurrent program to getdents(/dev/pts/) in one thread, whilst another opens and closes /dev/pts/ptmx: it crashes the kernel in seconds.

I also made a testcase and verified that your fix is fine. I also tried replacing open-close on /dev/ptmx with mkdir-rmdir but that does not trigger the error.

Quoted 1 line

Patch below, but I'd still like somebody else to look at this, please.

I wonder why no inode_lock on parent is needed for devpts_pty_new(), but I'm obviously not a VFS expert... So your patch looks good to me and clearly solves the issue.

thanks,
Jan

Quoted 16 lines

Will

--->8

fs/devpts/inode.c +2 -1
index c53814539070..50ddb95ff84c 100644
@@ -619,11 +619,17 @@ void *devpts_get_priv(struct dentry *dentry)
  */
 void devpts_pty_kill(struct dentry *dentry)
 {
-       WARN_ON_ONCE(dentry->d_sb->s_magic != DEVPTS_SUPER_MAGIC);
+       struct super_block *sb = dentry->d_sb;
+       struct dentry *parent = sb->s_root;
 
Quoted 1 line

fs/devpts/inode.c void *devpts_get_priv(struct dentry *dentry) · line 625

Quoted 1 line
fs/devpts/inode.c +1 -0
+       WARN_ON_ONCE(sb->s_magic != DEVPTS_SUPER_MAGIC);

Side note, I wonder if this is even needed anymore...

Quoted 11 lines
Quoted 9 lines

fs/devpts/inode.c void *devpts_get_priv(struct dentry *dentry) · line 634

Quoted 9 lines
fs/devpts/inode.c +4 -0
+
+       inode_lock(parent->d_inode);
        dentry->d_fsdata = NULL;
        drop_nlink(dentry->d_inode);
        d_delete(dentry);
+       inode_unlock(parent->d_inode);
+
        dput(dentry);   /* d_alloc_name() in devpts_pty_new() */
 }

This feels right but getting some feedback from others would be good.

This is going to be special at least because we are not coming through the normal unlink path and we are manipulating the dcache.

This looks plausible. If this is whats going on then we have had this bug for a very long time. I will see if I can make some time.

It looks like in the general case everything is serialized by the devpts_mutex. I wonder if just changing the order of operations here would be enough.

AKA: drop_nlink d_delete then dentry->d_fsdata. Ugh d_fsdata is not implicated so that won't help here.

I will look more shortly. I am in the middle in the move so I don't have time to complete the analysis today.

Eric

maintainer
#11 lore raw
Quoted 17 lines · Eric W. Biederman
Quoted 10 lines
Quoted 8 lines

fs/devpts/inode.c void *devpts_get_priv(struct dentry *dentry) · line 634

Quoted 8 lines
fs/devpts/inode.c +3 -0
+       inode_lock(parent->d_inode);
        dentry->d_fsdata = NULL;
        drop_nlink(dentry->d_inode);
        d_delete(dentry);
+       inode_unlock(parent->d_inode);
+
        dput(dentry);   /* d_alloc_name() in devpts_pty_new() */
 }

This feels right but getting some feedback from others would be good.

This is going to be special at least because we are not coming through the normal unlink path and we are manipulating the dcache.

This looks plausible. If this is whats going on then we have had this bug for a very long time. I will see if I can make some time.

Quoted 6 lines

It looks like in the general case everything is serialized by the devpts_mutex. I wonder if just changing the order of operations here would be enough.

AKA: drop_nlink d_delete then dentry->d_fsdata. Ugh d_fsdata is not implicated so that won't help here.

It certainly won't. The thing is, this

                if (!dir_emit(ctx, next->d_name.name, next->d_name.len,
                              d_inode(next)->i_ino, dt_type(d_inode(next))))

in dcache_readdir() obviously can block, so all we can hold over it is blocking locks. Which we do - specifically, ->i_rwsem on our directory.

It's actually worse than missing inode_lock() - consider the effects of mount --bind /mnt/foo /dev/pts/42. What happens when that thing goes away? Right, a lost mount...

I'll resurrect the "kernel-internal rm -rf done right" series and post it; devpts is not the only place suffering such problem (binfmt_misc, etc.)

#12 lore raw
Quoted 26 lines · Al Viro
Quoted 17 lines · Eric W. Biederman
Quoted 10 lines
Quoted 8 lines

fs/devpts/inode.c void *devpts_get_priv(struct dentry *dentry) · line 634

Quoted 8 lines
fs/devpts/inode.c +3 -0
+       inode_lock(parent->d_inode);
        dentry->d_fsdata = NULL;
        drop_nlink(dentry->d_inode);
        d_delete(dentry);
+       inode_unlock(parent->d_inode);
+
        dput(dentry);   /* d_alloc_name() in devpts_pty_new() */
 }

This feels right but getting some feedback from others would be good.

This is going to be special at least because we are not coming through the normal unlink path and we are manipulating the dcache.

This looks plausible. If this is whats going on then we have had this bug for a very long time. I will see if I can make some time.

Quoted 6 lines

It looks like in the general case everything is serialized by the devpts_mutex. I wonder if just changing the order of operations here would be enough.

AKA: drop_nlink d_delete then dentry->d_fsdata. Ugh d_fsdata is not implicated so that won't help here.

It certainly won't. The thing is, this

Quoted 8 lines
                if (!dir_emit(ctx, next->d_name.name, next->d_name.len,
                              d_inode(next)->i_ino, dt_type(d_inode(next))))

in dcache_readdir() obviously can block, so all we can hold over it is blocking locks. Which we do - specifically, ->i_rwsem on our directory.

It's actually worse than missing inode_lock() - consider the effects of mount --bind /mnt/foo /dev/pts/42. What happens when that thing goes away? Right, a lost mount...

Ha, I hadn't even considered that scenario. Urgh!

Quoted 3 lines

I'll resurrect the "kernel-internal rm -rf done right" series and post it; devpts is not the only place suffering such problem (binfmt_misc, etc.)

Thanks. I'm happy to test that it solves this issue if you throw me on cc.

Will

#13 lore raw

Hi Al,

Quoted 32 lines · Will Deacon
Quoted 31 lines · Al Viro
Quoted 17 lines · Eric W. Biederman
Quoted 10 lines
Quoted 8 lines

fs/devpts/inode.c void *devpts_get_priv(struct dentry *dentry) · line 634

Quoted 8 lines
fs/devpts/inode.c +3 -0
+       inode_lock(parent->d_inode);
        dentry->d_fsdata = NULL;
        drop_nlink(dentry->d_inode);
        d_delete(dentry);
+       inode_unlock(parent->d_inode);
+
        dput(dentry);   /* d_alloc_name() in devpts_pty_new() */
 }

This feels right but getting some feedback from others would be good.

This is going to be special at least because we are not coming through the normal unlink path and we are manipulating the dcache.

This looks plausible. If this is whats going on then we have had this bug for a very long time. I will see if I can make some time.

Quoted 6 lines

It looks like in the general case everything is serialized by the devpts_mutex. I wonder if just changing the order of operations here would be enough.

AKA: drop_nlink d_delete then dentry->d_fsdata. Ugh d_fsdata is not implicated so that won't help here.

It certainly won't. The thing is, this

                if (!dir_emit(ctx, next->d_name.name, next->d_name.len,
                              d_inode(next)->i_ino, dt_type(d_inode(next))))

in dcache_readdir() obviously can block, so all we can hold over it is blocking locks. Which we do - specifically, ->i_rwsem on our directory.

Quoted 9 lines
Quoted 3 lines

It's actually worse than missing inode_lock() - consider the effects of mount --bind /mnt/foo /dev/pts/42. What happens when that thing goes away? Right, a lost mount...

Ha, I hadn't even considered that scenario. Urgh!

Quoted 3 lines

I'll resurrect the "kernel-internal rm -rf done right" series and post it; devpts is not the only place suffering such problem (binfmt_misc, etc.)

I've not seen anything merged regarding this issue so I guess this is still open? We see a similar crash (dcache_readdir hitting a NULL inode ptr) but this time not with devpts.

Debugging is ongoing and we're not even sure which filesystem is having the issue. Is my assumption correct that we should only see this when d_delete(dentry) is called?

thanks,
Jan