NFS: Fix directory delegation verifier checks
Subsystems (3)
Anna Schumaker
maintainerPATCH 1/1 NFS: Fix directory delegation verifier checks
From: Anna Schumaker <[email protected]>
Doing this check in nfs_check_verifier() resulted in many, many more lookups on the wire when running Christoph's delegation benchmarking script. After some experimentation, I found that we can treat directory delegations exactly the same as having a delegated verifier when we reach nfs4_lookup_revalidate() for the best performance.
--- fs/nfs/dir.c | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-)
fs/nfs/dir.c +2 -19
@@ -1516,14 +1516,6 @@ static int nfs_check_verifier(struct inode *dir, struct dentry *dentry,if (!nfs_dentry_verify_change (dir, dentry))return 0 ; -/* -* If we have a directory delegation then we don't need to revalidate -* the directory. The delegation will either get recalled or we will -* receive a notification when it changes. -*/ -if (nfs_have_directory_delegation (dir)) -return 0 ; -/* Revalidate nfsi->cache_change_attribute before we declare a match */ if (nfs_mapping_need_revalidate_inode (dir)) {if (rcu_walk)
@@ -2216,13 +2208,6 @@ int nfs_atomic_open(struct inode *dir, struct dentry *dentry, }EXPORT_SYMBOL_GPL (nfs_atomic_open); -static int -nfs_lookup_revalidate_delegated_parent (struct inode *dir,struct dentry *dentry, -struct inode *inode) -{ -return nfs_lookup_revalidate_done (dir, dentry, inode,1 ); -} -static int nfs4_lookup_revalidate (struct inode *dir,const struct qstr *name,struct dentry *dentry,unsigned int flags)
@@ -2247,12 +2232,10 @@ nfs4_lookup_revalidate(struct inode *dir, const struct qstr *name,if (inode ==NULL ) goto full_reval; -if (nfs_verifier_is_delegated (dentry)) +if (nfs_verifier_is_delegated (dentry) || +nfs_have_directory_delegation (inode))return nfs_lookup_revalidate_delegated (dir, dentry, inode); -if (nfs_have_directory_delegation (dir)) -return nfs_lookup_revalidate_delegated_parent (dir, dentry, inode); -/* NFS only supports OPEN on regular files */ if (!S_ISREG (inode->i_mode)) goto full_reval;
Signature · 1 line
2.52.0
Quoted 11 lines · Anna Schumaker
From: Anna Schumaker <[email protected]>
Doing this check in
nfs_check_verifier()resulted in many, many more lookups on the wire when running Christoph's delegation benchmarking script. After some experimentation, I found that we can treat directory delegations exactly the same as having a delegated verifier when we reachnfs4_lookup_revalidate()for the best performance.Reported-by Christoph Hellwig <[email protected]>Fixes 156b09482933 ("NFS: Request a directory delegation on ACCESS, CREATE, and UNLINK")Signed-off-by Anna Schumaker <[email protected]>
I wish I could actually review this, but I don't actually understand the lookup revalidation logic enough for that. But it does fix the problem I saw, so at least:
Quoted 3 lines · Christoph Hellwig
I wish I could actually review this, but I don't actually understand the lookup revalidation logic enough for that. But it does fix the problem I saw, so at least:
Actually - I have to take this back. This patch makes generic/786 when run on NFS v4.2 go from just failing with:
"Server reported failure (1)"
to actually crashing the kernel:
[ 30.195133] run fstests generic/786 at 2025-12-23 01:04:02 [ 36.956689] Oops: general protection fault, probably for non-canonical address 0xcccccccccccccd0c: 0000 [#1] SMP NOPTI [ 36.958741] CPU: 0 UID: 0 PID: 3837 Comm: locktest Tainted: G N 6.19.0-rc2+ #4523 PREEMPT(full) [ 36.960855] Tainted: [N]=TEST [ 36.961443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 36.963167] RIP: 0010:nfs_end_delegation_return+0xda/0x390 [ 36.964321] Code: 49 89 ce e8 18 c1 bf ff 48 8b 85 60 ff ff ff 4c 8d 68 80 49 39 c6 74 50 4c 8b 7c 24 08 48 8b 1c 24 4d 8b 65 60 4d 85 e4 74 2e <49> 8b 44 24 40 a8 02 74 25 49 8b 44 24 40 f6 c4 02 75 1b 41 8b 47 [ 36.967856] RSP: 0018:ffffc900018e7cc0 EFLAGS: 00010286 [ 36.968862] RAX: ffff88810efc5e20 RBX: ffff888104fa6970 RCX: ffff888112570170 [ 36.970200] RDX: ffff888105780040 RSI: ffff8881189120e8 RDI: ffff888112570210 [ 36.971676] RBP: ffff888112570210 R08: 0000000000000000 R09: 0000000000000000 [ 36.973041] R10: ffffc900018e7d78 R11: fefefefefefefeff R12: cccccccccccccccc [ 36.974313] R13: ffff88810efc5da0 R14: ffff888112570170 R15: ffff888104fa6940 [ 36.975635] FS: 00007f0df5dc4740(0000) GS:ffff8882b3544000(0000) knlGS:0000000000000000 [ 36.977106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 36.978187] CR2: 00007f0df5e26450 CR3: 0000000118e1b001 CR4: 0000000000772ef0 [ 36.979629] PKRU: 55555554 [ 36.980205] Call Trace: [ 36.980706] <TASK> [ 36.981131] ? nfs_clear_verifier_delegated+0x50/0x70 [ 36.982109] nfs4_proc_setattr+0xff/0x110 [ 36.982894] nfs_setattr+0x1c8/0x410 [ 36.983634] notify_change+0x373/0x510 [ 36.984376] ? chmod_common+0xad/0x160 [ 36.985055] chmod_common+0xad/0x160 [ 36.985702] __x64_sys_chmod+0x56/0xb0 [ 36.986388] do_syscall_64+0x50/0xf80 [ 36.987097] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 36.988171] RIP: 0033:0x7f0df5ec6707 [ 36.988880] Code: 73 01 c3 48 8b 0d f1 76 0e 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 5a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 c1 76 0e 00 f7 d8 64 89 02 b8 [ 36.992424] RSP: 002b:00007fff3cd5bad8 EFLAGS: 00000206 ORIG_RAX: 000000000000005a [ 36.993747] RAX: ffffffffffffffda RBX: 0000558071f19b90 RCX: 00007f0df5ec6707 [ 36.995045] RDX: 0000000000000008 RSI: 00000000000001fd RDI: 00007fff3cd5cbc0 [ 36.996499] RBP: 0000000000000000 R08: 00000000000001fd R09: 0000000000000000 [ 36.997855] R10: 0000000000000003 R11: 0000000000000206 R12: 0000000000000035 [ 36.999218] R13: 0000000000000007 R14: 0000000000000000 R15: 0000558071f1ebc0 [ 37.000618] </TASK> [ 37.001039] Modules linked in: kvm_intel kvm irqbypass [ 37.002029] ---[ end trace 0000000000000000 ]--- [ 37.003342] RIP: 0010:nfs_end_delegation_return+0xda/0x390 [ 37.004372] Code: 49 89 ce e8 18 c1 bf ff 48 8b 85 60 ff ff ff 4c 8d 68 80 49 39 c6 74 50 4c 8b 7c 24 08 48 8b 1c 24 4d 8b 65 60 4d 85 e4 74 2e <49> 8b 44 24 40 a8 02 74 25 49 8b 44 24 40 f6 c4 02 75 1b 41 8b 47 [ 37.007875] RSP: 0018:ffffc900018e7cc0 EFLAGS: 00010286 [ 37.008902] RAX: ffff88810efc5e20 RBX: ffff888104fa6970 RCX: ffff888112570170 [ 37.010307] RDX: ffff888105780040 RSI: ffff8881189120e8 RDI: ffff888112570210 [ 37.011702] RBP: ffff888112570210 R08: 0000000000000000 R09: 0000000000000000 [ 37.012975] R10: ffffc900018e7d78 R11: fefefefefefefeff R12: cccccccccccccccc [ 37.014271] R13: ffff88810efc5da0 R14: ffff888112570170 R15: ffff888104fa6940 [ 37.015600] FS: 00007f0df5dc4740(0000) GS:ffff8882b3544000(0000) knlGS:0000000000000000 [ 37.017070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 37.018190] CR2: 00007f0df5e26450 CR3: 0000000118e1b002 CR4: 0000000000772ef0 [ 37.019525] PKRU: 55555554 [ 37.020028] BUG: sleeping function called from invalid context at kernel/locking/mutex.c:287 [ 37.021516] in_atomic(): 0, irqs_disabled(): 0, non_block: 0, pid: 3837, name: locktest [ 37.022851] preempt_count: 0, expected: 0 [ 37.023565] RCU nest depth: 1, expected: 0 [ 37.024235] CPU: 0 UID: 0 PID: 3837 Comm: locktest Tainted: G D N 6.19.0-rc2+ #4523 PREEMPT(full) [ 37.024240] Tainted: [D]=DIE, [N]=TEST [ 37.024241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 37.024242] Call Trace: [ 37.024243] <TASK> [ 37.024244] dump_stack_lvl+0x4b/0x70 [ 37.024250] __might_resched.cold+0xd3/0x10e [ 37.024255] mutex_lock+0x19/0x80 [ 37.024259] sched_mm_cid_exit+0x51/0x1e0 [ 37.024263] do_exit+0xb0/0xa40 [ 37.024265] ? __x64_sys_chmod+0x56/0xb0 [ 37.024269] make_task_dead+0x87/0x90 [ 37.024271] rewind_stack_and_make_dead+0x16/0x20 [ 37.024275] RIP: 0033:0x7f0df5ec6707 [ 37.024277] Code: 73 01 c3 48 8b 0d f1 76 0e 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 5a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 c1 76 0e 00 f7 d8 64 89 02 b8 [ 37.024278] RSP: 002b:00007fff3cd5bad8 EFLAGS: 00000206 ORIG_RAX: 000000000000005a [ 37.024280] RAX: ffffffffffffffda RBX: 0000558071f19b90 RCX: 00007f0df5ec6707 [ 37.024281] RDX: 0000000000000008 RSI: 00000000000001fd RDI: 00007fff3cd5cbc0 [ 37.024282] RBP: 0000000000000000 R08: 00000000000001fd R09: 0000000000000000 [ 37.024283] R10: 0000000000000003 R11: 0000000000000206 R12: 0000000000000035 [ 37.024284] R13: 0000000000000007 R14: 0000000000000000 R15: 0000558071f1ebc0 [ 37.024286] </TASK> [ 37.024761] ------------[ cut here ]------------ [ 37.041739] Voluntary context switch within RCU read-side critical section! [ 37.041740] WARNING: kernel/rcu/tree_plugin.h:332 at rcu_note_context_switch+0x39b/0x5e0, CPU#0: locktest/3837 [ 37.043557] Modules linked in: kvm_intel kvm irqbypass [ 37.044135] CPU: 0 UID: 0 PID: 3837 Comm: locktest Tainted: G D W N 6.19.0-rc2+ #4523 PREEMPT(full) [ 37.045192] Tainted: [D]=DIE, [W]=WARN, [N]=TEST [ 37.045680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 37.046655] RIP: 0010:rcu_note_context_switch+0x39b/0x5e0 [ 37.047211] Code: ef e8 f9 a7 34 01 c6 45 11 00 48 8b 55 28 4c 89 ef 48 89 c6 b9 01 00 00 00 e8 91 97 ff ff e9 d6 fc ff ff 48 8d 3d 25 4f 86 02 <67> 48 0f b9 3a e9 a6 fc ff ff 4c 8b 6d 20 4c 89 ef e8 bf a7 34 01 [ 37.049166] RSP: 0018:ffffc900018e7c18 EFLAGS: 00010002 [ 37.049714] RAX: 0000000000000001 RBX: ffff888105780040 RCX: 0000000000000000 [ 37.050451] RDX: 0000000000000001 RSI: ffffffff8304f78e RDI: ffffffff83c34c20 [ 37.051193] RBP: ffff888237c2a300 R08: ffff88810c4208f7 R09: 0000000000000002 [ 37.051957] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000000 [ 37.052702] R13: ffffffff846e55c0 R14: ffff888105780040 R15: ffffffff82608cb0 [ 37.053437] FS: 0000000000000000(0000) GS:ffff8882b3544000(0000) knlGS:0000000000000000 [ 37.054272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 37.054876] CR2: 00007f0df5e26450 CR3: 000000000325a003 CR4: 0000000000772ef0 [ 37.055627] PKRU: 55555554 [ 37.055892] Call Trace: [ 37.056138] <TASK> [ 37.056349] ? _raw_spin_unlock_irqrestore+0x1d/0x40 [ 37.056819] ? __pfx_rpc_wait_bit_killable+0x10/0x10 [ 37.057291] __schedule+0xa1/0xe30 [ 37.057622] ? preempt_count_add+0x73/0xb0 [ 37.058013] ? __pfx_rpc_wait_bit_killable+0x10/0x10 [ 37.058494] schedule+0x29/0xe0 [ 37.058839] rpc_wait_bit_killable+0xc/0x60 [ 37.059285] __wait_on_bit+0x2c/0x90 [ 37.059740] out_of_line_wait_on_bit+0x8e/0xb0 [ 37.060210] ? __pfx_wake_bit_function+0x10/0x10 [ 37.060700] nfs4_do_close+0x289/0x310 [ 37.061101] __put_nfs_open_context+0xc9/0x140 [ 37.061572] nfs_file_release+0x38/0x50 [ 37.061980] ? security_file_release+0x17/0x30 [ 37.062442] __fput+0xf2/0x2b0 [ 37.062773] task_work_run+0x57/0xa0 [ 37.063153] do_exit+0x273/0xa40 [ 37.063524] ? __x64_sys_chmod+0x56/0xb0 [ 37.063937] make_task_dead+0x87/0x90 [ 37.064331] rewind_stack_and_make_dead+0x16/0x20 [ 37.064819] RIP: 0033:0x7f0df5ec6707 [ 37.065200] Code: Unable to access opcode bytes at 0x7f0df5ec66dd. [ 37.065828] RSP: 002b:00007fff3cd5bad8 EFLAGS: 00000206 ORIG_RAX: 000000000000005a [ 37.066601] RAX: ffffffffffffffda RBX: 0000558071f19b90 RCX: 00007f0df5ec6707 [ 37.067328] RDX: 0000000000000008 RSI: 00000000000001fd RDI: 00007fff3cd5cbc0 [ 37.068095] RBP: 0000000000000000 R08: 00000000000001fd R09: 0000000000000000 [ 37.068848] R10: 0000000000000003 R11: 0000000000000206 R12: 0000000000000035 [ 37.069585] R13: 0000000000000007 R14: 0000000000000000 R15: 0000558071f1ebc0 [ 37.070319] </TASK> [ 37.070560] ---[ end trace 0000000000000000 ]---
Hi Christoph,
Quoted 198 lines · Christoph Hellwig
Quoted 4 lines · Christoph Hellwig
I wish I could actually review this, but I don't actually understand
the lookup revalidation logic enough for that. But it does fix the problem I saw, so at least:Actually - I have to take this back. This patch makes generic/786 when run on NFS v4.2 go from just failing with:
"Server reported failure (1)"
to actually crashing the kernel:
[ 30.195133] run fstests generic/786 at 2025-12-23 01:04:02 [ 36.956689] Oops: general protection fault, probably for non- canonical address
0xcccccccccccccd0c: 0000 [#1] SMP NOPTI [ 36.958741] CPU: 0 UID: 0 PID: 3837 Comm: locktest Tainted:
G N 6.19.0-rc2+ #4523 PREEMPT(full)
[ 36.960855] Tainted: [N]=TEST
[ 36.961443] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 36.963167] RIP: 0010:nfs_end_delegation_return+0xda/0x390 [ 36.964321] Code: 49 89 ce e8 18 c1 bf ff 48 8b 85 60 ff ff ff 4c 8d 68 80 49 39 c6 74 50 4c 8b 7c 24 08 48 8b 1c 24 4d 8b 65 60 4d 85 e4 74 2e <49> 8b 44 24 40 a8 02 74 25 49 8b 44 24 40 f6 c4 02 75 1b 41 8b 47
[ 36.967856] RSP: 0018:ffffc900018e7cc0 EFLAGS: 00010286 [ 36.968862] RAX:ffff88810efc5e20RBX:ffff888104fa6970RCX:ffff888112570170
[ 36.970200] RDX:ffff888105780040RSI:ffff8881189120e8RDI:ffff888112570210
[ 36.971676] RBP:ffff888112570210R08: 0000000000000000 R09:
0000000000000000
[ 36.973041] R10:ffffc900018e7d78R11: fefefefefefefeff R12:
cccccccccccccccc
[ 36.974313] R13:ffff88810efc5da0R14:ffff888112570170R15:ffff888104fa6940
[ 36.975635] FS: 00007f0df5dc4740(0000) GS:ffff8882b3544000(0000) knlGS:0000000000000000
[ 36.977106] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 36.978187] CR2:00007f0df5e26450CR3:0000000118e1b001CR4:0000000000772ef0
[ 36.979629] PKRU: 55555554
[ 36.980205] Call Trace:
[ 36.980706] <TASK>
[ 36.981131] ?nfs_clear_verifier_delegated+0x50/0x70
[ 36.982109]nfs4_proc_setattr+0xff/0x110
[ 36.982894]nfs_setattr+0x1c8/0x410
[ 36.983634]notify_change+0x373/0x510
[ 36.984376] ?chmod_common+0xad/0x160
[ 36.985055]chmod_common+0xad/0x160
[ 36.985702]__x64_sys_chmod+0x56/0xb0
[ 36.986388]do_syscall_64+0x50/0xf80
[ 36.987097]entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 36.988171] RIP: 0033:0x7f0df5ec6707
[ 36.988880] Code: 73 01 c3 48 8b 0d f1 76 0e 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 5a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 c1 76 0e 00 f7 d8 64 89 02 b8
[ 36.992424] RSP: 002b:00007fff3cd5bad8 EFLAGS: 00000206ORIG_RAX:000000000000005a
[ 36.993747] RAX: ffffffffffffffda RBX:0000558071f19b90RCX:00007f0df5ec6707
[ 36.995045] RDX: 0000000000000008 RSI:00000000000001fdRDI:00007fff3cd5cbc0
[ 36.996499] RBP: 0000000000000000 R08:00000000000001fdR09:
0000000000000000
[ 36.997855] R10: 0000000000000003 R11: 0000000000000206 R12:
0000000000000035
[ 36.999218] R13: 0000000000000007 R14: 0000000000000000 R15:0000558071f1ebc0
[ 37.000618] </TASK>
[ 37.001039] Modules linked in:kvm_intelkvm irqbypass [ 37.002029] ---[ end trace 0000000000000000 ]---
[ 37.003342] RIP: 0010:nfs_end_delegation_return+0xda/0x390 [ 37.004372] Code: 49 89 ce e8 18 c1 bf ff 48 8b 85 60 ff ff ff 4c 8d 68 80 49 39 c6 74 50 4c 8b 7c 24 08 48 8b 1c 24 4d 8b 65 60 4d 85 e4 74 2e <49> 8b 44 24 40 a8 02 74 25 49 8b 44 24 40 f6 c4 02 75 1b 41 8b 47
[ 37.007875] RSP: 0018:ffffc900018e7cc0 EFLAGS: 00010286 [ 37.008902] RAX:ffff88810efc5e20RBX:ffff888104fa6970RCX:ffff888112570170
[ 37.010307] RDX:ffff888105780040RSI:ffff8881189120e8RDI:ffff888112570210
[ 37.011702] RBP:ffff888112570210R08: 0000000000000000 R09:
0000000000000000
[ 37.012975] R10:ffffc900018e7d78R11: fefefefefefefeff R12:
cccccccccccccccc
[ 37.014271] R13:ffff88810efc5da0R14:ffff888112570170R15:ffff888104fa6940
[ 37.015600] FS: 00007f0df5dc4740(0000) GS:ffff8882b3544000(0000) knlGS:0000000000000000
[ 37.017070] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 37.018190] CR2:00007f0df5e26450CR3:0000000118e1b002CR4:0000000000772ef0
[ 37.019525] PKRU: 55555554
[ 37.020028] BUG: sleeping function called from invalid context atkernel/locking/mutex.c:287
[ 37.021516]in_atomic(): 0,irqs_disabled(): 0,non_block: 0, pid:
3837, name: locktest
[ 37.022851]preempt_count: 0, expected: 0
[ 37.023565] RCU nest depth: 1, expected: 0
[ 37.024235] CPU: 0 UID: 0 PID: 3837 Comm: locktest Tainted: G D N 6.19.0-rc2+ #4523 PREEMPT(full)
[ 37.024240] Tainted: [D]=DIE, [N]=TEST
[ 37.024241] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 37.024242] Call Trace:
[ 37.024243] <TASK>
[ 37.024244]dump_stack_lvl+0x4b/0x70
[ 37.024250]__might_resched.cold+0xd3/0x10e
[ 37.024255]mutex_lock+0x19/0x80
[ 37.024259]sched_mm_cid_exit+0x51/0x1e0
[ 37.024263]do_exit+0xb0/0xa40
[ 37.024265] ?__x64_sys_chmod+0x56/0xb0
[ 37.024269]make_task_dead+0x87/0x90
[ 37.024271]rewind_stack_and_make_dead+0x16/0x20
[ 37.024275] RIP: 0033:0x7f0df5ec6707
[ 37.024277] Code: 73 01 c3 48 8b 0d f1 76 0e 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 5a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 01 c3 48 8b 15 c1 76 0e 00 f7 d8 64 89 02 b8
[ 37.024278] RSP: 002b:00007fff3cd5bad8 EFLAGS: 00000206ORIG_RAX:000000000000005a
[ 37.024280] RAX: ffffffffffffffda RBX:0000558071f19b90RCX:00007f0df5ec6707
[ 37.024281] RDX: 0000000000000008 RSI:00000000000001fdRDI:00007fff3cd5cbc0
[ 37.024282] RBP: 0000000000000000 R08:00000000000001fdR09:
0000000000000000
[ 37.024283] R10: 0000000000000003 R11: 0000000000000206 R12:
0000000000000035
[ 37.024284] R13: 0000000000000007 R14: 0000000000000000 R15:0000558071f1ebc0
[ 37.024286] </TASK>
[ 37.024761] ------------[ cut here ]------------
[ 37.041739] Voluntary context switch within RCU read-side critical section!
[ 37.041740] WARNING:kernel/rcu/tree_plugin.h:332atrcu_note_context_switch+0x39b/0x5e0, CPU#0: locktest/3837 [ 37.043557] Modules linked in:kvm_intelkvm irqbypass [ 37.044135] CPU: 0 UID: 0 PID: 3837 Comm: locktest Tainted: G D W N 6.19.0-rc2+ #4523 PREEMPT(full)
[ 37.045192] Tainted: [D]=DIE, [W]=WARN, [N]=TEST
[ 37.045680] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 37.046655] RIP: 0010:rcu_note_context_switch+0x39b/0x5e0 [ 37.047211] Code: ef e8 f9 a7 34 01 c6 45 11 00 48 8b 55 28 4c 89 ef 48 89 c6 b9 01 00 00 00 e8 91 97 ff ff e9 d6 fc ff ff 48 8d 3d 25 4f 86 02 <67> 48 0f b9 3a e9 a6 fc ff ff 4c 8b 6d 20 4c 89 ef e8 bf a7 34 01
[ 37.049166] RSP: 0018:ffffc900018e7c18 EFLAGS: 00010002 [ 37.049714] RAX: 0000000000000001 RBX:ffff888105780040RCX:
0000000000000000
[ 37.050451] RDX: 0000000000000001 RSI:ffffffff8304f78eRDI:ffffffff83c34c20
[ 37.051193] RBP:ffff888237c2a300R08:ffff88810c4208f7R09:
0000000000000002
[ 37.051957] R10: 0000000000000001 R11: 0000000000000001 R12:
0000000000000000
[ 37.052702] R13:ffffffff846e55c0R14:ffff888105780040R15:ffffffff82608cb0
[ 37.053437] FS: 0000000000000000(0000) GS:ffff8882b3544000(0000) knlGS:0000000000000000
[ 37.054272] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 37.054876] CR2:00007f0df5e26450CR3:000000000325a003CR4:0000000000772ef0
[ 37.055627] PKRU: 55555554
[ 37.055892] Call Trace:
[ 37.056138] <TASK>
[ 37.056349] ?_raw_spin_unlock_irqrestore+0x1d/0x40
[ 37.056819] ?__pfx_rpc_wait_bit_killable+0x10/0x10
[ 37.057291]__schedule+0xa1/0xe30
[ 37.057622] ?preempt_count_add+0x73/0xb0
[ 37.058013] ?__pfx_rpc_wait_bit_killable+0x10/0x10
[ 37.058494]schedule+0x29/0xe0
[ 37.058839]rpc_wait_bit_killable+0xc/0x60
[ 37.059285]__wait_on_bit+0x2c/0x90
[ 37.059740]out_of_line_wait_on_bit+0x8e/0xb0
[ 37.060210] ?__pfx_wake_bit_function+0x10/0x10
[ 37.060700]nfs4_do_close+0x289/0x310
[ 37.061101]__put_nfs_open_context+0xc9/0x140
[ 37.061572]nfs_file_release+0x38/0x50
[ 37.061980] ?security_file_release+0x17/0x30
[ 37.062442]__fput+0xf2/0x2b0
[ 37.062773]task_work_run+0x57/0xa0
[ 37.063153]do_exit+0x273/0xa40
[ 37.063524] ?__x64_sys_chmod+0x56/0xb0
[ 37.063937]make_task_dead+0x87/0x90
[ 37.064331]rewind_stack_and_make_dead+0x16/0x20
[ 37.064819] RIP: 0033:0x7f0df5ec6707
[ 37.065200] Code: Unable to access opcode bytes at0x7f0df5ec66dd. [ 37.065828] RSP: 002b:00007fff3cd5bad8 EFLAGS: 00000206ORIG_RAX:000000000000005a
[ 37.066601] RAX: ffffffffffffffda RBX:0000558071f19b90RCX:00007f0df5ec6707
[ 37.067328] RDX: 0000000000000008 RSI:00000000000001fdRDI:00007fff3cd5cbc0
Quoted 9 lines
[ 37.068095] RBP: 0000000000000000 R08:
00000000000001fdR09:
0000000000000000
[ 37.068848] R10: 0000000000000003 R11: 0000000000000206 R12:
0000000000000035
[ 37.069585] R13: 0000000000000007 R14: 0000000000000000 R15:0000558071f1ebc0
[ 37.070319] </TASK>
[ 37.070560] ---[ end trace 0000000000000000 ]---
Does applying the following on top of Anna's patch fix the Oops?
Cheers
Trond
8<--------------------------------------------------------------
From 3485794e16f68659b26918ccb1c57b6210bd0048 Mon Sep 17 00:00:00 2001
Message-ID: <3485794e16f68659b26918ccb1c57b6210bd0048.1767217813.git.trond.myklebust@hammerspace.com>
From: Trond Myklebust <[email protected]>
Date: Wed, 31 Dec 2025 16:41:15 -0500
Subject: [PATCH] NFSv4: Fix nfs_clear_verifier_delegated() for delegated
directories
If the client returns a directory delegation, then look up all the child dentries, and clear their 'verifier delegated' bit, unless subject to a file delegation.
Similarly, if a file delegation is being returned, check if there is a directory delegation before clearing a 'verifier delegated' bit.
--- fs/nfs/dir.c | 59 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 51 insertions(+), 8 deletions(-)
fs/nfs/dir.c +51 -8
@@ -1440,7 +1440,8 @@ static void nfs_set_verifier_locked(struct dentry *dentry, unsigned long verf)if (!dir || !nfs_verify_change_attribute (dir, verf))return ; -if (inode &&NFS_PROTO (inode)->have_delegation (inode,FMODE_READ ,0 )) +if (NFS_PROTO (dir)->have_delegation (dir,FMODE_READ ,0 ) || + (inode &&NFS_PROTO (inode)->have_delegation (inode,FMODE_READ ,0 )))nfs_set_verifier_delegated (&verf); dentry->d_time = verf; }
@@ -1465,6 +1466,49 @@ void nfs_set_verifier(struct dentry *dentry, unsigned long verf)EXPORT_SYMBOL_GPL (nfs_set_verifier);#if IS_ENABLED (CONFIG_NFS_V4 ) +static void nfs_clear_verifier_file (struct inode *inode) +{ +struct dentry *alias; +struct inode *dir; + +hlist_for_each_entry (alias, &inode->i_dentry, d_u.d_alias) { +spin_lock (&alias->d_lock); + dir =d_inode_rcu (alias->d_parent); +if (!dir || + !NFS_PROTO (dir)->have_delegation (dir,FMODE_READ ,0 )) +nfs_unset_verifier_delegated (&alias->d_time); +spin_unlock (&alias->d_lock); + } +} + +static void nfs_clear_verifier_directory (struct inode *dir) +{ +struct dentry *this_parent; +struct dentry *dentry; +struct inode *inode; + +if (hlist_empty (&dir->i_dentry)) +return ; + this_parent = + hlist_entry(dir->i_dentry.first, struct dentry, d_u.d_alias); + +spin_lock (&this_parent->d_lock); +nfs_unset_verifier_delegated (&this_parent->d_time); + dentry =d_first_child (this_parent); +hlist_for_each_entry_from (dentry, d_sib) { +if (unlikely (dentry->d_flags &DCACHE_DENTRY_CURSOR )) +continue ; + inode =d_inode_rcu (dentry); +if (inode && +NFS_PROTO (inode)->have_delegation (inode,FMODE_READ ,0 )) +continue ; +spin_lock_nested (&dentry->d_lock,DENTRY_D_LOCK_NESTED ); +nfs_unset_verifier_delegated (&dentry->d_time); +spin_unlock (&dentry->d_lock); + } +spin_unlock (&this_parent->d_lock); +} +/** * nfs_clear_verifier_delegated - clear the dir verifier delegation tag * @inode: pointer to inode
@@ -1477,17 +1521,16 @@ EXPORT_SYMBOL_GPL(nfs_set_verifier);*/ void nfs_clear_verifier_delegated (struct inode *inode) { -struct dentry *alias; -if (!inode)return ; +rcu_read_lock ();spin_lock (&inode->i_lock); -hlist_for_each_entry (alias, &inode->i_dentry, d_u.d_alias) { -spin_lock (&alias->d_lock); -nfs_unset_verifier_delegated (&alias->d_time); -spin_unlock (&alias->d_lock); - } +if (S_ISREG (inode->i_mode)) +nfs_clear_verifier_file (inode); +else if (S_ISDIR (inode->i_mode)) +nfs_clear_verifier_directory (inode);spin_unlock (&inode->i_lock); +rcu_read_unlock (); }EXPORT_SYMBOL_GPL (nfs_clear_verifier_delegated);#endif /* IS_ENABLED(CONFIG_NFS_V4) */
Signature · 1 line
2.52.0
Quoted 1 line · Trond Myklebust
Does applying the following on top of Anna's patch fix the Oops?
It does. But now generic/633 crashes reliably:
generic/633 2s ... [ 58.670535] run fstests generic/633 at 2026-01-02 02:00:17
[ 58.865568] process 'vfstest' launched '/dev/fd/4/file1' with NULL argv: empty string added [ 58.897522] Oops: general protection fault, probably for non-canonical address 0xcccccccccccccd0c: 0000 [#1] SMP NOPTI [ 58.898234] CPU: 0 UID: 0 PID: 3852 Comm: vfstest Tainted: G N 6.19.0-rc2+ #4535 PREEMPT(full) [ 58.898829] Tainted: [N]=TEST [ 58.899013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 58.899594] RIP: 0010:nfs_end_delegation_return+0xda/0x390 [ 58.899922] Code: 49 89 ce e8 58 be bf ff 48 8b 85 60 ff ff ff 4c 8d 68 80 49 39 c6 74 50 4c 8b 7c 24 08 48 8b 1c 24 4d 8b 65 60 4d 85 e4 74 2e <49> 8b 44 24 40 a8 02 74 25 49 8b 44 24 40 f6 c4 02 75 1b 41 8b 47 [ 58.901063] RSP: 0018:ffffc90001947c90 EFLAGS: 00010286 [ 58.901419] RAX: ffff88811a3122c0 RBX: ffff888105268970 RCX: ffff888111a9a210 [ 58.901827] RDX: ffff8881039320c0 RSI: ffff888105268940 RDI: ffff888111a9a2b0 [ 58.902236] RBP: ffff888111a9a2b0 R08: 0000000000000000 R09: 0000000000000000 [ 58.902696] R10: ffffc90001947d48 R11: fefefefefefefeff R12: cccccccccccccccc [ 58.903112] R13: ffff88811a312240 R14: ffff888111a9a210 R15: ffff888105268940 [ 58.903594] FS: 00007f04f92c6740(0000) GS:ffff8882b353d000(0000) knlGS:0000000000000000 [ 58.904125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 58.904483] CR2: 000055c55fe522d8 CR3: 000000011a3f1004 CR4: 0000000000772ef0 [ 58.904927] PKRU: 55555554 [ 58.905132] Call Trace: [ 58.905294] <TASK> [ 58.905434] ? _raw_spin_unlock+0x13/0x30 [ 58.905689] nfs4_proc_setattr+0xff/0x110 [ 58.905947] nfs_setattr+0x1c8/0x410 [ 58.906175] notify_change+0x373/0x510 [ 58.906415] ? init_object+0x5a/0xc0 [ 58.906643] ? chown_common+0x1ec/0x220 [ 58.906885] chown_common+0x1ec/0x220 [ 58.907126] do_fchownat+0xc3/0xf0 [ 58.907358] __x64_sys_fchownat+0x1a/0x30 [ 58.907611] do_syscall_64+0x50/0xf80 [ 58.907848] entry_SYSCALL_64_after_hwframe+0x76/0x7e [ 58.908171] RIP: 0033:0x7f04f93c8e4a [ 58.908390] Code: 48 8b 0d b1 6f 0e 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 04 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 7e 6f 0e 00 f7 d8 64 89 01 48 [ 58.909467] RSP: 002b:00007ffe4cd3ec98 EFLAGS: 00000246 ORIG_RAX: 0000000000000104 [ 58.909893] RAX: ffffffffffffffda RBX: 00007ffe4cd3ecb0 RCX: 00007f04f93c8e4a [ 58.910336] RDX: 0000000000002710 RSI: 000055c54836500e RDI: 0000000000000003 [ 58.910872] RBP: 000055c55fd522a0 R08: 0000000000000100 R09: 0000000000000001 [ 58.911350] R10: 0000000000002710 R11: 0000000000000246 R12: 0000000000000006 [ 58.911797] R13: 0000000000002710 R14: 0000000000002710 R15: 000055c55fd52313 [ 58.912234] </TASK> [ 58.912373] Modules linked in: kvm_intel kvm irqbypass [ 58.912717] ---[ end trace 0000000000000000 ]--- [ 58.913829] RIP: 0010:nfs_end_delegation_return+0xda/0x390 [ 58.914177] Code: 49 89 ce e8 58 be bf ff 48 8b 85 60 ff ff ff 4c 8d 68 80 49 39 c6 74 50 4c 8b 7c 24 08 48 8b 1c 24 4d 8b 65 60 4d 85 e4 74 2e <49> 8b 44 24 40 a8 02 74 25 49 8b 44 24 40 f6 c4 02 75 1b 41 8b 47 [ 58.915541] RSP: 0018:ffffc90001947c90 EFLAGS: 00010286 [ 58.915871] RAX: ffff88811a3122c0 RBX: ffff888105268970 RCX: ffff888111a9a210 [ 58.916317] RDX: ffff8881039320c0 RSI: ffff888105268940 RDI: ffff888111a9a2b0 [ 58.916766] RBP: ffff888111a9a2b0 R08: 0000000000000000 R09: 0000000000000000 [ 58.917299] R10: ffffc90001947d48 R11: fefefefefefefeff R12: cccccccccccccccc [ 58.917709] R13: ffff88811a312240 R14: ffff888111a9a210 R15: ffff888105268940 [ 58.918109] FS: 00007f04f92c6740(0000) GS:ffff8882b353d000(0000) knlGS:0000000000000000 [ 58.918560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 58.918944] CR2: 00007fbf39569000 CR3: 000000011a3f1004 CR4: 0000000000772ef0 [ 58.919622] PKRU: 55555554
Trond Myklebust
maintainerQuoted 85 lines · On Mon, 2026-01-05 at 22:21 -0800, [email protected] wrote:
Quoted 1 line · Trond Myklebust
Does applying the following on top of Anna's patch fix the Oops?
It does. But now generic/633 crashes reliably:
generic/633 2s ... [ 58.670535] run fstests generic/633 at 2026- 01-02 02:00:17
[ 58.865568] process 'vfstest' launched '/dev/fd/4/file1' with NULL
argv: empty string added
[ 58.897522] Oops: general protection fault, probably for non- canonical address0xcccccccccccccd0c: 0000 [#1] SMP NOPTI [ 58.898234] CPU: 0 UID: 0 PID: 3852 Comm: vfstest Tainted:
G N 6.19.0-rc2+ #4535 PREEMPT(full)
[ 58.898829] Tainted: [N]=TEST
[ 58.899013] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 58.899594] RIP: 0010:nfs_end_delegation_return+0xda/0x390 [ 58.899922] Code: 49 89 ce e8 58 be bf ff 48 8b 85 60 ff ff ff 4c 8d 68 80 49 39 c6 74 50 4c 8b 7c 24 08 48 8b 1c 24 4d 8b 65 60 4d 85 e4 74 2e <49> 8b 44 24 40 a8 02 74 25 49 8b 44 24 40 f6 c4 02 75 1b 41 8b 47
[ 58.901063] RSP: 0018:ffffc90001947c90 EFLAGS: 00010286 [ 58.901419] RAX:ffff88811a3122c0RBX:ffff888105268970RCX:ffff888111a9a210
[ 58.901827] RDX:ffff8881039320c0RSI:ffff888105268940RDI:ffff888111a9a2b0
[ 58.902236] RBP:ffff888111a9a2b0R08: 0000000000000000 R09:
0000000000000000
[ 58.902696] R10:ffffc90001947d48R11: fefefefefefefeff R12:
cccccccccccccccc
[ 58.903112] R13:ffff88811a312240R14:ffff888111a9a210R15:ffff888105268940
[ 58.903594] FS: 00007f04f92c6740(0000) GS:ffff8882b353d000(0000) knlGS:0000000000000000
[ 58.904125] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 58.904483] CR2:000055c55fe522d8CR3:000000011a3f1004CR4:0000000000772ef0
[ 58.904927] PKRU: 55555554
[ 58.905132] Call Trace:
[ 58.905294] <TASK>
[ 58.905434] ?_raw_spin_unlock+0x13/0x30
[ 58.905689]nfs4_proc_setattr+0xff/0x110
[ 58.905947]nfs_setattr+0x1c8/0x410
[ 58.906175]notify_change+0x373/0x510
[ 58.906415] ?init_object+0x5a/0xc0
[ 58.906643] ?chown_common+0x1ec/0x220
[ 58.906885]chown_common+0x1ec/0x220
[ 58.907126]do_fchownat+0xc3/0xf0
[ 58.907358]__x64_sys_fchownat+0x1a/0x30
[ 58.907611]do_syscall_64+0x50/0xf80
[ 58.907848]entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 58.908171] RIP: 0033:0x7f04f93c8e4a
[ 58.908390] Code: 48 8b 0d b1 6f 0e 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 49 89 ca b8 04 01 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 7e 6f 0e 00 f7 d8 64 89 01 48
[ 58.909467] RSP: 002b:00007ffe4cd3ec98 EFLAGS: 00000246ORIG_RAX:
0000000000000104
[ 58.909893] RAX: ffffffffffffffda RBX:00007ffe4cd3ecb0RCX:00007f04f93c8e4a
[ 58.910336] RDX: 0000000000002710 RSI:000055c54836500eRDI:
0000000000000003
[ 58.910872] RBP:000055c55fd522a0R08: 0000000000000100 R09:
0000000000000001
[ 58.911350] R10: 0000000000002710 R11: 0000000000000246 R12:
0000000000000006
[ 58.911797] R13: 0000000000002710 R14: 0000000000002710 R15:000055c55fd52313
[ 58.912234] </TASK>
[ 58.912373] Modules linked in:kvm_intelkvm irqbypass [ 58.912717] ---[ end trace 0000000000000000 ]---
[ 58.913829] RIP: 0010:nfs_end_delegation_return+0xda/0x390 [ 58.914177] Code: 49 89 ce e8 58 be bf ff 48 8b 85 60 ff ff ff 4c 8d 68 80 49 39 c6 74 50 4c 8b 7c 24 08 48 8b 1c 24 4d 8b 65 60 4d 85 e4 74 2e <49> 8b 44 24 40 a8 02 74 25 49 8b 44 24 40 f6 c4 02 75 1b 41 8b 47
[ 58.915541] RSP: 0018:ffffc90001947c90 EFLAGS: 00010286 [ 58.915871] RAX:ffff88811a3122c0RBX:ffff888105268970RCX:ffff888111a9a210
[ 58.916317] RDX:ffff8881039320c0RSI:ffff888105268940RDI:ffff888111a9a2b0
[ 58.916766] RBP:ffff888111a9a2b0R08: 0000000000000000 R09:
0000000000000000
[ 58.917299] R10:ffffc90001947d48R11: fefefefefefefeff R12:
cccccccccccccccc
Quoted 8 lines
[ 58.917709] R13:
ffff88811a312240R14:ffff888111a9a210R15:ffff888105268940
[ 58.918109] FS: 00007f04f92c6740(0000) GS:ffff8882b353d000(0000) knlGS:0000000000000000
[ 58.918560] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 58.918944] CR2:00007fbf39569000CR3:000000011a3f1004CR4:0000000000772ef0
[ 58.919622] PKRU: 55555554
Sigh... One last patch on top of all the previous ones, but if we hit another issue I think we need to consider just disabling directory delegations on the client until all the remaining issues can be fixed in the next release.
Anna, are you able to reproduce these bugs?
8<-----------------------------------------------------------------
From cbf97626edbf8c0b619d37aca8d6da77f46e69d6 Mon Sep 17 00:00:00 2001
Message-ID: <cbf97626edbf8c0b619d37aca8d6da77f46e69d6.1767719343.git.trond.myklebust@hammerspace.com>
From: Trond Myklebust <[email protected]>
Date: Tue, 6 Jan 2026 11:54:32 -0500
Subject: [PATCH] NFSv4.x: Directory delegations don't require any state
recovery
The state recovery code in nfs_end_delegation_return() is intended to allow regular files to recover cached open and lock state. It has no function for directory delegations, and may cause corruption.
--- fs/nfs/delegation.c | 5 +++++ 1 file changed, 5 insertions(+)
fs/nfs/delegation.c +5 -0
@@ -581,6 +581,10 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegationif (delegation ==NULL )return 0 ; +/* Directory delegations don't require any state recovery */ +if (!S_ISREG (inode->i_mode)) + goto out_return; +if (!issync) mode |=O_NONBLOCK ;/* Recall of any remaining application leases */
@@ -604,6 +608,7 @@ static int nfs_end_delegation_return(struct inode *inode, struct nfs_delegation goto out; } +out_return: err =nfs_do_return_delegation (inode, delegation, issync); out:/* Refcount matched in nfs_start_delegation_return_locked() */
Signature · 7 lines
2.52.0 -- Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected], [email protected]
Quoted 4 lines · Trond Myklebust
Sigh... One last patch on top of all the previous ones, but if we hit another issue I think we need to consider just disabling directory delegations on the client until all the remaining issues can be fixed in the next release.
This still crasheѕ generic/633:
generic/633 4s ... [ 73.075526] run fstests generic/633 at 2026-01-07 05:21:37
[ 73.286318] process 'vfstest' launched '/dev/fd/4/file1' with NULL argv: empty string added [ 73.314625] Oops: general protection fault, probably for non-canonical address 0xcccccccccccccd50: 0000 [#1] SMP NOPTI [ 73.315391] CPU: 1 UID: 0 PID: 100 Comm: kworker/u8:3 Tainted: G N 6.19.0-rc4+ #4540 PREEMPT(full) [ 73.316043] Tainted: [N]=TEST [ 73.316229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 73.316786] Workqueue: rpciod rpc_async_schedule [ 73.317066] RIP: 0010:nfs_inode_find_state_and_recover+0x9c/0x260 [ 73.317430] Code: 01 0f 85 83 00 00 00 49 8b 84 24 80 00 00 00 4c 8d 60 80 48 39 c3 0f 84 68 01 00 00 4d 8b 7c 24 60 4d 85 ff 74 e1 48 8b 7d 00 <49> 39 bf 84 00 00 00 75 af 8b 4d 08 41 39 8f 8c 00 00 00 75 a3 48 [ 73.318513] RSP: 0018:ffffc900001bfd00 EFLAGS: 00010286 [ 73.318868] RAX: ffff88810cfb98e0 RBX: ffff8881d68d8c50 RCX: 0000000000000000 [ 73.319354] RDX: ffff88810330a0c0 RSI: ffff88811a86d914 RDI: b6162427695ded30 [ 73.319832] RBP: ffff88811a86d918 R08: ffff88810c685100 R09: ffff88810c685130 [ 73.320270] R10: 0000000000000003 R11: fefefefefefefeff R12: ffff88810cfb9860 [ 73.320698] R13: ffff8881179b2800 R14: 0000000000000000 R15: cccccccccccccccc [ 73.321135] FS: 0000000000000000(0000) GS:ffff8882b363d000(0000) knlGS:0000000000000000 [ 73.321640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 73.321994] CR2: 0000559277cef238 CR3: 0000000161792002 CR4: 0000000000772ef0 [ 73.322422] PKRU: 55555554 [ 73.322592] Call Trace: [ 73.322749] <TASK> [ 73.322889] nfs4_delegreturn_done+0x1b7/0x380 [ 73.323173] ? __pfx_rpc_exit_task+0x10/0x10 [ 73.323475] rpc_exit_task+0x5c/0x170 [ 73.323755] __rpc_execute+0xb1/0x490 [ 73.324000] rpc_async_schedule+0x2a/0x40 [ 73.324257] process_one_work+0x16c/0x330 [ 73.324515] worker_thread+0x254/0x3a0 [ 73.324762] ? __pfx_worker_thread+0x10/0x10 [ 73.325042] kthread+0x117/0x230 [ 73.325303] ? __pfx_kthread+0x10/0x10 [ 73.325557] ? __pfx_kthread+0x10/0x10 [ 73.325812] ret_from_fork+0x1b6/0x200 [ 73.326045] ? __pfx_kthread+0x10/0x10 [ 73.326278] ret_from_fork_asm+0x1a/0x30 [ 73.326535] </TASK> [ 73.326681] Modules linked in: kvm_intel kvm irqbypass [ 73.327046] ---[ end trace 0000000000000000 ]---
Trond Myklebust
maintainerQuoted 55 lines · On Tue, 2026-01-06 at 21:23 -0800, [email protected] wrote:
Quoted 6 lines · Trond Myklebust
Sigh... One last patch on top of all the previous ones, but if we hit
another issue I think we need to consider just disabling directory delegations on the client until all the remaining issues can be fixed
in the next release.This still crasheѕ generic/633:
generic/633 4s ... [ 73.075526] run fstests generic/633 at 2026- 01-07 05:21:37
[ 73.286318] process 'vfstest' launched '/dev/fd/4/file1' with NULL
argv: empty string added
[ 73.314625] Oops: general protection fault, probably for non- canonical address0xcccccccccccccd50: 0000 [#1] SMP NOPTI [ 73.315391] CPU: 1 UID: 0 PID: 100 Comm: kworker/u8:3 Tainted:
G N 6.19.0-rc4+ #4540 PREEMPT(full)
[ 73.316043] Tainted: [N]=TEST
[ 73.316229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 73.316786] Workqueue: rpciodrpc_async_schedule
[ 73.317066] RIP: 0010:nfs_inode_find_state_and_recover+0x9c/0x260 [ 73.317430] Code: 01 0f 85 83 00 00 00 49 8b 84 24 80 00 00 00 4c 8d 60 80 48 39 c3 0f 84 68 01 00 00 4d 8b 7c 24 60 4d 85 ff 74 e1 48 8b 7d 00 <49> 39 bf 84 00 00 00 75 af 8b 4d 08 41 39 8f 8c 00 00 00 75 a3 48
[ 73.318513] RSP: 0018:ffffc900001bfd00 EFLAGS: 00010286 [ 73.318868] RAX:ffff88810cfb98e0RBX:ffff8881d68d8c50RCX:
0000000000000000
[ 73.319354] RDX:ffff88810330a0c0RSI:ffff88811a86d914RDI:b6162427695ded30
[ 73.319832] RBP:ffff88811a86d918R08:ffff88810c685100R09:ffff88810c685130
[ 73.320270] R10: 0000000000000003 R11: fefefefefefefeff R12:ffff88810cfb9860
[ 73.320698] R13:ffff8881179b2800R14: 0000000000000000 R15:
cccccccccccccccc
[ 73.321135] FS: 0000000000000000(0000) GS:ffff8882b363d000(0000) knlGS:0000000000000000
[ 73.321640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 73.321994] CR2:0000559277cef238CR3: 0000000161792002 CR4:0000000000772ef0
[ 73.322422] PKRU: 55555554
[ 73.322592] Call Trace:
[ 73.322749] <TASK>
[ 73.322889]nfs4_delegreturn_done+0x1b7/0x380
[ 73.323173] ?__pfx_rpc_exit_task+0x10/0x10
[ 73.323475]rpc_exit_task+0x5c/0x170
[ 73.323755]__rpc_execute+0xb1/0x490
[ 73.324000]rpc_async_schedule+0x2a/0x40
[ 73.324257]process_one_work+0x16c/0x330
[ 73.324515]worker_thread+0x254/0x3a0
[ 73.324762] ?__pfx_worker_thread+0x10/0x10
[ 73.325042]kthread+0x117/0x230
Quoted 8 lines
[ 73.325303] ?
__pfx_kthread+0x10/0x10
[ 73.325557] ?__pfx_kthread+0x10/0x10
[ 73.325812]ret_from_fork+0x1b6/0x200
[ 73.326045] ?__pfx_kthread+0x10/0x10
[ 73.326278]ret_from_fork_asm+0x1a/0x30
[ 73.326535] </TASK>
[ 73.326681] Modules linked in:kvm_intelkvm irqbypass [ 73.327046] ---[ end trace 0000000000000000 ]---
Thanks for doing all this testing, Christoph. I really appreciate it. The previous patch was incomplete. This is incremental to yesterday's patch, but I'll squash them together in the testing branch, since they're both about blocking state recovery in the non-regular file case.
8<-----------------------------------------------------------------
From 534676d290af6fae6ad1b067b81c13523340bc83 Mon Sep 17 00:00:00 2001
Message-ID: <534676d290af6fae6ad1b067b81c13523340bc83.1767798220.git.trond.myklebust@hammerspace.com>
From: Trond Myklebust <[email protected]>
Date: Wed, 7 Jan 2026 10:01:58 -0500
Subject: [PATCH] NFSv4.x: Directory delegations don't require any state
recovery (part 2)
This patch will be squashed with part 1.
--- fs/nfs/nfs4state.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
fs/nfs/nfs4state.c +5 -1
@@ -1445,6 +1445,8 @@ void nfs_inode_find_state_and_recover(struct inode *inode,struct nfs4_state *state;bool found = false; +if (!S_ISREG (inode->i_mode)) + goto out;rcu_read_lock ();list_for_each_entry_rcu (ctx, &nfsi->open_files, list) { state = ctx->state;
@@ -1466,7 +1468,7 @@ void nfs_inode_find_state_and_recover(struct inode *inode, found = true; }rcu_read_unlock (); - +out:nfs_inode_find_delegation_state_and_recover (inode, stateid);if (found)nfs4_schedule_state_manager (clp);
@@ -1478,6 +1480,8 @@ static void nfs4_state_mark_open_context_bad(struct nfs4_state *state, int err)struct nfs_inode *nfsi =NFS_I (inode);struct nfs_open_context *ctx; +if (!S_ISREG (inode->i_mode)) +return ;rcu_read_lock ();list_for_each_entry_rcu (ctx, &nfsi->open_files, list) {if (ctx->state != state)
Signature · 1 line
2.52.0
Quoted 5 lines · Trond Myklebust
Thanks for doing all this testing, Christoph. I really appreciate it. The previous patch was incomplete. This is incremental to yesterday's patch, but I'll squash them together in the testing branch, since they're both about blocking state recovery in the non-regular file case.
This fixes generic/633. I've kicked off a full xfstests run and will report back.
Quoted 9 lines · On Wed, Jan 07, 2026 at 07:30:48AM -0800, [email protected] wrote:
Quoted 5 lines · Trond Myklebust
Thanks for doing all this testing, Christoph. I really appreciate it. The previous patch was incomplete. This is incremental to yesterday's patch, but I'll squash them together in the testing branch, since they're both about blocking state recovery in the non-regular file case.
This fixes generic/633. I've kicked off a full xfstests run and will report back.
xfstests survives without crashing. However since 6.19-rc a bunch of timestamp related tests keep faіling.
Hi Trond,
Quoted 79 lines · Trond Myklebust
Quoted 55 lines · On Tue, 2026-01-06 at 21:23 -0800, [email protected] wrote:
Quoted 6 lines · Trond Myklebust
Sigh... One last patch on top of all the previous ones, but if we hit
another issue I think we need to consider just disabling directory delegations on the client until all the remaining issues can be fixed
in the next release.This still crasheѕ generic/633:
generic/633 4s ... [ 73.075526] run fstests generic/633 at 2026- 01-07 05:21:37
[ 73.286318] process 'vfstest' launched '/dev/fd/4/file1' with NULL
argv: empty string added
[ 73.314625] Oops: general protection fault, probably for non- canonical address0xcccccccccccccd50: 0000 [#1] SMP NOPTI [ 73.315391] CPU: 1 UID: 0 PID: 100 Comm: kworker/u8:3 Tainted:
G N 6.19.0-rc4+ #4540 PREEMPT(full)
[ 73.316043] Tainted: [N]=TEST
[ 73.316229] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[ 73.316786] Workqueue: rpciodrpc_async_schedule
[ 73.317066] RIP: 0010:nfs_inode_find_state_and_recover+0x9c/0x260 [ 73.317430] Code: 01 0f 85 83 00 00 00 49 8b 84 24 80 00 00 00 4c 8d 60 80 48 39 c3 0f 84 68 01 00 00 4d 8b 7c 24 60 4d 85 ff 74 e1 48 8b 7d 00 <49> 39 bf 84 00 00 00 75 af 8b 4d 08 41 39 8f 8c 00 00 00 75 a3 48
[ 73.318513] RSP: 0018:ffffc900001bfd00 EFLAGS: 00010286 [ 73.318868] RAX:ffff88810cfb98e0RBX:ffff8881d68d8c50RCX:
0000000000000000
[ 73.319354] RDX:ffff88810330a0c0RSI:ffff88811a86d914RDI:b6162427695ded30
[ 73.319832] RBP:ffff88811a86d918R08:ffff88810c685100R09:ffff88810c685130
[ 73.320270] R10: 0000000000000003 R11: fefefefefefefeff R12:ffff88810cfb9860
[ 73.320698] R13:ffff8881179b2800R14: 0000000000000000 R15:
cccccccccccccccc
[ 73.321135] FS: 0000000000000000(0000) GS:ffff8882b363d000(0000) knlGS:0000000000000000
[ 73.321640] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 73.321994] CR2:0000559277cef238CR3: 0000000161792002 CR4:0000000000772ef0
[ 73.322422] PKRU: 55555554
[ 73.322592] Call Trace:
[ 73.322749] <TASK>
[ 73.322889]nfs4_delegreturn_done+0x1b7/0x380
[ 73.323173] ?__pfx_rpc_exit_task+0x10/0x10
[ 73.323475]rpc_exit_task+0x5c/0x170
[ 73.323755]__rpc_execute+0xb1/0x490
[ 73.324000]rpc_async_schedule+0x2a/0x40
[ 73.324257]process_one_work+0x16c/0x330
[ 73.324515]worker_thread+0x254/0x3a0
[ 73.324762] ?__pfx_worker_thread+0x10/0x10
[ 73.325042]kthread+0x117/0x230Quoted 8 lines
[ 73.325303] ?
__pfx_kthread+0x10/0x10
[ 73.325557] ?__pfx_kthread+0x10/0x10
[ 73.325812]ret_from_fork+0x1b6/0x200
[ 73.326045] ?__pfx_kthread+0x10/0x10
[ 73.326278]ret_from_fork_asm+0x1a/0x30
[ 73.326535] </TASK>
[ 73.326681] Modules linked in:kvm_intelkvm irqbypass [ 73.327046] ---[ end trace 0000000000000000 ]---Thanks for doing all this testing, Christoph. I really appreciate it. The previous patch was incomplete. This is incremental to yesterday's patch, but I'll squash them together in the testing branch, since they're both about blocking state recovery in the non-regular file case.
8<-----------------------------------------------------------------
From534676d290af6fae6ad1b067b81c13523340bc83Mon Sep 17 00:00:00 2001
Message-ID: <534676d290af6fae6ad1b067b81c13523340bc83.1767798220.git.trond.myklebust@hammerspace.com>
From: Trond Myklebust <[email protected]>
Date: Wed, 7 Jan 2026 10:01:58 -0500
Subject: [PATCH] NFSv4.x: Directory delegations don't require any state
recovery (part 2)
Quoted 1 line
This patch will be squashed with part 1.
I finally hit Christoph's crasher this morning, and this patch did help when running with NFS v4.1 but I still see the problem with v4.2. I haven't dug into it enough to know if it's just harder to hit now, though.
Do you want me to take back over debugging this from here? Anna
Quoted 37 lines
Signed-off-by Trond Myklebust <[email protected]>--- fs/nfs/nfs4state.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)fs/nfs/nfs4state.c +5
-1@@ -1445,6 +1445,8 @@ void nfs_inode_find_state_and_recover(struct inode *inode,struct nfs4_state *state;bool found = false; +if (!S_ISREG (inode->i_mode)) + goto out;rcu_read_lock ();list_for_each_entry_rcu (ctx, &nfsi->open_files, list) { state = ctx->state;@@ -1466,7 +1468,7 @@ void nfs_inode_find_state_and_recover(struct inode *inode, found = true; }rcu_read_unlock (); - +out:nfs_inode_find_delegation_state_and_recover (inode, stateid);if (found)nfs4_schedule_state_manager (clp);@@ -1478,6 +1480,8 @@ static void nfs4_state_mark_open_context_bad(struct nfs4_state *state, int err)struct nfs_inode *nfsi =NFS_I (inode);struct nfs_open_context *ctx; +if (!S_ISREG (inode->i_mode)) +return ;rcu_read_lock ();list_for_each_entry_rcu (ctx, &nfsi->open_files, list) {if (ctx->state != state)
Al Viro
maintainer[cc to fsdevel added]
Quoted 11 lines · Trond Myklebust
fs/nfs/dir.c +11
-0+static void nfs_clear_verifier_directory (struct inode *dir) +{ +struct dentry *this_parent; +struct dentry *dentry; +struct inode *inode; + +if (hlist_empty (&dir->i_dentry)) +return ; + this_parent = + hlist_entry(dir->i_dentry.first, struct dentry, d_u.d_alias); +
fs/nfs/dir.c void nfs_set_verifier(struct dentry *dentry, unsigned long verf) · line 1504
Quoted 10 lines
fs/nfs/dir.c +10
-0+spin_lock (&this_parent->d_lock); +nfs_unset_verifier_delegated (&this_parent->d_time); + dentry =d_first_child (this_parent); +hlist_for_each_entry_from (dentry, d_sib) { +if (unlikely (dentry->d_flags &DCACHE_DENTRY_CURSOR )) +continue ; + inode =d_inode_rcu (dentry); +if (inode && +NFS_PROTO (inode)->have_delegation (inode,FMODE_READ ,0 )) +continue ;
What's to stop the inode from being freed right under you? You are *not* guaranteed to be holding rcu_read_lock(), unless I'm missing something in the call chain, so... what's going on here?
Al Viro
maintainerQuoted 21 lines · Al Viro
[cc to fsdevel added]
Quoted 17 lines · Trond Myklebust
fs/nfs/dir.c +17
-0+static void nfs_clear_verifier_directory (struct inode *dir) +{ +struct dentry *this_parent; +struct dentry *dentry; +struct inode *inode; + +if (hlist_empty (&dir->i_dentry)) +return ; + this_parent = + hlist_entry(dir->i_dentry.first, struct dentry, d_u.d_alias); + +spin_lock (&this_parent->d_lock); +nfs_unset_verifier_delegated (&this_parent->d_time); + dentry =d_first_child (this_parent); +hlist_for_each_entry_from (dentry, d_sib) { +if (unlikely (dentry->d_flags &DCACHE_DENTRY_CURSOR )) +continue ;
Quoted 8 lines
fs/nfs/dir.c void nfs_set_verifier(struct dentry *dentry, unsigned long verf) · line 1504
Quoted 4 lines
fs/nfs/dir.c +4
-0+ inode =d_inode_rcu (dentry); +if (inode && +NFS_PROTO (inode)->have_delegation (inode,FMODE_READ ,0 )) +continue ;What's to stop the inode from being freed right under you? You are *not* guaranteed to be holding
rcu_read_lock(), unless I'm missing something in the call chain, so... what's going on here?
While we are at it, what are cursors doing on NFS directories? AFAICS, you are not using simple_dir_operations there or anything that would call dcache_dir_open(), for that matter. Confused...
Al Viro
maintainerQuoted 21 lines · Al Viro
[cc to fsdevel added]
Quoted 17 lines · Trond Myklebust
fs/nfs/dir.c +17
-0+static void nfs_clear_verifier_directory (struct inode *dir) +{ +struct dentry *this_parent; +struct dentry *dentry; +struct inode *inode; + +if (hlist_empty (&dir->i_dentry)) +return ; + this_parent = + hlist_entry(dir->i_dentry.first, struct dentry, d_u.d_alias); + +spin_lock (&this_parent->d_lock); +nfs_unset_verifier_delegated (&this_parent->d_time); + dentry =d_first_child (this_parent); +hlist_for_each_entry_from (dentry, d_sib) { +if (unlikely (dentry->d_flags &DCACHE_DENTRY_CURSOR )) +continue ;
Quoted 8 lines
fs/nfs/dir.c void nfs_set_verifier(struct dentry *dentry, unsigned long verf) · line 1504
Quoted 4 lines
fs/nfs/dir.c +4
-0+ inode =d_inode_rcu (dentry); +if (inode && +NFS_PROTO (inode)->have_delegation (inode,FMODE_READ ,0 )) +continue ;What's to stop the inode from being freed right under you? You are *not* guaranteed to be holding
rcu_read_lock(), unless I'm missing something in the call chain, so... what's going on here?
Incidentally, nfs_clear_verifier_file() doesn't need to bother with checking for NULL dir - alias is found in ->i_dentry of inode with ->i_lock held, which means it can't have progressed through __dentry_kill() to dropping the reference to parent. So holding alias->d_lock (which stabilizes ->d_parent) is enough - ->d_parent is still pinned by alias and it can't have become negative.
"Can't have progressed through __dentry_kill()" is critical here - that's the difference from nfs_set_verifier_locked() case.
Folks, this stuff can get seriously subtle; _please_ ask around on fsdevel if you are doing something non-trivial with it.