WARNING at fs/nfs/write.c:743 nfs_inode_remove_request with -rc6
Subsystems (2)
Hi all,
I've been running into the following warning on an arm64 system running 3.17-rc6 with 64k pages. I've been unable to reproduce with a smaller page size (4k).
I don't yet have a concrete reproducer, but I've seen it hit a few times today just running a machine with an NFS root filesystem and using ssh. The warning seems to happen in parallel on the two CPUs, but I'm pretty confident that our test_and_clear_bit implementation has the relevant atomic instructions and memory barriers.
Any ideas?
Will
--->8
------------[ cut here ]------------ WARNING: CPU: 1 PID: 1023 at fs/nfs/write.c:743 nfs_inode_remove_request+0xe4/0xf0() Modules linked in: CPU: 1 PID: 1023 Comm: kworker/1:2 Not tainted 3.17.0-rc6 #1 Workqueue: nfsiod rpc_async_release Call trace: [<fffffe0000096410>] dump_backtrace+0x0/0x130 [<fffffe0000096550>] show_stack+0x10/0x1c [<fffffe00004cda94>] dump_stack+0x74/0xbc [<fffffe00000b4d20>] warn_slowpath_common+0x8c/0xb4 [<fffffe00000b4e0c>] warn_slowpath_null+0x14/0x20 [<fffffe000027a6a8>] nfs_inode_remove_request+0xe0/0xf0 [<fffffe000027b704>] nfs_write_completion+0xb4/0x150 [<fffffe0000276ef4>] nfs_pgio_release+0x34/0x44 [<fffffe00004ac2d0>] rpc_free_task+0x24/0x4c [<fffffe00004ac5c0>] rpc_async_release+0xc/0x18 [<fffffe00000c89e8>] process_one_work+0x140/0x32c [<fffffe00000c9338>] worker_thread+0x13c/0x470 [<fffffe00000cd9e4>] kthread+0xd0/0xe8 ---[ end trace 6f044efb83f0811b ]--- ------------[ cut here ]------------ WARNING: CPU: 0 PID: 621 at fs/nfs/write.c:743 nfs_inode_remove_request+0xe4/0xf0() CPU: 0 PID: 621 Comm: kworker/0:2 Tainted: G W 3.17.0-rc6 #1 Workqueue: nfsiod rpc_async_release Call trace: [<fffffe0000096410>] dump_backtrace+0x0/0x130 [<fffffe0000096550>] show_stack+0x10/0x1c [<fffffe00004cda94>] dump_stack+0x74/0xbc [<fffffe00000b4d20>] warn_slowpath_common+0x8c/0xb4 [<fffffe00000b4e0c>] warn_slowpath_null+0x14/0x20 [<fffffe000027a6a8>] nfs_inode_remove_request+0xe0/0xf0 [<fffffe000027b704>] nfs_write_completion+0xb4/0x150 [<fffffe0000276ef4>] nfs_pgio_release+0x34/0x44 [<fffffe00004ac2d0>] rpc_free_task+0x24/0x4c [<fffffe00004ac5c0>] rpc_async_release+0xc/0x18 [<fffffe00000c89e8>] process_one_work+0x140/0x32c [<fffffe00000c9338>] worker_thread+0x13c/0x470 [<fffffe00000cd9e4>] kthread+0xd0/0xe8 ---[ end trace 6f044efb83f0811c ]---
Quoted 12 lines · Will Deacon
Hi all,
I've been running into the following warning on an arm64 system running 3.17-rc6 with 64k pages. I've been unable to reproduce with a smaller page size (4k).
I don't yet have a concrete reproducer, but I've seen it hit a few times today just running a machine with an NFS root filesystem and using ssh. The warning seems to happen in parallel on the two CPUs, but I'm pretty confident that our
test_and_clear_bitimplementation has the relevant atomic instructions and memory barriers.
Quoted 3 lines
Any ideas?
Will
So it looks like we’re either calling nfs_inode_remove_request twice on a request, or somehow not grabbing the inode reference for some request that is in the async write path. It’s interesting that these come in pairs - that has to mean something!
Any more info on how to reproduce this would be really great. Unfortunately I don’t have access to an arm64 system.
If it’s possible, could we get a packet trace around when this happens? This is pure speculation, but this might have something to do the resend path - a commit fails and all the requests on the commit list have to be resent.
Have you noticed any side effects from this? That WARN_ON_ONCE was added to sanity test the new page group code and we need to fix this, but I’m wondering if anything “bad” happens…
-dros
Quoted 43 lines
--->8
------------[ cut here ]------------ WARNING: CPU: 1 PID: 1023 at fs/nfs/write.c:743 nfs_inode_remove_request+0xe4/0xf0() Modules linked in: CPU: 1 PID: 1023 Comm: kworker/1:2 Not tainted 3.17.0-rc6 #1 Workqueue: nfsiod rpc_async_release Call trace: [<fffffe0000096410>] dump_backtrace+0x0/0x130 [<fffffe0000096550>] show_stack+0x10/0x1c [<fffffe00004cda94>] dump_stack+0x74/0xbc [<fffffe00000b4d20>] warn_slowpath_common+0x8c/0xb4 [<fffffe00000b4e0c>] warn_slowpath_null+0x14/0x20 [<fffffe000027a6a8>] nfs_inode_remove_request+0xe0/0xf0 [<fffffe000027b704>] nfs_write_completion+0xb4/0x150 [<fffffe0000276ef4>] nfs_pgio_release+0x34/0x44 [<fffffe00004ac2d0>] rpc_free_task+0x24/0x4c [<fffffe00004ac5c0>] rpc_async_release+0xc/0x18 [<fffffe00000c89e8>] process_one_work+0x140/0x32c [<fffffe00000c9338>] worker_thread+0x13c/0x470 [<fffffe00000cd9e4>] kthread+0xd0/0xe8 ---[ end trace 6f044efb83f0811b ]--- ------------[ cut here ]------------ WARNING: CPU: 0 PID: 621 at fs/nfs/write.c:743 nfs_inode_remove_request+0xe4/0xf0() CPU: 0 PID: 621 Comm: kworker/0:2 Tainted: G W 3.17.0-rc6 #1 Workqueue: nfsiod rpc_async_release Call trace: [<fffffe0000096410>] dump_backtrace+0x0/0x130 [<fffffe0000096550>] show_stack+0x10/0x1c [<fffffe00004cda94>] dump_stack+0x74/0xbc [<fffffe00000b4d20>] warn_slowpath_common+0x8c/0xb4 [<fffffe00000b4e0c>] warn_slowpath_null+0x14/0x20 [<fffffe000027a6a8>] nfs_inode_remove_request+0xe0/0xf0 [<fffffe000027b704>] nfs_write_completion+0xb4/0x150 [<fffffe0000276ef4>] nfs_pgio_release+0x34/0x44 [<fffffe00004ac2d0>] rpc_free_task+0x24/0x4c [<fffffe00004ac5c0>] rpc_async_release+0xc/0x18 [<fffffe00000c89e8>] process_one_work+0x140/0x32c [<fffffe00000c9338>] worker_thread+0x13c/0x470 [<fffffe00000cd9e4>] kthread+0xd0/0xe8 ---[ end trace 6f044efb83f0811c ]---
Quoted 11 lines · Weston Andros Adamson
Quoted 10 lines · Will Deacon
I've been running into the following warning on an arm64 system running 3.17-rc6 with 64k pages. I've been unable to reproduce with a smaller page size (4k).
I don't yet have a concrete reproducer, but I've seen it hit a few times today just running a machine with an NFS root filesystem and using ssh. The warning seems to happen in parallel on the two CPUs, but I'm pretty confident that our
test_and_clear_bitimplementation has the relevant atomic instructions and memory barriers.
Quoted 5 lines
Quoted 1 line
Any ideas?
So it looks like we’re either calling
nfs_inode_remove_requesttwice on a request, or somehow not grabbing the inode reference for some request that is in the async write path. It’s interesting that these come in pairs - that has to mean something!
Indeed. I have 6 CPUs on this system too, so it's not a per-cpu thing.
Quoted 2 lines
Any more info on how to reproduce this would be really great. Unfortunately I don’t have access to an arm64 system.
I've not spotted a pattern other than using 64k pages, yet. If I manage to get a reproducer, I'll let you know.
Quoted 3 lines
If it’s possible, could we get a packet trace around when this happens? This is pure speculation, but this might have something to do the resend path - a commit fails and all the requests on the commit list have to be resent.
Sure, once I can reproduce it reliably, then I'll try to do that.
Quoted 3 lines
Have you noticed any side effects from this? That
WARN_ON_ONCEwas added to sanity test the new page group code and we need to fix this, but I’m wondering if anything “bad” happens…
I've not noticed anything. In fact, this happened during an LTP run and I didn't see any regressions in the test results.
Will
Quoted 4 lines · Will Deacon
Quoted 2 lines · Weston Andros Adamson
Any more info on how to reproduce this would be really great. Unfortunately I don’t have access to an arm64 system.
Quoted 8 lines
I've not spotted a pattern other than using 64k pages, yet. If I manage to get a reproducer, I'll let you know.
Quoted 3 lines
If it’s possible, could we get a packet trace around when this happens? This is pure speculation, but this might have something to do the resend path - a commit fails and all the requests on the commit list have to be resent.
Sure, once I can reproduce it reliably, then I'll try to do that.
Right, a bunch of DDing from /dev/zero over an ssh session triggers this very quickly. I've put a binary tcpdump here:
http://www.willdeacon.ukfsn.org/bitbucket/oopsen/nfs/tcpdump.bin
You may want to filter out the ssh packets. The only `interesting' thing to me is a retransmission about half way through, but I don't know what I'm looking for.
The NFS server is 10.1.203.204 and the client is 10.1.203.24.
Will
Quoted 17 lines · Will Deacon
Quoted 4 lines · Will Deacon
Quoted 2 lines · Weston Andros Adamson
Any more info on how to reproduce this would be really great. Unfortunately I don’t have access to an arm64 system.
Quoted 8 lines
I've not spotted a pattern other than using 64k pages, yet. If I manage to get a reproducer, I'll let you know.
Quoted 3 lines
If it’s possible, could we get a packet trace around when this happens? This is pure speculation, but this might have something to do the resend path - a commit fails and all the requests on the commit list have to be resent.
Sure, once I can reproduce it reliably, then I'll try to do that.
Right, a bunch of DDing from
/dev/zeroover an ssh session triggers this very quickly. I've put a binary tcpdump here:
Quoted 7 lines
http://www.willdeacon.ukfsn.org/bitbucket/oopsen/nfs/tcpdump.bin
You may want to filter out the ssh packets. The only `interesting' thing to me is a retransmission about half way through, but I don't know what I'm looking for.
The NFS server is 10.1.203.204 and the client is 10.1.203.24.
Thanks! You are using NFSv2, so there are no commits - that rules out my hunch.
-dros
Quoted 21 lines · Weston Andros Adamson
Quoted 19 lines · Will Deacon
Quoted 4 lines · Will Deacon
Quoted 2 lines · Weston Andros Adamson
Any more info on how to reproduce this would be really great. Unfortunately I don’t have access to an arm64 system.
Quoted 8 lines
I've not spotted a pattern other than using 64k pages, yet. If I manage to get a reproducer, I'll let you know.
Quoted 3 lines
If it’s possible, could we get a packet trace around when this happens? This is pure speculation, but this might have something to do the resend path - a commit fails and all the requests on the commit list have to be resent.
Sure, once I can reproduce it reliably, then I'll try to do that.
Right, a bunch of DDing from
/dev/zeroover an ssh session triggers this very quickly. I've put a binary tcpdump here:http://www.willdeacon.ukfsn.org/bitbucket/oopsen/nfs/tcpdump.bin
Quoted 7 lines
Quoted 5 lines
You may want to filter out the ssh packets. The only `interesting' thing to me is a retransmission about half way through, but I don't know what I'm looking for.
The NFS server is 10.1.203.204 and the client is 10.1.203.24.
Thanks! You are using NFSv2, so there are no commits - that rules out my hunch.
Wait a second - the whole point of this extra reference (that the WARN_ON_ONCE is related to) is to handle the pass off to commit lists.
Maybe I’m just doing the wrong thing here with NFSv2!
More soon.
-dros
Quoted 29 lines · Weston Andros Adamson
Quoted 27 lines · Weston Andros Adamson
Quoted 24 lines · Will Deacon
Quoted 4 lines · Will Deacon
Quoted 2 lines · Weston Andros Adamson
Any more info on how to reproduce this would be really great. Unfortunately I don’t have access to an arm64 system.
Quoted 8 lines
I've not spotted a pattern other than using 64k pages, yet. If I manage to get a reproducer, I'll let you know.
Quoted 3 lines
If it’s possible, could we get a packet trace around when this happens? This is pure speculation, but this might have something to do the resend path - a commit fails and all the requests on the commit list have to be resent.
Sure, once I can reproduce it reliably, then I'll try to do that.
Right, a bunch of DDing from
/dev/zeroover an ssh session triggers this very quickly. I've put a binary tcpdump here:http://www.willdeacon.ukfsn.org/bitbucket/oopsen/nfs/tcpdump.bin
You may want to filter out the ssh packets. The only `interesting' thing to me is a retransmission about half way through, but I don't know what I'm looking for.
The NFS server is 10.1.203.204 and the client is 10.1.203.24.
Quoted 8 lines
Quoted 1 line
Thanks! You are using NFSv2, so there are no commits - that rules out my hunch.
Wait a second - the whole point of this extra reference (that the
WARN_ON_ONCEis related to) is to handle the pass off to commit lists.Maybe I’m just doing the wrong thing here with NFSv2!
More soon.
Actually, can you see if this is reproducible with NFSv3?
Thanks,
-dros
Quoted 9 lines · Weston Andros Adamson
Quoted 6 lines · Weston Andros Adamson
Wait a second - the whole point of this extra reference (that the
WARN_ON_ONCEis related to) is to handle the pass off to commit lists.Maybe I’m just doing the wrong thing here with NFSv2!
More soon.
Actually, can you see if this is reproducible with NFSv3?
Just tried that, and I'm unable to trigger the problem with NFSv3.
Will
Hi Andros,
Quoted 5 lines · Will Deacon
Quoted 4 lines · Weston Andros Adamson
Quoted 3 lines · Weston Andros Adamson
Wait a second - the whole point of this extra reference (that the
WARN_ON_ONCEis related to) is to handle the pass off to commit lists.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Maybe I’m just doing the wrong thing here with NFSv2!
More soon.
Actually, can you see if this is reproducible with NFSv3?
Just tried that, and I'm unable to trigger the problem with NFSv3.
Any more ideas? I can run with NFSv3 instead, but it's a shame to leave v2 broken.
Will
Quoted 10 lines · Will Deacon
Hi Andros,
Quoted 7 lines · Will Deacon
Quoted 6 lines · Weston Andros Adamson
Quoted 5 lines · Weston Andros Adamson
Wait a second - the whole point of this extra reference (that the
WARN_ON_ONCEis related to) is to handle the pass off to commit lists.Maybe I’m just doing the wrong thing here with NFSv2!
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
More soon.
Actually, can you see if this is reproducible with NFSv3?
Just tried that, and I'm unable to trigger the problem with NFSv3.
Any more ideas? I can run with NFSv3 instead, but it's a shame to leave v2 broken.
Hi Will,
I do plan on looking at this more very soon, but since it’s v2 only and just warning (and not crashing anything) it didn’t jump to the top of my list. It is only printing the warnings, right? That’s no reason to switch things up ;)
It might be interesting to get some rpcdebug output around the time of the warning, but I’m not sure it will be useful.
Since I cannot reproduce this, my next step is to inspect the code with an NFSv2 with 64k pages setup in mind.
-dros
Hey Will,
I’ve been very busy, but I haven’t forgotten about your bug report!
I think the WARN_ON_ONCE is just wrong, there are cases where the PG_INODE_REF flag is legitimately not set. The flag is set so that sub requests can mimmic the parent request’s reference count.
Could you please run the reproducer again, unmount, then check the contents of /proc/fs/nfsfs/servers?
If it’s empty like example output below, then we know that requests are being properly dereferenced.
# cat /proc/fs/nfsfs/servers NV SERVER PORT USE HOSTNAME #
If there are any servers still listed (I’m assuming this is the only mount on your client for the test), then we have a problem.
-dros
Quoted 34 lines · Weston Andros Adamson
Quoted 10 lines · Will Deacon
Hi Andros,
Quoted 7 lines · Will Deacon
Quoted 6 lines · Weston Andros Adamson
Quoted 5 lines · Weston Andros Adamson
Wait a second - the whole point of this extra reference (that the
WARN_ON_ONCEis related to) is to handle the pass off to commit lists.Maybe I’m just doing the wrong thing here with NFSv2!
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
More soon.
Actually, can you see if this is reproducible with NFSv3?
Just tried that, and I'm unable to trigger the problem with NFSv3.
Any more ideas? I can run with NFSv3 instead, but it's a shame to leave v2 broken.
Hi Will,
I do plan on looking at this more very soon, but since it’s v2 only and just warning (and not crashing anything) it didn’t jump to the top of my list. It is only printing the warnings, right? That’s no reason to switch things up ;)
It might be interesting to get some rpcdebug output around the time of the warning, but I’m not sure it will be useful.
Since I cannot reproduce this, my next step is to inspect the code with an NFSv2 with 64k pages setup in mind.
-dros
Quoted 1 line · Weston Andros Adamson
Hey Will,
Hi again, Andros,
Quoted 1 line
I’ve been very busy, but I haven’t forgotten about your bug report!
No problem, I've been busy too. I just checked with -rc1 and I can reproduce the issue there too.
Quoted 6 lines
I think the
WARN_ON_ONCEis just wrong, there are cases where thePG_INODE_REFflag is legitimately not set. The flag is set so that sub requests can mimmic the parent request’s reference count.Could you please run the reproducer again, unmount, then check the contents of
/proc/fs/nfsfs/servers?
Sure. I just tried this and that file is empty after unmounting.
Will
Quoted 10 lines · Will Deacon
Quoted 1 line · Weston Andros Adamson
Hey Will,
Hi again, Andros,
Quoted 1 line
I’ve been very busy, but I haven’t forgotten about your bug report!
No problem, I've been busy too. I just checked with -rc1 and I can reproduce the issue there too.
Quoted 8 lines
Quoted 6 lines
I think the
WARN_ON_ONCEis just wrong, there are cases where thePG_INODE_REFflag is legitimately not set. The flag is set so that sub requests can mimmic the parent request’s reference count.Could you please run the reproducer again, unmount, then check the contents of
/proc/fs/nfsfs/servers?Sure. I just tried this and that file is empty after unmounting.
Sorry for the delay (again) - I just got back from a two week holiday.
This is good news! I think we can safely remove this WARN_ON_ONCE and call it a day… or maybe calling it a month is more accurate ;)
-dros
Quoted 14 lines · Weston Andros Adamson
Quoted 13 lines · Will Deacon
Quoted 1 line · Weston Andros Adamson
I’ve been very busy, but I haven’t forgotten about your bug report!
No problem, I've been busy too. I just checked with -rc1 and I can reproduce the issue there too.
Quoted 6 lines
I think the
WARN_ON_ONCEis just wrong, there are cases where thePG_INODE_REFflag is legitimately not set. The flag is set so that sub requests can mimmic the parent request’s reference count.Could you please run the reproducer again, unmount, then check the contents of
/proc/fs/nfsfs/servers?
Quoted 6 lines
Quoted 1 line
Sure. I just tried this and that file is empty after unmounting.
Sorry for the delay (again) - I just got back from a two week holiday.
This is good news! I think we can safely remove this
WARN_ON_ONCEand call it a day… or maybe calling it a month is more accurate ;)
Great! Feel free to add my reported-by/tested-by on that patch.
Will