fuse: mark DAX inode releases as blocking
Subsystems (2)
PATCH 1/1 fuse: mark DAX inode releases as blocking
Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from fuseblk workers") made fputs on closing files always asynchronous.
As cleaning up DAX inodes may require issuing a number of synchronous request for releasing the mappings, completing the release request from the worker thread may lead to it hanging like this:
[ 21.386751] Workqueue: events virtio_fs_requests_done_work [ 21.386769] Call trace: [ 21.386770] __switch_to+0xe4/0x140 [ 21.386780] __schedule+0x294/0x72c [ 21.386787] schedule+0x24/0x90 [ 21.386794] request_wait_answer+0x184/0x298 [ 21.386799] __fuse_simple_request+0x1f4/0x320 [ 21.386805] fuse_send_removemapping+0x80/0xa0 [ 21.386810] dmap_removemapping_list+0xac/0xfc [ 21.386814] inode_reclaim_dmap_range.constprop.0+0xd0/0x204 [ 21.386820] fuse_dax_inode_cleanup+0x28/0x5c [ 21.386825] fuse_evict_inode+0x120/0x190 [ 21.386834] evict+0x188/0x320 [ 21.386847] iput_final+0xb0/0x20c [ 21.386854] iput+0xa0/0xbc [ 21.386862] fuse_release_end+0x18/0x2c [ 21.386868] fuse_request_end+0x9c/0x2c0 [ 21.386872] virtio_fs_request_complete+0x150/0x384 [ 21.386879] virtio_fs_requests_done_work+0x18c/0x37c [ 21.386885] process_one_work+0x15c/0x2e8 [ 21.386891] worker_thread+0x278/0x480 [ 21.386898] kthread+0xd0/0xdc [ 21.386902] ret_from_fork+0x10/0x20
Here, the virtio-fs worker_thread is waiting on request_wait_answer() for a reply from the virtio-fs server that is already in the virtqueue but will never be processed since it's that same worker thread the one in charge of consuming the elements from the virtqueue.
To address this issue, when relesing a DAX inode mark the operation as potentially blocking. Doing this will ensure these release requests are processed on a different worker thread.
--- fs/fuse/file.c | 6 ++++++ 1 file changed, 6 insertions(+)
fs/fuse/file.c +6 -0
@@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)fuse_simple_request (ff->fm, args);fuse_release_end (ff->fm, args,0 ); }else { +/* +* DAX inodes may need to issue a number of synchronous +* request for clearing the mappings. +*/ +if (ra && ra->inode &&FUSE_IS_DAX (ra->inode)) + args->may_block = true; args->end = fuse_release_end;if (fuse_simple_background (ff->fm, args,GFP_KERNEL | __GFP_NOFAIL))
Signature · 1 line
2.52.0
Quoted 16 lines · Sergio Lopez
Commit
26e5c67deb2e("fuse: fix livelock in synchronous file put from fuseblk workers") made fputs on closing files always asynchronous.As cleaning up DAX inodes may require issuing a number of synchronous request for releasing the mappings, completing the release request from the worker thread may lead to it hanging like this:
[ 21.386751] Workqueue: events virtio_fs_requests_done_work [ 21.386769] Call trace: [ 21.386770] __switch_to+0xe4/0x140 [ 21.386780] __schedule+0x294/0x72c [ 21.386787] schedule+0x24/0x90 [ 21.386794] request_wait_answer+0x184/0x298 [ 21.386799] __fuse_simple_request+0x1f4/0x320 [ 21.386805] fuse_send_removemapping+0x80/0xa0 [ 21.386810] dmap_removemapping_list+0xac/0xfc
Quoted 8 lines
[ 21.386814] inode_reclaim_dmap_range.constprop.0+0xd0/0x204 [ 21.386820] fuse_dax_inode_cleanup+0x28/0x5c [ 21.386825] fuse_evict_inode+0x120/0x190 [ 21.386834] evict+0x188/0x320 [ 21.386847] iput_final+0xb0/0x20c [ 21.386854] iput+0xa0/0xbc [ 21.386862] fuse_release_end+0x18/0x2c [ 21.386868] fuse_request_end+0x9c/0x2c0
Ok, so this is the reply from the async FUSE_RELEASE command. But then we iput the inode, which results in fuse issuing a new synchronous command from within the completion for the first command.
Ouch.
Quoted 11 lines
[ 21.386872] virtio_fs_request_complete+0x150/0x384 [ 21.386879] virtio_fs_requests_done_work+0x18c/0x37c [ 21.386885] process_one_work+0x15c/0x2e8 [ 21.386891] worker_thread+0x278/0x480 [ 21.386898] kthread+0xd0/0xdc [ 21.386902] ret_from_fork+0x10/0x20Here, the virtio-fs
worker_threadis waiting onrequest_wait_answer()for a reply from the virtio-fs server that is already in the virtqueue but will never be processed since it's that same worker thread the one in charge of consuming the elements from the virtqueue.
Yes. Ow.
Quoted 3 lines
To address this issue, when relesing a DAX inode mark the operation as potentially blocking. Doing this will ensure these release requests are processed on a different worker thread.
I wonder if you've solved this problem report?
https://github.com/Nevuly/WSL2-Rolling-Kernel-Issue/issues/38
Naturally they reverted the patch, emailed me, and refused to talk about this on the public list, which is why nobody's heard of this until now.
Quoted 10 lines
Signed-off-by Sergio Lopez <[email protected]>--- fs/fuse/file.c | 6 ++++++ 1 file changed, 6 insertions(+)fs/fuse/file.c +0
-0
fs/fuse/file.c static void fuse_file_put(struct fuse_file *ff, bool sync) · line 125
Quoted 9 lines
fs/fuse/file.c +6
-0@@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)fuse_simple_request (ff->fm, args);fuse_release_end (ff->fm, args,0 ); }else { +/* +* DAX inodes may need to issue a number of synchronous +* request for clearing the mappings. +*/ +if (ra && ra->inode &&FUSE_IS_DAX (ra->inode)) + args->may_block = true;
There's no documentation for what may_block does, but there are so few uses of it that I can tell that this is kicking the FUSE_RELEASE completion to a workqueue instead of processing it directly, which eliminates the livelock.
I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.
Insofar as I understand anything in fuse, I think this is the right thing to do. Thanks for fixing this.
--D
Quoted 7 lines
fs/fuse/file.c +0
-0args->end = fuse_release_end;if (fuse_simple_background (ff->fm, args,GFP_KERNEL | __GFP_NOFAIL))Signature · 1 line
2.52.0
Quoted 4 lines · Darrick J. Wong
I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.
I had also observed similar issue where the FUSE daemon thread is hanging in:
request_wait_answerfuse_simple_requestfuse_flush_times
fuse write_inodewriteback_single_inodewrite_inode_nowfuse_release_fput
At that time I had no idea how FUSE daemon thread could trigger fuse file release and thus I didn't dive into this further...
I think commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from fuseblk workers") is not adequate in this case, as the commit only makes FUSE_RELEASE request asynchronously, while in this case the daemon thread can wait for FUSE_WRITE and FUSE_SETATTR.
Maybe the very first entry i.e. fuse_release() needs to be executed in an asynchronous context (e.g. workqueue)...
Signature · 2 lines
Thanks, Jingbo
Quoted 22 lines · Jingbo Xu
Quoted 4 lines · Darrick J. Wong
I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.
I had also observed similar issue where the FUSE daemon thread is hanging in:
request_wait_answerfuse_simple_requestfuse_flush_times
fusewrite_inodewriteback_single_inodewrite_inode_nowfuse_release_fputAt that time I had no idea how FUSE daemon thread could trigger fuse file release and thus I didn't dive into this further...
Quoted 7 lines
I think commit
26e5c67deb2e("fuse: fix livelock in synchronous file put from fuseblk workers") is not adequate in this case, as the commit only makesFUSE_RELEASErequest asynchronously, while in this case the daemon thread can wait forFUSE_WRITEandFUSE_SETATTR.Maybe the very first entry i.e.
fuse_release()needs to be executed in an asynchronous context (e.g. workqueue)...
Do you have the rest of the stacktrace? We need to know if it's running in the context of the worker thread, as otherwise it must be a different problem.
Thanks,
Sergio.
Quoted 29 lines · Sergio Lopez Pascual
Quoted 27 lines · Jingbo Xu
Quoted 4 lines · Darrick J. Wong
I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.
I had also observed similar issue where the FUSE daemon thread is hanging in:
request_wait_answerfuse_simple_requestfuse_flush_times
fusewrite_inodewriteback_single_inodewrite_inode_nowfuse_release_fputAt that time I had no idea how FUSE daemon thread could trigger fuse file release and thus I didn't dive into this further...
I think commit
26e5c67deb2e("fuse: fix livelock in synchronous file put from fuseblk workers") is not adequate in this case, as the commit only makesFUSE_RELEASErequest asynchronously, while in this case the daemon thread can wait forFUSE_WRITEandFUSE_SETATTR.
Quoted 6 lines
Quoted 2 lines
Maybe the very first entry i.e.
fuse_release()needs to be executed in an asynchronous context (e.g. workqueue)...Do you have the rest of the stacktrace? We need to know if it's running in the context of the worker thread, as otherwise it must be a different problem.
This is observed in scenarios of FUSE rather than virtiofs.
The complete stacktrace of FUSE daemon worker thread (in userspace) is like:
request_wait_answer
fuse_simple_request
fuse_flush_times
fuse write_inode
writeback_single_inode
write_inode_now
fuse_release
__fput
____fput
task_work_run
exit_to_user_mode_prepare
syscall exit_to_user_mode
do_syscall_64
entry_SYSCALL_64_after_hwframe
Signature · 2 lines
Thanks, Jingbo
Quoted 50 lines · Darrick J. Wong
Quoted 16 lines · Sergio Lopez
Commit
26e5c67deb2e("fuse: fix livelock in synchronous file put from fuseblk workers") made fputs on closing files always asynchronous.As cleaning up DAX inodes may require issuing a number of synchronous request for releasing the mappings, completing the release request from the worker thread may lead to it hanging like this:
[ 21.386751] Workqueue: events virtio_fs_requests_done_work [ 21.386769] Call trace: [ 21.386770] __switch_to+0xe4/0x140 [ 21.386780] __schedule+0x294/0x72c [ 21.386787] schedule+0x24/0x90 [ 21.386794] request_wait_answer+0x184/0x298 [ 21.386799] __fuse_simple_request+0x1f4/0x320 [ 21.386805] fuse_send_removemapping+0x80/0xa0 [ 21.386810] dmap_removemapping_list+0xac/0xfcQuoted 8 lines
[ 21.386814] inode_reclaim_dmap_range.constprop.0+0xd0/0x204 [ 21.386820] fuse_dax_inode_cleanup+0x28/0x5c [ 21.386825] fuse_evict_inode+0x120/0x190 [ 21.386834] evict+0x188/0x320 [ 21.386847] iput_final+0xb0/0x20c [ 21.386854] iput+0xa0/0xbc [ 21.386862] fuse_release_end+0x18/0x2c [ 21.386868] fuse_request_end+0x9c/0x2c0Ok, so this is the reply from the async
FUSE_RELEASEcommand. But then we iput the inode, which results in fuse issuing a new synchronous command from within the completion for the first command.Ouch.
Quoted 11 lines
[ 21.386872] virtio_fs_request_complete+0x150/0x384 [ 21.386879] virtio_fs_requests_done_work+0x18c/0x37c [ 21.386885] process_one_work+0x15c/0x2e8 [ 21.386891] worker_thread+0x278/0x480 [ 21.386898] kthread+0xd0/0xdc [ 21.386902] ret_from_fork+0x10/0x20Here, the virtio-fs
worker_threadis waiting onrequest_wait_answer()for a reply from the virtio-fs server that is already in the virtqueue but will never be processed since it's that same worker thread the one in charge of consuming the elements from the virtqueue.Yes. Ow.
Quoted 3 lines
To address this issue, when relesing a DAX inode mark the operation as potentially blocking. Doing this will ensure these release requests are processed on a different worker thread.
Quoted 5 lines
I wonder if you've solved this problem report?
https://github.com/Nevuly/WSL2-Rolling-Kernel-Issue/issues/38Naturally they reverted the patch, emailed me, and refused to talk about this on the public list, which is why nobody's heard of this until now.
If they're using DAX, that's very likely. I've discovered it while testing a new kernel with libkrun/muvm, which does use multiple virtio-fs devices, one of them as root and another one with DAX, which turns to be a pretty good testbed for virtio-fs (nothing better than running Steam, and bunch of games under FEX+Proton for stress-testing multiple subsystems with a real-world workload ;-).
Quoted 25 lines
Quoted 10 lines
Signed-off-by Sergio Lopez <[email protected]>--- fs/fuse/file.c | 6 ++++++ 1 file changed, 6 insertions(+)fs/fuse/file.c +0
-0fs/fuse/file.c static void fuse_file_put(struct fuse_file *ff, bool sync) · line 125
Quoted 9 lines
fs/fuse/file.c +6
-0@@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)fuse_simple_request (ff->fm, args);fuse_release_end (ff->fm, args,0 ); }else { +/* +* DAX inodes may need to issue a number of synchronous +* request for clearing the mappings. +*/ +if (ra && ra->inode &&FUSE_IS_DAX (ra->inode)) + args->may_block = true;There's no documentation for what
may_blockdoes, but there are so few uses of it that I can tell that this is kicking theFUSE_RELEASEcompletion to a workqueue instead of processing it directly, which eliminates the livelock.
Quoted 4 lines
I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.
Perhaps we could check for PF_WQ_WORKER and, at the very least, emit a warning.
Thanks,
Sergio.
Quoted 58 lines · Sergio Lopez Pascual
Quoted 50 lines · Darrick J. Wong
Quoted 16 lines · Sergio Lopez
Commit
26e5c67deb2e("fuse: fix livelock in synchronous file put from fuseblk workers") made fputs on closing files always asynchronous.As cleaning up DAX inodes may require issuing a number of synchronous request for releasing the mappings, completing the release request from the worker thread may lead to it hanging like this:
[ 21.386751] Workqueue: events virtio_fs_requests_done_work [ 21.386769] Call trace: [ 21.386770] __switch_to+0xe4/0x140 [ 21.386780] __schedule+0x294/0x72c [ 21.386787] schedule+0x24/0x90 [ 21.386794] request_wait_answer+0x184/0x298 [ 21.386799] __fuse_simple_request+0x1f4/0x320 [ 21.386805] fuse_send_removemapping+0x80/0xa0 [ 21.386810] dmap_removemapping_list+0xac/0xfcQuoted 8 lines
[ 21.386814] inode_reclaim_dmap_range.constprop.0+0xd0/0x204 [ 21.386820] fuse_dax_inode_cleanup+0x28/0x5c [ 21.386825] fuse_evict_inode+0x120/0x190 [ 21.386834] evict+0x188/0x320 [ 21.386847] iput_final+0xb0/0x20c [ 21.386854] iput+0xa0/0xbc [ 21.386862] fuse_release_end+0x18/0x2c [ 21.386868] fuse_request_end+0x9c/0x2c0Ok, so this is the reply from the async
FUSE_RELEASEcommand. But then we iput the inode, which results in fuse issuing a new synchronous command from within the completion for the first command.Ouch.
Quoted 11 lines
[ 21.386872] virtio_fs_request_complete+0x150/0x384 [ 21.386879] virtio_fs_requests_done_work+0x18c/0x37c [ 21.386885] process_one_work+0x15c/0x2e8 [ 21.386891] worker_thread+0x278/0x480 [ 21.386898] kthread+0xd0/0xdc [ 21.386902] ret_from_fork+0x10/0x20Here, the virtio-fs
worker_threadis waiting onrequest_wait_answer()for a reply from the virtio-fs server that is already in the virtqueue but will never be processed since it's that same worker thread the one in charge of consuming the elements from the virtqueue.Yes. Ow.
Quoted 3 lines
To address this issue, when relesing a DAX inode mark the operation as potentially blocking. Doing this will ensure these release requests are processed on a different worker thread.
Quoted 5 lines
I wonder if you've solved this problem report?
https://github.com/Nevuly/WSL2-Rolling-Kernel-Issue/issues/38Naturally they reverted the patch, emailed me, and refused to talk about this on the public list, which is why nobody's heard of this until now.
Quoted 6 lines
If they're using DAX, that's very likely. I've discovered it while testing a new kernel with libkrun/muvm, which does use multiple virtio-fs devices, one of them as root and another one with DAX, which turns to be a pretty good testbed for virtio-fs (nothing better than running Steam, and bunch of games under FEX+Proton for stress-testing multiple subsystems with a real-world workload ;-).
Hehe. :)
Quoted 25 lines
Quoted 25 lines
Quoted 10 lines
Signed-off-by Sergio Lopez <[email protected]>--- fs/fuse/file.c | 6 ++++++ 1 file changed, 6 insertions(+)fs/fuse/file.c +0
-0fs/fuse/file.c static void fuse_file_put(struct fuse_file *ff, bool sync) · line 125
Quoted 9 lines
fs/fuse/file.c +6
-0@@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)fuse_simple_request (ff->fm, args);fuse_release_end (ff->fm, args,0 ); }else { +/* +* DAX inodes may need to issue a number of synchronous +* request for clearing the mappings. +*/ +if (ra && ra->inode &&FUSE_IS_DAX (ra->inode)) + args->may_block = true;There's no documentation for what
may_blockdoes, but there are so few uses of it that I can tell that this is kicking theFUSE_RELEASEcompletion to a workqueue instead of processing it directly, which eliminates the livelock.
Quoted 7 lines
Quoted 4 lines
I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.
Perhaps we could check for
PF_WQ_WORKERand, at the very least, emit a warning.
I don't think that will work:
PF_WQ_WORKER only tells us if the current process is a kernel workqueue. Replies to fuse command are written to the fuse device, which means that we're running in the process context of the fuse server when we get to fuse_dev_do_write -> fuse_request_end -> req->args->end -> fuse_release_end.
--D
Quoted 3 lines
Thanks,
Sergio.
Quoted 37 lines · Sergio Lopez
Commit
26e5c67deb2e("fuse: fix livelock in synchronous file put from fuseblk workers") made fputs on closing files always asynchronous.As cleaning up DAX inodes may require issuing a number of synchronous request for releasing the mappings, completing the release request from the worker thread may lead to it hanging like this:
[ 21.386751] Workqueue: events virtio_fs_requests_done_work [ 21.386769] Call trace: [ 21.386770] __switch_to+0xe4/0x140 [ 21.386780] __schedule+0x294/0x72c [ 21.386787] schedule+0x24/0x90 [ 21.386794] request_wait_answer+0x184/0x298 [ 21.386799] __fuse_simple_request+0x1f4/0x320 [ 21.386805] fuse_send_removemapping+0x80/0xa0 [ 21.386810] dmap_removemapping_list+0xac/0xfc [ 21.386814] inode_reclaim_dmap_range.constprop.0+0xd0/0x204 [ 21.386820] fuse_dax_inode_cleanup+0x28/0x5c [ 21.386825] fuse_evict_inode+0x120/0x190 [ 21.386834] evict+0x188/0x320 [ 21.386847] iput_final+0xb0/0x20c [ 21.386854] iput+0xa0/0xbc [ 21.386862] fuse_release_end+0x18/0x2c [ 21.386868] fuse_request_end+0x9c/0x2c0 [ 21.386872] virtio_fs_request_complete+0x150/0x384 [ 21.386879] virtio_fs_requests_done_work+0x18c/0x37c [ 21.386885] process_one_work+0x15c/0x2e8 [ 21.386891] worker_thread+0x278/0x480 [ 21.386898] kthread+0xd0/0xdc [ 21.386902] ret_from_fork+0x10/0x20Here, the virtio-fs
worker_threadis waiting onrequest_wait_answer()for a reply from the virtio-fs server that is already in the virtqueue but will never be processed since it's that same worker thread the one in charge of consuming the elements from the virtqueue.
Quoted 5 lines
To address this issue, when relesing a DAX inode mark the operation as potentially blocking. Doing this will ensure these release requests are processed on a different worker thread.
Signed-off-by Sergio Lopez <[email protected]>
Applied, thanks.
Miklos