Skip to contentSkip to conversation

fuse: mark DAX inode releases as blocking

PATCH Applied Sergio Lopez 4 participants 8 messages Jan 26 – Feb 17

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fuse: mark DAX inode releases as blocking

Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from fuseblk workers") made fputs on closing files always asynchronous.

As cleaning up DAX inodes may require issuing a number of synchronous request for releasing the mappings, completing the release request from the worker thread may lead to it hanging like this:

[   21.386751] Workqueue: events virtio_fs_requests_done_work
[   21.386769] Call trace:
[   21.386770]  __switch_to+0xe4/0x140
[   21.386780]  __schedule+0x294/0x72c
[   21.386787]  schedule+0x24/0x90
[   21.386794]  request_wait_answer+0x184/0x298
[   21.386799]  __fuse_simple_request+0x1f4/0x320
[   21.386805]  fuse_send_removemapping+0x80/0xa0
[   21.386810]  dmap_removemapping_list+0xac/0xfc
[   21.386814]  inode_reclaim_dmap_range.constprop.0+0xd0/0x204
[   21.386820]  fuse_dax_inode_cleanup+0x28/0x5c
[   21.386825]  fuse_evict_inode+0x120/0x190
[   21.386834]  evict+0x188/0x320
[   21.386847]  iput_final+0xb0/0x20c
[   21.386854]  iput+0xa0/0xbc
[   21.386862]  fuse_release_end+0x18/0x2c
[   21.386868]  fuse_request_end+0x9c/0x2c0
[   21.386872]  virtio_fs_request_complete+0x150/0x384
[   21.386879]  virtio_fs_requests_done_work+0x18c/0x37c
[   21.386885]  process_one_work+0x15c/0x2e8
[   21.386891]  worker_thread+0x278/0x480
[   21.386898]  kthread+0xd0/0xdc
[   21.386902]  ret_from_fork+0x10/0x20

Here, the virtio-fs worker_thread is waiting on request_wait_answer() for a reply from the virtio-fs server that is already in the virtqueue but will never be processed since it's that same worker thread the one in charge of consuming the elements from the virtqueue.

To address this issue, when relesing a DAX inode mark the operation as potentially blocking. Doing this will ensure these release requests are processed on a different worker thread.

---
 fs/fuse/file.c | 6 ++++++
 1 file changed, 6 insertions(+)
fs/fuse/file.c +6 -0
index 3b2a171e652f..a65c5d32a34b 100644
@@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)
 			fuse_simple_request(ff->fm, args);
 			fuse_release_end(ff->fm, args, 0);
 		} else {
+			/*
+			 * DAX inodes may need to issue a number of synchronous
+			 * request for clearing the mappings.
+			 */
+			if (ra && ra->inode && FUSE_IS_DAX(ra->inode))
+				args->may_block = true;
 			args->end = fuse_release_end;
 			if (fuse_simple_background(ff->fm, args,
 						   GFP_KERNEL | __GFP_NOFAIL))
Signature · 1 line
2.52.0
Reviewed-by on v1 1/1
#1 lore raw
Quoted 16 lines · Sergio Lopez

Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from fuseblk workers") made fputs on closing files always asynchronous.

As cleaning up DAX inodes may require issuing a number of synchronous request for releasing the mappings, completing the release request from the worker thread may lead to it hanging like this:

[   21.386751] Workqueue: events virtio_fs_requests_done_work
[   21.386769] Call trace:
[   21.386770]  __switch_to+0xe4/0x140
[   21.386780]  __schedule+0x294/0x72c
[   21.386787]  schedule+0x24/0x90
[   21.386794]  request_wait_answer+0x184/0x298
[   21.386799]  __fuse_simple_request+0x1f4/0x320
[   21.386805]  fuse_send_removemapping+0x80/0xa0
[   21.386810]  dmap_removemapping_list+0xac/0xfc
Quoted 8 lines
[   21.386814]  inode_reclaim_dmap_range.constprop.0+0xd0/0x204
[   21.386820]  fuse_dax_inode_cleanup+0x28/0x5c
[   21.386825]  fuse_evict_inode+0x120/0x190
[   21.386834]  evict+0x188/0x320
[   21.386847]  iput_final+0xb0/0x20c
[   21.386854]  iput+0xa0/0xbc
[   21.386862]  fuse_release_end+0x18/0x2c
[   21.386868]  fuse_request_end+0x9c/0x2c0

Ok, so this is the reply from the async FUSE_RELEASE command. But then we iput the inode, which results in fuse issuing a new synchronous command from within the completion for the first command.

Ouch.

Quoted 11 lines
[   21.386872]  virtio_fs_request_complete+0x150/0x384
[   21.386879]  virtio_fs_requests_done_work+0x18c/0x37c
[   21.386885]  process_one_work+0x15c/0x2e8
[   21.386891]  worker_thread+0x278/0x480
[   21.386898]  kthread+0xd0/0xdc
[   21.386902]  ret_from_fork+0x10/0x20

Here, the virtio-fs worker_thread is waiting on request_wait_answer() for a reply from the virtio-fs server that is already in the virtqueue but will never be processed since it's that same worker thread the one in charge of consuming the elements from the virtqueue.

Yes. Ow.

Quoted 3 lines

To address this issue, when relesing a DAX inode mark the operation as potentially blocking. Doing this will ensure these release requests are processed on a different worker thread.

I wonder if you've solved this problem report?
https://github.com/Nevuly/WSL2-Rolling-Kernel-Issue/issues/38

Naturally they reverted the patch, emailed me, and refused to talk about this on the public list, which is why nobody's heard of this until now.

Quoted 10 lines
---
 fs/fuse/file.c | 6 ++++++
 1 file changed, 6 insertions(+)
fs/fuse/file.c +0 -0
index 3b2a171e652f..a65c5d32a34b 100644

fs/fuse/file.c static void fuse_file_put(struct fuse_file *ff, bool sync) · line 125

Quoted 9 lines
fs/fuse/file.c +6 -0
@@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)
 			fuse_simple_request(ff->fm, args);
 			fuse_release_end(ff->fm, args, 0);
 		} else {
+			/*
+			 * DAX inodes may need to issue a number of synchronous
+			 * request for clearing the mappings.
+			 */
+			if (ra && ra->inode && FUSE_IS_DAX(ra->inode))
+				args->may_block = true;

There's no documentation for what may_block does, but there are so few uses of it that I can tell that this is kicking the FUSE_RELEASE completion to a workqueue instead of processing it directly, which eliminates the livelock.

I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.

Insofar as I understand anything in fuse, I think this is the right thing to do. Thanks for fixing this.

--D

Quoted 7 lines
fs/fuse/file.c +0 -0
 			args->end = fuse_release_end;
 			if (fuse_simple_background(ff->fm, args,
 						   GFP_KERNEL | __GFP_NOFAIL))
Signature · 1 line
2.52.0
#2 lore raw
Quoted 4 lines · Darrick J. Wong

I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.

I had also observed similar issue where the FUSE daemon thread is hanging in:

request_wait_answer
fuse_simple_request
fuse_flush_times
fuse write_inode
writeback_single_inode
write_inode_now
fuse_release
_fput

At that time I had no idea how FUSE daemon thread could trigger fuse file release and thus I didn't dive into this further...

I think commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from fuseblk workers") is not adequate in this case, as the commit only makes FUSE_RELEASE request asynchronously, while in this case the daemon thread can wait for FUSE_WRITE and FUSE_SETATTR.

Maybe the very first entry i.e. fuse_release() needs to be executed in an asynchronous context (e.g. workqueue)...

Signature · 2 lines
Thanks,
Jingbo
#3 lore raw
Quoted 22 lines · Jingbo Xu
Quoted 4 lines · Darrick J. Wong

I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.

I had also observed similar issue where the FUSE daemon thread is hanging in:

request_wait_answer
fuse_simple_request
fuse_flush_times
fuse write_inode
writeback_single_inode
write_inode_now
fuse_release
_fput

At that time I had no idea how FUSE daemon thread could trigger fuse file release and thus I didn't dive into this further...

Quoted 7 lines

I think commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from fuseblk workers") is not adequate in this case, as the commit only makes FUSE_RELEASE request asynchronously, while in this case the daemon thread can wait for FUSE_WRITE and FUSE_SETATTR.

Maybe the very first entry i.e. fuse_release() needs to be executed in an asynchronous context (e.g. workqueue)...

Do you have the rest of the stacktrace? We need to know if it's running in the context of the worker thread, as otherwise it must be a different problem.

Thanks,
Sergio.

#4 lore raw
Quoted 29 lines · Sergio Lopez Pascual
Quoted 27 lines · Jingbo Xu
Quoted 4 lines · Darrick J. Wong

I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.

I had also observed similar issue where the FUSE daemon thread is hanging in:

request_wait_answer
fuse_simple_request
fuse_flush_times
fuse write_inode
writeback_single_inode
write_inode_now
fuse_release
_fput

At that time I had no idea how FUSE daemon thread could trigger fuse file release and thus I didn't dive into this further...

I think commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from fuseblk workers") is not adequate in this case, as the commit only makes FUSE_RELEASE request asynchronously, while in this case the daemon thread can wait for FUSE_WRITE and FUSE_SETATTR.

Quoted 6 lines
Quoted 2 lines

Maybe the very first entry i.e. fuse_release() needs to be executed in an asynchronous context (e.g. workqueue)...

Do you have the rest of the stacktrace? We need to know if it's running in the context of the worker thread, as otherwise it must be a different problem.

This is observed in scenarios of FUSE rather than virtiofs.

The complete stacktrace of FUSE daemon worker thread (in userspace) is like:

request_wait_answer
fuse_simple_request
fuse_flush_times
fuse write_inode
writeback_single_inode
write_inode_now
fuse_release
__fput
____fput
task_work_run
exit_to_user_mode_prepare
syscall exit_to_user_mode
do_syscall_64
entry_SYSCALL_64_after_hwframe

Signature · 2 lines
Thanks,
Jingbo
#5 lore raw
In reply to Darrick J. Wong
Quoted 50 lines · Darrick J. Wong
Quoted 16 lines · Sergio Lopez

Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from fuseblk workers") made fputs on closing files always asynchronous.

As cleaning up DAX inodes may require issuing a number of synchronous request for releasing the mappings, completing the release request from the worker thread may lead to it hanging like this:

[   21.386751] Workqueue: events virtio_fs_requests_done_work
[   21.386769] Call trace:
[   21.386770]  __switch_to+0xe4/0x140
[   21.386780]  __schedule+0x294/0x72c
[   21.386787]  schedule+0x24/0x90
[   21.386794]  request_wait_answer+0x184/0x298
[   21.386799]  __fuse_simple_request+0x1f4/0x320
[   21.386805]  fuse_send_removemapping+0x80/0xa0
[   21.386810]  dmap_removemapping_list+0xac/0xfc
Quoted 8 lines
[   21.386814]  inode_reclaim_dmap_range.constprop.0+0xd0/0x204
[   21.386820]  fuse_dax_inode_cleanup+0x28/0x5c
[   21.386825]  fuse_evict_inode+0x120/0x190
[   21.386834]  evict+0x188/0x320
[   21.386847]  iput_final+0xb0/0x20c
[   21.386854]  iput+0xa0/0xbc
[   21.386862]  fuse_release_end+0x18/0x2c
[   21.386868]  fuse_request_end+0x9c/0x2c0

Ok, so this is the reply from the async FUSE_RELEASE command. But then we iput the inode, which results in fuse issuing a new synchronous command from within the completion for the first command.

Ouch.

Quoted 11 lines
[   21.386872]  virtio_fs_request_complete+0x150/0x384
[   21.386879]  virtio_fs_requests_done_work+0x18c/0x37c
[   21.386885]  process_one_work+0x15c/0x2e8
[   21.386891]  worker_thread+0x278/0x480
[   21.386898]  kthread+0xd0/0xdc
[   21.386902]  ret_from_fork+0x10/0x20

Here, the virtio-fs worker_thread is waiting on request_wait_answer() for a reply from the virtio-fs server that is already in the virtqueue but will never be processed since it's that same worker thread the one in charge of consuming the elements from the virtqueue.

Yes. Ow.

Quoted 3 lines

To address this issue, when relesing a DAX inode mark the operation as potentially blocking. Doing this will ensure these release requests are processed on a different worker thread.

Quoted 5 lines

I wonder if you've solved this problem report?
https://github.com/Nevuly/WSL2-Rolling-Kernel-Issue/issues/38

Naturally they reverted the patch, emailed me, and refused to talk about this on the public list, which is why nobody's heard of this until now.

If they're using DAX, that's very likely. I've discovered it while testing a new kernel with libkrun/muvm, which does use multiple virtio-fs devices, one of them as root and another one with DAX, which turns to be a pretty good testbed for virtio-fs (nothing better than running Steam, and bunch of games under FEX+Proton for stress-testing multiple subsystems with a real-world workload ;-).

Quoted 25 lines
Quoted 10 lines
---
 fs/fuse/file.c | 6 ++++++
 1 file changed, 6 insertions(+)
fs/fuse/file.c +0 -0
index 3b2a171e652f..a65c5d32a34b 100644

fs/fuse/file.c static void fuse_file_put(struct fuse_file *ff, bool sync) · line 125

Quoted 9 lines
fs/fuse/file.c +6 -0
@@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)
 			fuse_simple_request(ff->fm, args);
 			fuse_release_end(ff->fm, args, 0);
 		} else {
+			/*
+			 * DAX inodes may need to issue a number of synchronous
+			 * request for clearing the mappings.
+			 */
+			if (ra && ra->inode && FUSE_IS_DAX(ra->inode))
+				args->may_block = true;

There's no documentation for what may_block does, but there are so few uses of it that I can tell that this is kicking the FUSE_RELEASE completion to a workqueue instead of processing it directly, which eliminates the livelock.

Quoted 4 lines

I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.

Perhaps we could check for PF_WQ_WORKER and, at the very least, emit a warning.

Thanks,
Sergio.

#6 lore raw
Quoted 58 lines · Sergio Lopez Pascual
Quoted 50 lines · Darrick J. Wong
Quoted 16 lines · Sergio Lopez

Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from fuseblk workers") made fputs on closing files always asynchronous.

As cleaning up DAX inodes may require issuing a number of synchronous request for releasing the mappings, completing the release request from the worker thread may lead to it hanging like this:

[   21.386751] Workqueue: events virtio_fs_requests_done_work
[   21.386769] Call trace:
[   21.386770]  __switch_to+0xe4/0x140
[   21.386780]  __schedule+0x294/0x72c
[   21.386787]  schedule+0x24/0x90
[   21.386794]  request_wait_answer+0x184/0x298
[   21.386799]  __fuse_simple_request+0x1f4/0x320
[   21.386805]  fuse_send_removemapping+0x80/0xa0
[   21.386810]  dmap_removemapping_list+0xac/0xfc
Quoted 8 lines
[   21.386814]  inode_reclaim_dmap_range.constprop.0+0xd0/0x204
[   21.386820]  fuse_dax_inode_cleanup+0x28/0x5c
[   21.386825]  fuse_evict_inode+0x120/0x190
[   21.386834]  evict+0x188/0x320
[   21.386847]  iput_final+0xb0/0x20c
[   21.386854]  iput+0xa0/0xbc
[   21.386862]  fuse_release_end+0x18/0x2c
[   21.386868]  fuse_request_end+0x9c/0x2c0

Ok, so this is the reply from the async FUSE_RELEASE command. But then we iput the inode, which results in fuse issuing a new synchronous command from within the completion for the first command.

Ouch.

Quoted 11 lines
[   21.386872]  virtio_fs_request_complete+0x150/0x384
[   21.386879]  virtio_fs_requests_done_work+0x18c/0x37c
[   21.386885]  process_one_work+0x15c/0x2e8
[   21.386891]  worker_thread+0x278/0x480
[   21.386898]  kthread+0xd0/0xdc
[   21.386902]  ret_from_fork+0x10/0x20

Here, the virtio-fs worker_thread is waiting on request_wait_answer() for a reply from the virtio-fs server that is already in the virtqueue but will never be processed since it's that same worker thread the one in charge of consuming the elements from the virtqueue.

Yes. Ow.

Quoted 3 lines

To address this issue, when relesing a DAX inode mark the operation as potentially blocking. Doing this will ensure these release requests are processed on a different worker thread.

Quoted 5 lines

I wonder if you've solved this problem report?
https://github.com/Nevuly/WSL2-Rolling-Kernel-Issue/issues/38

Naturally they reverted the patch, emailed me, and refused to talk about this on the public list, which is why nobody's heard of this until now.

Quoted 6 lines

If they're using DAX, that's very likely. I've discovered it while testing a new kernel with libkrun/muvm, which does use multiple virtio-fs devices, one of them as root and another one with DAX, which turns to be a pretty good testbed for virtio-fs (nothing better than running Steam, and bunch of games under FEX+Proton for stress-testing multiple subsystems with a real-world workload ;-).

Hehe. :)

Quoted 25 lines
Quoted 25 lines
Quoted 10 lines
---
 fs/fuse/file.c | 6 ++++++
 1 file changed, 6 insertions(+)
fs/fuse/file.c +0 -0
index 3b2a171e652f..a65c5d32a34b 100644

fs/fuse/file.c static void fuse_file_put(struct fuse_file *ff, bool sync) · line 125

Quoted 9 lines
fs/fuse/file.c +6 -0
@@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)
 			fuse_simple_request(ff->fm, args);
 			fuse_release_end(ff->fm, args, 0);
 		} else {
+			/*
+			 * DAX inodes may need to issue a number of synchronous
+			 * request for clearing the mappings.
+			 */
+			if (ra && ra->inode && FUSE_IS_DAX(ra->inode))
+				args->may_block = true;

There's no documentation for what may_block does, but there are so few uses of it that I can tell that this is kicking the FUSE_RELEASE completion to a workqueue instead of processing it directly, which eliminates the livelock.

Quoted 7 lines
Quoted 4 lines

I wonder if fuse ought to grow the ability to whine when something is trying to issue a synchronous fuse command while running in a command queue completion context (aka the worker threads) but I don't know how difficult that would *really* be.

Perhaps we could check for PF_WQ_WORKER and, at the very least, emit a warning.

I don't think that will work:

PF_WQ_WORKER only tells us if the current process is a kernel workqueue. Replies to fuse command are written to the fuse device, which means that we're running in the process context of the fuse server when we get to fuse_dev_do_write -> fuse_request_end -> req->args->end -> fuse_release_end.

--D

Quoted 3 lines

Thanks,
Sergio.

maintainer Applied
#7 lore raw
In reply to Sergio Lopez
Quoted 37 lines · Sergio Lopez

Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from fuseblk workers") made fputs on closing files always asynchronous.

As cleaning up DAX inodes may require issuing a number of synchronous request for releasing the mappings, completing the release request from the worker thread may lead to it hanging like this:

[   21.386751] Workqueue: events virtio_fs_requests_done_work
[   21.386769] Call trace:
[   21.386770]  __switch_to+0xe4/0x140
[   21.386780]  __schedule+0x294/0x72c
[   21.386787]  schedule+0x24/0x90
[   21.386794]  request_wait_answer+0x184/0x298
[   21.386799]  __fuse_simple_request+0x1f4/0x320
[   21.386805]  fuse_send_removemapping+0x80/0xa0
[   21.386810]  dmap_removemapping_list+0xac/0xfc
[   21.386814]  inode_reclaim_dmap_range.constprop.0+0xd0/0x204
[   21.386820]  fuse_dax_inode_cleanup+0x28/0x5c
[   21.386825]  fuse_evict_inode+0x120/0x190
[   21.386834]  evict+0x188/0x320
[   21.386847]  iput_final+0xb0/0x20c
[   21.386854]  iput+0xa0/0xbc
[   21.386862]  fuse_release_end+0x18/0x2c
[   21.386868]  fuse_request_end+0x9c/0x2c0
[   21.386872]  virtio_fs_request_complete+0x150/0x384
[   21.386879]  virtio_fs_requests_done_work+0x18c/0x37c
[   21.386885]  process_one_work+0x15c/0x2e8
[   21.386891]  worker_thread+0x278/0x480
[   21.386898]  kthread+0xd0/0xdc
[   21.386902]  ret_from_fork+0x10/0x20

Here, the virtio-fs worker_thread is waiting on request_wait_answer() for a reply from the virtio-fs server that is already in the virtqueue but will never be processed since it's that same worker thread the one in charge of consuming the elements from the virtqueue.

Quoted 5 lines

To address this issue, when relesing a DAX inode mark the operation as potentially blocking. Doing this will ensure these release requests are processed on a different worker thread.

Applied, thanks.

Miklos