fuse: avoid unnecessary spinlock bump
Subsystems (3)
PATCH 1/1 fuse: avoid unnecessary spinlock bump
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
--- fs/fuse/dax.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/fuse/dax.c +1 -1
@@ -138,9 +138,9 @@ static struct fuse_dax_mapping *alloc_dax_mapping(struct fuse_conn_dax *fcd)WARN_ON (fcd->nr_free_ranges <=0 ); fcd->nr_free_ranges--; } +__kick_dmap_free_worker (fcd,0 );spin_unlock (&fcd->lock); -kick_dmap_free_worker (fcd,0 );return dmap; }
Signature · 1 line
2.27.0
Quoted 8 lines · Jeffle Xu
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
Suggested-by Liu Jiang <[email protected]>Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 5 lines · Jeffle Xu
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
Suggested-by Liu Jiang <[email protected]>Signed-off-by Jeffle Xu <[email protected]>
Looks good to me. Have you done any testing to make sure nothing is broken.
Vivek
Quoted 22 lines
--- fs/fuse/dax.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)fs/fuse/dax.c +1
-1@@ -138,9 +138,9 @@ static struct fuse_dax_mapping *alloc_dax_mapping(struct fuse_conn_dax *fcd)WARN_ON (fcd->nr_free_ranges <=0 ); fcd->nr_free_ranges--; } +__kick_dmap_free_worker (fcd,0 );spin_unlock (&fcd->lock); -kick_dmap_free_worker (fcd,0 );return dmap; }Signature · 1 line
2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 9 lines · Vivek Goyal
Quoted 5 lines · Jeffle Xu
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
Suggested-by Liu Jiang <[email protected]>Signed-off-by Jeffle Xu <[email protected]>Looks good to me. Have you done any testing to make sure nothing is broken.
xfstests -g quick shows no regression. The tested virtiofs is mounted with "dax=always".
Signature · 2 lines
Thanks, Jeffle
Quoted 7 lines · JeffleXu
Quoted 4 lines · Vivek Goyal
Quoted 3 lines · Jeffle Xu
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Suggested-by Liu Jiang <[email protected]>Signed-off-by Jeffle Xu <[email protected]>Looks good to me. Have you done any testing to make sure nothing is broken.
xfstests -g quick shows no regression. The tested virtiofs is mounted with "dax=always".
I think xfstests might not trigger reclaim. You probably will have to run something like blogbench with a small dax window like 1G so that heavy reclaim happens.
For fun, I sometimes used to run it with a window of just say 16 dax ranges so that reclaim was so heavy that if there was a bug, it will show up.
Thanks
Vivek
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 14 lines · Vivek Goyal
Quoted 13 lines · JeffleXu
Quoted 9 lines · Vivek Goyal
Quoted 5 lines · Jeffle Xu
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
Suggested-by Liu Jiang <[email protected]>Signed-off-by Jeffle Xu <[email protected]>Looks good to me. Have you done any testing to make sure nothing is broken.
Quoted 6 lines
Quoted 2 lines
xfstests -g quick shows no regression. The tested virtiofs is mounted with "dax=always".
I think xfstests might not trigger reclaim. You probably will have to run something like blogbench with a small dax window like 1G so that heavy reclaim happens.
Actually, I configured the DAX window to 8MB, i.e. 4 slots when running xfstests. Thus I think the reclaim path is most likely triggered.
Quoted 5 lines
For fun, I sometimes used to run it with a window of just say 16 dax ranges so that reclaim was so heavy that if there was a bug, it will show up.
Yeah, my colleague had ever reported that a DAX window of 4KB will cause hang in our internal OS (which is 4.19, we back ported virtiofs to 4.19). But then I found that this issue doesn't exist in the latest upstream. The reason seems that in the upstream kernel, devm_memremap_pages() called in virtio_fs_setup_dax() will fail directly since the dax window (4KB) is not aligned with the sparse memory section.
Signature · 2 lines
Thanks, Jeffle
Quoted 35 lines · JeffleXu
Quoted 14 lines · Vivek Goyal
Quoted 13 lines · JeffleXu
Quoted 9 lines · Vivek Goyal
Quoted 5 lines · Jeffle Xu
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
Suggested-by Liu Jiang <[email protected]>Signed-off-by Jeffle Xu <[email protected]>Looks good to me. Have you done any testing to make sure nothing is broken.
Quoted 6 lines
Quoted 2 lines
xfstests -g quick shows no regression. The tested virtiofs is mounted with "dax=always".
I think xfstests might not trigger reclaim. You probably will have to run something like blogbench with a small dax window like 1G so that heavy reclaim happens.
Actually, I configured the DAX window to 8MB, i.e. 4 slots when running xfstests. Thus I think the reclaim path is most likely triggered.
Quoted 5 lines
For fun, I sometimes used to run it with a window of just say 16 dax ranges so that reclaim was so heavy that if there was a bug, it will show up.
Quoted 6 lines
Yeah, my colleague had ever reported that a DAX window of 4KB will cause hang in our internal OS (which is 4.19, we back ported virtiofs to 4.19). But then I found that this issue doesn't exist in the latest upstream. The reason seems that in the upstream kernel,
devm_memremap_pages()called invirtio_fs_setup_dax()will fail directly since the dax window (4KB) is not aligned with the sparse memory section.
Given our default chunk size is 2MB (FUSE_DAX_SHIFT), may be it is not a bad idea to enforce some minimum cache window size. IIRC, even one range is not enough. Minimum 2 are required for reclaim to not deadlock.
Hence, I guess it is not a bad idea to check for cache window size and if it is too small, reject it and disable dax.
Thanks
Vivek
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 43 lines · Vivek Goyal
Quoted 35 lines · JeffleXu
Quoted 14 lines · Vivek Goyal
Quoted 13 lines · JeffleXu
Quoted 9 lines · Vivek Goyal
Quoted 5 lines · Jeffle Xu
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
Suggested-by Liu Jiang <[email protected]>Signed-off-by Jeffle Xu <[email protected]>Looks good to me. Have you done any testing to make sure nothing is broken.
Quoted 6 lines
Quoted 2 lines
xfstests -g quick shows no regression. The tested virtiofs is mounted with "dax=always".
I think xfstests might not trigger reclaim. You probably will have to run something like blogbench with a small dax window like 1G so that heavy reclaim happens.
Actually, I configured the DAX window to 8MB, i.e. 4 slots when running xfstests. Thus I think the reclaim path is most likely triggered.
Quoted 5 lines
For fun, I sometimes used to run it with a window of just say 16 dax ranges so that reclaim was so heavy that if there was a bug, it will show up.
Quoted 6 lines
Yeah, my colleague had ever reported that a DAX window of 4KB will cause hang in our internal OS (which is 4.19, we back ported virtiofs to 4.19). But then I found that this issue doesn't exist in the latest upstream. The reason seems that in the upstream kernel,
devm_memremap_pages()called invirtio_fs_setup_dax()will fail directly since the dax window (4KB) is not aligned with the sparse memory section.
Quoted 3 lines
Given our default chunk size is 2MB (
FUSE_DAX_SHIFT), may be it is not a bad idea to enforce some minimum cache window size. IIRC, even one range is not enough. Minimum 2 are required for reclaim to not deadlock.
Curiously, why minimum 1 range is not adequate? In which case minimum 2 are required?
Signature · 2 lines
Thanks, Jeffle
Quoted 46 lines · JeffleXu
Quoted 43 lines · Vivek Goyal
Quoted 41 lines · JeffleXu
Quoted 14 lines · Vivek Goyal
Quoted 13 lines · JeffleXu
Quoted 9 lines · Vivek Goyal
Quoted 5 lines · Jeffle Xu
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
Suggested-by Liu Jiang <[email protected]>Signed-off-by Jeffle Xu <[email protected]>Looks good to me. Have you done any testing to make sure nothing is broken.
Quoted 6 lines
Quoted 2 lines
xfstests -g quick shows no regression. The tested virtiofs is mounted with "dax=always".
I think xfstests might not trigger reclaim. You probably will have to run something like blogbench with a small dax window like 1G so that heavy reclaim happens.
Actually, I configured the DAX window to 8MB, i.e. 4 slots when running xfstests. Thus I think the reclaim path is most likely triggered.
Quoted 5 lines
For fun, I sometimes used to run it with a window of just say 16 dax ranges so that reclaim was so heavy that if there was a bug, it will show up.
Yeah, my colleague had ever reported that a DAX window of 4KB will cause hang in our internal OS (which is 4.19, we back ported virtiofs to 4.19). But then I found that this issue doesn't exist in the latest upstream. The reason seems that in the upstream kernel,
devm_memremap_pages()called invirtio_fs_setup_dax()will fail directly since the dax window (4KB) is not aligned with the sparse memory section.
Quoted 6 lines
Quoted 3 lines
Given our default chunk size is 2MB (
FUSE_DAX_SHIFT), may be it is not a bad idea to enforce some minimum cache window size. IIRC, even one range is not enough. Minimum 2 are required for reclaim to not deadlock.Curiously, why minimum 1 range is not adequate? In which case minimum 2 are required?
Frankly speaking, right now I don't remember. I have vague memories of concluding in the past that 1 range is not sufficient. But if you like dive deeper, and try with one range and see if you can introduce deadlock.
Thanks
Vivek
Quoted 51 lines · Vivek Goyal
Quoted 50 lines · JeffleXu
Quoted 46 lines · Vivek Goyal
Quoted 35 lines · JeffleXu
Quoted 14 lines · Vivek Goyal
Quoted 13 lines · JeffleXu
Quoted 9 lines · Vivek Goyal
Quoted 5 lines · Jeffle Xu
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
Suggested-by Liu Jiang <[email protected]>Signed-off-by Jeffle Xu <[email protected]>Looks good to me. Have you done any testing to make sure nothing is broken.
Quoted 6 lines
Quoted 2 lines
xfstests -g quick shows no regression. The tested virtiofs is mounted with "dax=always".
I think xfstests might not trigger reclaim. You probably will have to run something like blogbench with a small dax window like 1G so that heavy reclaim happens.
Actually, I configured the DAX window to 8MB, i.e. 4 slots when running xfstests. Thus I think the reclaim path is most likely triggered.
Quoted 5 lines
For fun, I sometimes used to run it with a window of just say 16 dax ranges so that reclaim was so heavy that if there was a bug, it will show up.
Quoted 6 lines
Yeah, my colleague had ever reported that a DAX window of 4KB will cause hang in our internal OS (which is 4.19, we back ported virtiofs to 4.19). But then I found that this issue doesn't exist in the latest upstream. The reason seems that in the upstream kernel,
devm_memremap_pages()called invirtio_fs_setup_dax()will fail directly since the dax window (4KB) is not aligned with the sparse memory section.Given our default chunk size is 2MB (
FUSE_DAX_SHIFT), may be it is not a bad idea to enforce some minimum cache window size. IIRC, even one range is not enough. Minimum 2 are required for reclaim to not deadlock.
Quoted 8 lines
Quoted 2 lines
Curiously, why minimum 1 range is not adequate? In which case minimum 2 are required?
Frankly speaking, right now I don't remember. I have vague memories of concluding in the past that 1 range is not sufficient. But if you like dive deeper, and try with one range and see if you can introduce deadlock.
Alright, thanks.
Signature · 6 lines
Thanks, Jeffle _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 10 lines · JeffleXu
Quoted 7 lines · Vivek Goyal
Quoted 5 lines · JeffleXu
On 4/8/22 8:06 PM, Vivek Goyal wrote:
Curiously, why minimum 1 range is not adequate? In which case minimum 2 are required?
Quoted 8 lines
Quoted 5 lines
Frankly speaking, right now I don't remember. I have vague memories of concluding in the past that 1 range is not sufficient. But if you like dive deeper, and try with one range and see if you can introduce deadlock.
Alright, thanks.
Out of interest, how are you testing this at all? A patch from Dharmendra had been merged last week into libfuse to let it know about flags2, as we need that for our patches. But we didn't update the FLAGS yet to add in DAX on the libfuse side.
Is this used by virtio fs? Or is there another libfuse out there that should know about these flags (I think glusterfs has its own, but probably not using dax?).
Also, testing is always good, although I don't see how Jeffs patch would be able break anything here.
Thanks,
Bernd
Quoted 23 lines · Bernd Schubert
Quoted 10 lines · JeffleXu
Quoted 7 lines · Vivek Goyal
Quoted 5 lines · JeffleXu
On 4/8/22 8:06 PM, Vivek Goyal wrote:
Curiously, why minimum 1 range is not adequate? In which case minimum 2 are required?Quoted 8 lines
Quoted 5 lines
Frankly speaking, right now I don't remember. I have vague memories of concluding in the past that 1 range is not sufficient. But if you like dive deeper, and try with one range and see if you can introduce deadlock.
Alright, thanks.
Quoted 6 lines
Out of interest, how are you testing this at all? A patch from Dharmendra had been merged last week into libfuse to let it know about flags2, as we need that for our patches. But we didn't update the FLAGS yet to add in DAX on the libfuse side.
Is this used by virtio fs?
Yes, idea is that this is used by virtiofs. Now looks like there are multiple implementations of virtiofs daemon and they are either not using libfuse or have forked off libfuse or created a new libfuse equivalent in rust etc. So as fuse kernel gets updated, people are updating their corresponding code as need be.
For example, we have C version of virtiofsd in qemu. That has taken code from libfuse and built on top of it. BTW, C version of virtiofsd is deprecated now and not lot of new development is expected to take place there.
Then there is rust version of virtiofsd where new development is taking place and which is replacement of C virtiofsd.
https://gitlab.com/virtio-fs/virtiofsd
This does not use libfuse at all.
And I think other folks (like developers from Alibaba) have probably written their own implementation of virtiofsd. I am not sure what exactly are they using.
I see there is rust crate for fuse.
And there is one in cloud-hypervisor project.
https://github.com/cloud-hypervisor/fuse-backend-rs
Quoted 4 lines
Or is there another libfuse out there that should
know about these flags (I think glusterfs has its own, but probably not using dax?).
So server side of fuse seem to be all fragmented to me. People have written their own implementations based on their needs.
Quoted 2 lines
Also, testing is always good, although I don't see how Jeffs patch would be able break anything here.
Agreed. I worry about testing constantly as well. qemu version of virtiofsd does not have DAX support yet. Rust version DAX support is also minimal.
So for testing DAX, I have to rely on out of tree patches from qemu here if any changes in virtiofs client happen.
https://gitlab.com/virtio-fs/qemu/-/tree/virtio-fs-dev
Jeffle is probably relying on their own virtiofsd implementation for DAX testing.
Thanks
Vivek
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 10 lines · Vivek Goyal
On Mon, Apr 11, 2022 at 03:20:05PM +0200, Bernd Schubert wrote:
So for testing DAX, I have to rely on out of tree patches from qemu here if any changes in virtiofs client happen.
https://gitlab.com/virtio-fs/qemu/-/tree/virtio-fs-dev
Jeffle is probably relying on their own virtiofsd implementation for DAX testing.
Actually I also use the C version virtiofsd in the above described repository for testing :)
Signature · 6 lines
Thanks, Jeffle _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 6 lines · Jeffle Xu
Move dmap free worker kicker inside the critical region, so that extra spinlock lock/unlock could be avoided.
Suggested-by Liu Jiang <[email protected]>Signed-off-by Jeffle Xu <[email protected]>
Thanks, applied.
Miklos