virtiofs,fuse: support per-file DAX
Subsystems (4)
v2 cover virtiofs,fuse: support per-file DAX
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
There are three related scenarios:
1. Alloc inode: get per-file DAX flag from fuse_attr.flags. (patch 3)
2. Per-file DAX flag changes when the file has been opened. (patch 3) In this case, the dentry and inode are all marked as DONT_CACHE, and the DAX state won't be updated until the file is closed and reopened later.
3. Users can change the per-file DAX flag inside the guest by chattr(1). (patch 4)
4. Create new files under directories with DAX enabled. When creating new files in ext4/xfs on host, the new created files will inherit the per-file DAX flag from the directory, and thus the new created files in virtiofs will also inherit the per-file DAX flag if the fuse server derives fuse_attr.flags from the underlying ext4/xfs inode's per-file DAX flag.
Any comment is welcome.
[1] commit 9cb20f94afcd ("fs/ext4: Make DAX mount option a tri-state")
[2] commit 02beb2686ff9 ("fs/xfs: Make DAX mount option a tri-state")
changes since v1:
- add support for changing per-file DAX flags inside guest (patch 4)
v1:https://www.spinics.net/lists/linux-virtualization/msg51008.html
Jeffle Xu (4):
fuse: add fuse_should_enable_dax() helper
fuse: Make DAX mount option a tri-state
fuse: add per-file DAX flag
fuse: support changing per-file DAX flag inside guest
fs/fuse/dax.c | 36 ++++++++++++++++++++++++++++++++++-- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 16 ++++++++++++---- fs/fuse/inode.c | 7 +++++-- fs/fuse/ioctl.c | 9 ++++++--- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- include/uapi/linux/fuse.h | 5 +++++ 7 files changed, 78 insertions(+), 15 deletions(-)
Signature · 1 line
2.27.0
PATCH v2 1/4 fuse: add fuse_should_enable_dax() helper
This is in prep for following per-file DAX checking.
--- fs/fuse/dax.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
fs/fuse/dax.c +9 -1
@@ -1336,11 +1336,19 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -void fuse_dax_inode_init (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode);if (!fc->dax) +return false; + +return true; +} + +void fuse_dax_inode_init (struct inode *inode) +{ +if (!fuse_should_enable_dax (inode))return ; inode->i_flags |=S_DAX ;
Signature · 1 line
2.27.0
PATCH v2 2/4 fuse: Make DAX mount option a tri-state
In reply to Jeffle XuWe add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'.
By the time this patch is applied, 'inode' mode is actually equal to 'always' mode, before the per-file DAX flag is introduced in the following patch.
--- fs/fuse/dax.c | 13 ++++++++++++- fs/fuse/fuse_i.h | 11 +++++++++-- fs/fuse/inode.c | 2 +- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- 4 files changed, 36 insertions(+), 6 deletions(-)
fs/fuse/dax.c +12 -1
@@ -70,6 +70,9 @@ struct fuse_inode_dax { };struct fuse_conn_dax { +/** dax mode: FUSE_DAX_MOUNT_* (always, never or per-file) **/ +unsigned int mode; +/* DAX device */ struct dax_device *dev;
@@ -1288,7 +1291,8 @@ static int fuse_dax_mem_range_init(struct fuse_conn_dax *fcd)return ret; } -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev) +int fuse_dax_conn_alloc (struct fuse_conn *fc,unsigned int mode, +struct dax_device *dax_dev) {struct fuse_conn_dax *fcd;int err;
@@ -1301,6 +1305,7 @@ int fuse_dax_conn_alloc(struct fuse_conn *fc, struct dax_device *dax_dev)return -ENOMEM ;spin_lock_init (&fcd->lock); + fcd->mode = mode; fcd->dev = dax_dev; err =fuse_dax_mem_range_init (fcd);if (err) {
@@ -1339,10 +1344,16 @@ static const struct address_space_operations fuse_dax_file_aops = {static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); +unsigned int mode;if (!fc->dax)return false; + mode = fc->dax->mode; + +if (mode ==FUSE_DAX_MOUNT_NEVER ) +return false; +return true; }
fs/fuse/fuse_i.h +9 -2
@@ -487,6 +487,12 @@ struct fuse_dev {struct list_head entry; }; +enum { +FUSE_DAX_MOUNT_INODE , +FUSE_DAX_MOUNT_ALWAYS , +FUSE_DAX_MOUNT_NEVER , +}; +struct fuse_fs_context {int fd;unsigned int rootmode;
@@ -503,7 +509,7 @@ struct fuse_fs_context { bool no_control:1; bool no_force_umount:1; bool legacy_opts_show:1; - bool dax:1; +unsigned int dax;unsigned int max_read;unsigned int blksize;const char *subtype;
@@ -1242,7 +1248,8 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);int fuse_dax_mmap (struct file *file,struct vm_area_struct *vma);int fuse_dax_break_layouts (struct inode *inode,u64 dmap_start,u64 dmap_end); -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev); +int fuse_dax_conn_alloc (struct fuse_conn *fc,unsigned int mode, +struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode);
fs/fuse/inode.c +1 -1
@@ -1434,7 +1434,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) sb->s_subtype = ctx->subtype; ctx->subtype =NULL ;if (IS_ENABLED (CONFIG_FUSE_DAX )) { - err =fuse_dax_conn_alloc (fc, ctx->dax_dev); + err =fuse_dax_conn_alloc (fc, ctx->dax, ctx->dax_dev);if (err) goto err; }
fs/fuse/virtio_fs.c +14 -2
@@ -88,12 +88,21 @@ struct virtio_fs_req_work {static int virtio_fs_enqueue_req (struct virtio_fs_vq *fsvq,struct fuse_req *req,bool in_flight); +static const struct constant_table dax_param_enums[] = { + {"inode" ,FUSE_DAX_MOUNT_INODE }, + {"always" ,FUSE_DAX_MOUNT_ALWAYS }, + {"never" ,FUSE_DAX_MOUNT_NEVER }, + {} +}; +enum {OPT_DAX , +OPT_DAX_ENUM , };static const struct fs_parameter_spec virtio_fs_parameters[] = {fsparam_flag ("dax" ,OPT_DAX ), +fsparam_enum ("dax" ,OPT_DAX_ENUM , dax_param_enums), {} };
@@ -110,7 +119,10 @@ static int virtio_fs_parse_param(struct fs_context *fc,switch (opt) {case OPT_DAX : - ctx->dax =1 ; + ctx->dax =FUSE_DAX_MOUNT_ALWAYS ; +break ; +case OPT_DAX_ENUM : + ctx->dax = result.uint_32;break ;default :return -EINVAL ;
@@ -1326,7 +1338,7 @@ static int virtio_fs_fill_super(struct super_block *sb, struct fs_context *fsc)/* virtiofs allocates and installs its own fuse devices */ ctx->fudptr =NULL ; -if (ctx->dax) { +if (ctx->dax !=FUSE_DAX_MOUNT_NEVER ) {if (!fs->dax_dev) { err = -EINVAL ; pr_err("virtio-fs: dax can't be enabled as filesystem"
Signature · 6 lines
2.27.0 _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Vivek Goyal
maintainerRe: [PATCH v2 2/4] fuse: Make DAX mount option a tri-state
Quoted 20 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'.
By the time this patch is applied, 'inode' mode is actually equal to 'always' mode, before the per-file DAX flag is introduced in the following patch.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 13 ++++++++++++- fs/fuse/fuse_i.h | 11 +++++++++-- fs/fuse/inode.c | 2 +- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- 4 files changed, 36 insertions(+), 6 deletions(-)fs/fuse/dax.c +0
-0
fs/fuse/dax.c struct fuse_inode_dax { · line 74
Quoted 5 lines
fs/fuse/dax.c +2
-0@@ -70,6 +70,9 @@ struct fuse_inode_dax { };struct fuse_conn_dax { +/** dax mode: FUSE_DAX_MOUNT_* (always, never or per-file) **/ +unsigned int mode;
Why "/**" ?
How about make it something like "enum fuse_dax_mode mode" instead?
enum fuse_dax_mode dax_mode;
Quoted 44 lines
fs/fuse/dax.c +10
-1+/* DAX device */ struct dax_device *dev;@@ -1288,7 +1291,8 @@ static int fuse_dax_mem_range_init(struct fuse_conn_dax *fcd)return ret; } -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev) +int fuse_dax_conn_alloc (struct fuse_conn *fc,unsigned int mode, +struct dax_device *dax_dev) {struct fuse_conn_dax *fcd;int err;@@ -1301,6 +1305,7 @@ int fuse_dax_conn_alloc(struct fuse_conn *fc, struct dax_device *dax_dev)return -ENOMEM ;spin_lock_init (&fcd->lock); + fcd->mode = mode; fcd->dev = dax_dev; err =fuse_dax_mem_range_init (fcd);if (err) {@@ -1339,10 +1344,16 @@ static const struct address_space_operations fuse_dax_file_aops = {static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); +unsigned int mode;if (!fc->dax)return false; + mode = fc->dax->mode; + +if (mode ==FUSE_DAX_MOUNT_NEVER ) +return false; +return true; }fs/fuse/fuse_i.h +0
-0
fs/fuse/fuse_i.h struct fuse_dev { · line 490
Quoted 4 lines
fs/fuse/fuse_i.h +1
-0@@ -487,6 +487,12 @@ struct fuse_dev {struct list_head entry; }; +enum {
And this becomes.
enum fuse_dax_mode {
};
fs/fuse/fuse_i.h struct fuse_dev { · line 494
Quoted 4 lines
fs/fuse/fuse_i.h +4
-0+FUSE_DAX_MOUNT_INODE , +FUSE_DAX_MOUNT_ALWAYS , + FUSE_DAX_MOUNT_NEVER, +};
How about getting rid of "MOUNT" and just do.
FUSE_DAX_INODE,
FUSE_DAX_ALWAYS,
FUSE_DAX_NEVER,
fs/fuse/fuse_i.h struct fuse_fs_context { · line 512
Quoted 10 lines
fs/fuse/fuse_i.h +2
-1+struct fuse_fs_context {int fd;unsigned int rootmode;@@ -503,7 +509,7 @@ struct fuse_fs_context { bool no_control:1; bool no_force_umount:1; bool legacy_opts_show:1; - bool dax:1; +unsigned int dax;
enum fuse_dax_mode dax_mode;
fs/fuse/fuse_i.h ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to); · line 1251
Quoted 9 lines
fs/fuse/fuse_i.h +1
-1unsigned int max_read;unsigned int blksize;const char *subtype;@@ -1242,7 +1248,8 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);int fuse_dax_mmap (struct file *file,struct vm_area_struct *vma);int fuse_dax_break_layouts (struct inode *inode,u64 dmap_start,u64 dmap_end); -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev); +int fuse_dax_conn_alloc(struct fuse_conn *fc, unsigned int mode,
^^ enum fuse_dax_mode dax_mode
Quoted 44 lines
fs/fuse/fuse_i.h +1
-0+ struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode);fs/fuse/inode.c +1
-1@@ -1434,7 +1434,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) sb->s_subtype = ctx->subtype; ctx->subtype =NULL ;if (IS_ENABLED (CONFIG_FUSE_DAX )) { - err =fuse_dax_conn_alloc (fc, ctx->dax_dev); + err =fuse_dax_conn_alloc (fc, ctx->dax, ctx->dax_dev);if (err) goto err; }fs/fuse/virtio_fs.c +9
-0@@ -88,12 +88,21 @@ struct virtio_fs_req_work {static int virtio_fs_enqueue_req (struct virtio_fs_vq *fsvq,struct fuse_req *req,bool in_flight); +static const struct constant_table dax_param_enums[] = { + {"inode" ,FUSE_DAX_MOUNT_INODE }, + {"always" ,FUSE_DAX_MOUNT_ALWAYS }, + {"never" ,FUSE_DAX_MOUNT_NEVER }, + {} +}; +enum {OPT_DAX , +OPT_DAX_ENUM , };static const struct fs_parameter_spec virtio_fs_parameters[] = {fsparam_flag ("dax" ,OPT_DAX ), +fsparam_enum ("dax" ,OPT_DAX_ENUM , dax_param_enums), {} };
fs/fuse/virtio_fs.c static int virtio_fs_parse_param(struct fs_context *fc, · line 125
Quoted 8 lines
fs/fuse/virtio_fs.c +4
-1@@ -110,7 +119,10 @@ static int virtio_fs_parse_param(struct fs_context *fc,switch (opt) {case OPT_DAX : - ctx->dax =1 ; + ctx->dax =FUSE_DAX_MOUNT_ALWAYS ; +break ; +case OPT_DAX_ENUM : + ctx->dax = result.uint_32;
Do we want to check here if result.uint_32 has one of the allowed values. FUSE_DAX_MOUNT_INODE, FUSE_DAX_MOUNT_ALWAYS or FUSE_DAX_MOUNT_NEVER. Or VFS has already taken care of that?
Vivek
Quoted 15 lines
fs/fuse/virtio_fs.c +1
-1break ;default :return -EINVAL ;@@ -1326,7 +1338,7 @@ static int virtio_fs_fill_super(struct super_block *sb, struct fs_context *fsc)/* virtiofs allocates and installs its own fuse devices */ ctx->fudptr =NULL ; -if (ctx->dax) { +if (ctx->dax !=FUSE_DAX_MOUNT_NEVER ) {if (!fs->dax_dev) { err = -EINVAL ; pr_err("virtio-fs: dax can't be enabled as filesystem"Signature · 1 line
2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v2 2/4] fuse: Make DAX mount option a tri-state
Quoted 23 lines · Vivek Goyal
Quoted 22 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'.
By the time this patch is applied, 'inode' mode is actually equal to 'always' mode, before the per-file DAX flag is introduced in the following patch.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 13 ++++++++++++- fs/fuse/fuse_i.h | 11 +++++++++-- fs/fuse/inode.c | 2 +- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- 4 files changed, 36 insertions(+), 6 deletions(-)fs/fuse/dax.c +0
-0@@ -70,6 +70,9 @@ struct fuse_inode_dax { };
Quoted 5 lines
fs/fuse/dax.c struct fuse_inode_dax { · line 74
Quoted 3 lines
fs/fuse/dax.c +2
-0struct fuse_conn_dax { +/** dax mode: FUSE_DAX_MOUNT_* (always, never or per-file) **/ +unsigned int mode;Why "/**" ?
I copied this comment style from fuse in v4.19... Anyway, I will fix this.
Quoted 4 lines
How about make it something like "enum
fuse_dax_modemode" instead?enum
fuse_dax_modedax_mode;
OK.
Quoted 48 lines
Quoted 48 lines
fs/fuse/dax.c +10
-1+/* DAX device */ struct dax_device *dev;@@ -1288,7 +1291,8 @@ static int fuse_dax_mem_range_init(struct fuse_conn_dax *fcd)return ret; } -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev) +int fuse_dax_conn_alloc (struct fuse_conn *fc,unsigned int mode, +struct dax_device *dax_dev) {struct fuse_conn_dax *fcd;int err;@@ -1301,6 +1305,7 @@ int fuse_dax_conn_alloc(struct fuse_conn *fc, struct dax_device *dax_dev)return -ENOMEM ;spin_lock_init (&fcd->lock); + fcd->mode = mode; fcd->dev = dax_dev; err =fuse_dax_mem_range_init (fcd);if (err) {@@ -1339,10 +1344,16 @@ static const struct address_space_operations fuse_dax_file_aops = {static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); +unsigned int mode;if (!fc->dax)return false; + mode = fc->dax->mode; + +if (mode ==FUSE_DAX_MOUNT_NEVER ) +return false; +return true; }fs/fuse/fuse_i.h +0
-0@@ -487,6 +487,12 @@ struct fuse_dev {struct list_head entry; };
Quoted 5 lines
fs/fuse/fuse_i.h struct fuse_dev { · line 490
Quoted 1 line
fs/fuse/fuse_i.h +1
-0+enum {And this becomes.
enum
fuse_dax_mode{
};
OK.
Quoted 11 lines
fs/fuse/fuse_i.h struct fuse_dev { · line 494
Quoted 5 lines
fs/fuse/fuse_i.h +4
-0+FUSE_DAX_MOUNT_INODE , +FUSE_DAX_MOUNT_ALWAYS , + FUSE_DAX_MOUNT_NEVER, +};How about getting rid of "MOUNT" and just do.
FUSE_DAX_INODE,
FUSE_DAX_ALWAYS,
FUSE_DAX_NEVER,
OK.
Quoted 13 lines
fs/fuse/fuse_i.h struct fuse_fs_context { · line 512
Quoted 11 lines
fs/fuse/fuse_i.h +2
-1+struct fuse_fs_context {int fd;unsigned int rootmode;@@ -503,7 +509,7 @@ struct fuse_fs_context { bool no_control:1; bool no_force_umount:1; bool legacy_opts_show:1; - bool dax:1; +unsigned int dax;enum
fuse_dax_modedax_mode;
OK.
Quoted 12 lines
fs/fuse/fuse_i.h ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to); · line 1251
Quoted 10 lines
fs/fuse/fuse_i.h +1
-1unsigned int max_read;unsigned int blksize;const char *subtype;@@ -1242,7 +1248,8 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);int fuse_dax_mmap (struct file *file,struct vm_area_struct *vma);int fuse_dax_break_layouts (struct inode *inode,u64 dmap_start,u64 dmap_end); -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev); +int fuse_dax_conn_alloc(struct fuse_conn *fc, unsigned int mode,^^ enum fuse_dax_mode dax_mode
OK.
Quoted 45 lines
Quoted 45 lines
fs/fuse/fuse_i.h +1
-0+ struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode);fs/fuse/inode.c +1
-1@@ -1434,7 +1434,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) sb->s_subtype = ctx->subtype; ctx->subtype =NULL ;if (IS_ENABLED (CONFIG_FUSE_DAX )) { - err =fuse_dax_conn_alloc (fc, ctx->dax_dev); + err =fuse_dax_conn_alloc (fc, ctx->dax, ctx->dax_dev);if (err) goto err; }fs/fuse/virtio_fs.c +9
-0@@ -88,12 +88,21 @@ struct virtio_fs_req_work {static int virtio_fs_enqueue_req (struct virtio_fs_vq *fsvq,struct fuse_req *req,bool in_flight); +static const struct constant_table dax_param_enums[] = { + {"inode" ,FUSE_DAX_MOUNT_INODE }, + {"always" ,FUSE_DAX_MOUNT_ALWAYS }, + {"never" ,FUSE_DAX_MOUNT_NEVER }, + {} +}; +enum {OPT_DAX , +OPT_DAX_ENUM , };static const struct fs_parameter_spec virtio_fs_parameters[] = {fsparam_flag ("dax" ,OPT_DAX ), +fsparam_enum ("dax" ,OPT_DAX_ENUM , dax_param_enums), {} };@@ -110,7 +119,10 @@ static int virtio_fs_parse_param(struct fs_context *fc,
Quoted 11 lines
fs/fuse/virtio_fs.c static int virtio_fs_parse_param(struct fs_context *fc, · line 125
Quoted 7 lines
fs/fuse/virtio_fs.c +4
-1switch (opt) {case OPT_DAX : - ctx->dax =1 ; + ctx->dax =FUSE_DAX_MOUNT_ALWAYS ; +break ; +case OPT_DAX_ENUM : + ctx->dax = result.uint_32;Do we want to check here if result.uint_32 has one of the allowed values.
FUSE_DAX_MOUNT_INODE,FUSE_DAX_MOUNT_ALWAYSorFUSE_DAX_MOUNT_NEVER. Or VFS has already taken care of that?
VFS will ensure that the returned result.uint_32 can only be among the values defined in 'struct constant_table', or fs_parse() will return -EINVAL.
Signature · 2 lines
Thanks, Jeffle
PATCH v2 3/4 fuse: add per-file DAX flag
In reply to Jeffle XuAdd one flag for fuse_attr.flags indicating if DAX shall be enabled for this file.
When the per-file DAX flag changes for an *opened* file, the state of the file won't be updated until this file is closed and reopened later.
--- fs/fuse/dax.c | 21 +++++++++++++++++---- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 5 +++-- fs/fuse/inode.c | 5 ++++- include/uapi/linux/fuse.h | 5 +++++ 5 files changed, 31 insertions(+), 9 deletions(-)
fs/fuse/dax.c +17 -4
@@ -1341,7 +1341,7 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -static bool fuse_should_enable_dax (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode,unsigned int flags) {struct fuse_conn *fc =get_fuse_conn (inode);unsigned int mode;
@@ -1354,18 +1354,31 @@ static bool fuse_should_enable_dax(struct inode *inode)if (mode ==FUSE_DAX_MOUNT_NEVER )return false; -return true; +if (mode ==FUSE_DAX_MOUNT_ALWAYS ) +return true; + +WARN_ON (mode !=FUSE_DAX_MOUNT_INODE ); +return flags &FUSE_ATTR_DAX ; } -void fuse_dax_inode_init (struct inode *inode) +void fuse_dax_inode_init (struct inode *inode,unsigned int flags) { -if (!fuse_should_enable_dax (inode)) +if (!fuse_should_enable_dax (inode, flags))return ; inode->i_flags |=S_DAX ; inode->i_data.a_ops = &fuse_dax_file_aops; } +void fuse_dax_dontcache (struct inode *inode,bool newdax) +{ +struct fuse_conn *fc =get_fuse_conn (inode); + +if (fc->dax && fc->dax->mode ==FUSE_DAX_MOUNT_INODE && +IS_DAX (inode) != newdax) +d_mark_dontcache (inode); +} +bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment) {if (fc->dax && (map_alignment >FUSE_DAX_SHIFT )) {
fs/fuse/file.c +2 -2
@@ -3142,7 +3142,7 @@ static const struct address_space_operations fuse_file_aops = { .write_end = fuse_write_end, }; -void fuse_init_file_inode (struct inode *inode) +void fuse_init_file_inode (struct inode *inode,struct fuse_attr *attr) {struct fuse_inode *fi =get_fuse_inode (inode);
@@ -3156,5 +3156,5 @@ void fuse_init_file_inode(struct inode *inode) fi->writepages =RB_ROOT ;if (IS_ENABLED (CONFIG_FUSE_DAX )) -fuse_dax_inode_init (inode); +fuse_dax_inode_init (inode, attr->flags); }
fs/fuse/fuse_i.h +3 -2
@@ -1000,7 +1000,7 @@ int fuse_notify_poll_wakeup(struct fuse_conn *fc,/** * Initialize file operations on a regular file */ -void fuse_init_file_inode (struct inode *inode); +void fuse_init_file_inode (struct inode *inode,struct fuse_attr *attr);/** * Initialize inode operations on regular files and special files
@@ -1252,8 +1252,9 @@ int fuse_dax_conn_alloc(struct fuse_conn *fc, unsigned int mode, struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi); -void fuse_dax_inode_init (struct inode *inode); +void fuse_dax_inode_init (struct inode *inode,unsigned int flags);void fuse_dax_inode_cleanup (struct inode *inode); +void fuse_dax_dontcache (struct inode *inode,bool newdax);bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment);void fuse_dax_cancel_work (struct fuse_conn *fc);
fs/fuse/inode.c +4 -1
@@ -269,6 +269,9 @@ void fuse_change_attributes(struct inode *inode, struct fuse_attr *attr,if (inval)invalidate_inode_pages2 (inode->i_mapping); } + +if (IS_ENABLED (CONFIG_FUSE_DAX )) +fuse_dax_dontcache (inode, attr->flags &FUSE_ATTR_DAX ); }static void fuse_init_inode (struct inode *inode,struct fuse_attr *attr)
@@ -281,7 +284,7 @@ static void fuse_init_inode(struct inode *inode, struct fuse_attr *attr) inode->i_ctime.tv_nsec = attr->ctimensec;if (S_ISREG (inode->i_mode)) {fuse_init_common (inode); -fuse_init_file_inode (inode); +fuse_init_file_inode (inode, attr); }else if (S_ISDIR (inode->i_mode))fuse_init_dir (inode);else if (S_ISLNK (inode->i_mode))
include/uapi/linux/fuse.h +5 -0
@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_ATTR_DAX */ #ifndef _LINUX_FUSE_H
@@ -449,8 +452,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)/** * Open flags
Signature · 1 line
2.27.0
Vivek Goyal
maintainerRe: [PATCH v2 3/4] fuse: add per-file DAX flag
Quoted 7 lines · Jeffle Xu
Add one flag for fuse_attr.flags indicating if DAX shall be enabled for this file.
When the per-file DAX flag changes for an *opened* file, the state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>
[..]
Quoted 15 lines
include/uapi/linux/fuse.h +3
-0@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_ATTR_DAX */ #ifndef _LINUX_FUSE_H
include/uapi/linux/fuse.h struct fuse_file_lock { · line 458
Quoted 7 lines
include/uapi/linux/fuse.h +2
-0@@ -449,8 +452,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)
Generic fuse changes (addition of FUSE_ATTR_DAX) should probably in a separate patch.
I am not clear on one thing. If we are planning to rely on persistent inode attr (FS_XFLAG_DAX as per Documentation/filesystems/dax.rst), then why fuse server needs to communicate the state of that attr using a flag? Can client directly query it? I am not sure where at these attrs stored and if fuse protocol currently supports it.
What about flag STATX_ATTR_DAX. We probably should report that too in stat if we are using dax on the inode?
Vivek
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v2 3/4] fuse: add per-file DAX flag
Quoted 23 lines · Vivek Goyal
Quoted 7 lines · Jeffle Xu
Add one flag for fuse_attr.flags indicating if DAX shall be enabled for this file.
When the per-file DAX flag changes for an *opened* file, the state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>[..]
Quoted 13 lines
include/uapi/linux/fuse.h +3
-0@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_ATTR_DAX */
Quoted 12 lines
include/uapi/linux/fuse.h struct fuse_file_lock { · line 458
Quoted 9 lines
include/uapi/linux/fuse.h +2
-0#ifndef _LINUX_FUSE_H@@ -449,8 +452,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)Generic fuse changes (addition of
FUSE_ATTR_DAX) should probably in a separate patch.
Got it.
Quoted 6 lines
I am not clear on one thing. If we are planning to rely on persistent inode attr (
FS_XFLAG_DAXas perDocumentation/filesystems/dax.rst), then why fuse server needs to communicate the state of that attr using a flag? Can client directly query it? I am not sure where at these attrs stored and if fuse protocol currently supports it.
There are two issues.
1. FUSE server side: Algorithm of deciding whether DAX is enabled for a file.
As I previously replied in [1], FUSE server must enable DAX if the backend file is flagged with FS_XFLAG_DAX, to make the FS_XFLAG_DAX previously set by FUSE client effective.
But I will argue that FUSE server also has the flexibility of the algorithm implementation. Even if guest queries FS_XFLAG_DAX by GETFLAGS/FSGETXATTR ioctl, FUSE server can still enable DAX when the backend file is not FS_XFLAG_DAX flagged.
2. The protocol between server and client.
extending LOOKUP vs. LOOKUP + GETFLAGS/FSGETXATTR ioctl
As I said in [1], client can directly query the FS_XFLAG_DAX flag, but there will be one more round trip.
[1]
https://lore.kernel.org/linux-fsdevel/[email protected]/T/#m3f3407158b2c028694c85d82d0d6bd0387f4e24e
Quoted 4 lines
What about flag
STATX_ATTR_DAX. We probably should report that too in stat if we are using dax on the inode?
VFS will automatically report STATX_ATTR_DAX if inode is in DAX mode, e.g., in vfs_getattr_nosec().
Signature · 6 lines
Thanks, Jeffle _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Vivek Goyal
maintainerRe: [PATCH v2 3/4] fuse: add per-file DAX flag
Quoted 75 lines · JeffleXu
Quoted 23 lines · Vivek Goyal
Quoted 7 lines · Jeffle Xu
Add one flag for fuse_attr.flags indicating if DAX shall be enabled for this file.
When the per-file DAX flag changes for an *opened* file, the state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>[..]
Quoted 13 lines
include/uapi/linux/fuse.h +3
-0@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_ATTR_DAX */ Quoted 12 lines
include/uapi/linux/fuse.h struct fuse_file_lock { · line 458
Quoted 9 lines
include/uapi/linux/fuse.h +2
-0#ifndef _LINUX_FUSE_H@@ -449,8 +452,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)Generic fuse changes (addition of
FUSE_ATTR_DAX) should probably in a separate patch.Got it.
Quoted 6 lines
I am not clear on one thing. If we are planning to rely on persistent inode attr (
FS_XFLAG_DAXas perDocumentation/filesystems/dax.rst), then why fuse server needs to communicate the state of that attr using a flag? Can client directly query it? I am not sure where at these attrs stored and if fuse protocol currently supports it.There are two issues.
1. FUSE server side: Algorithm of deciding whether DAX is enabled for a file.
As I previously replied in [1], FUSE server must enable DAX if the backend file is flagged with
FS_XFLAG_DAX, to make theFS_XFLAG_DAXpreviously set by FUSE client effective.But I will argue that FUSE server also has the flexibility of the algorithm implementation. Even if guest queries
FS_XFLAG_DAXby GETFLAGS/FSGETXATTR ioctl, FUSE server can still enable DAX when the backend file is notFS_XFLAG_DAXflagged.2. The protocol between server and client.
extending LOOKUP vs. LOOKUP + GETFLAGS/FSGETXATTR ioctl
As I said in [1], client can directly query the
FS_XFLAG_DAXflag, but there will be one more round trip.[1]
https://lore.kernel.org/linux-fsdevel/[email protected]/T/#m3f3407158b2c028694c85d82d0d6bd0387f4e24eQuoted 1 line
Quoted 6 lines
Quoted 3 lines
What about flag
STATX_ATTR_DAX. We probably should report that too in stat if we are using dax on the inode?VFS will automatically report
STATX_ATTR_DAXif inode is in DAX mode, e.g., invfs_getattr_nosec().
Good to know. Given user will know which files are using dax and which ones are not, it is even more important to define semantics properly. In what cases DAX will be driven by FS_XFLAGS_DAX attr and in what cases DAX will completely be driven by server.
May be we should divide it in two patch series. First patch series implements "-o dax=inode" and server follows FS_XFLAGS_DAX attr and reports during lookup/getattr/.....
And once that is merged this can be ehanced with "-o dax=server" where server is free to choose what files dax should be used on. Only if this is still needed.
Vivek
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v2 3/4] fuse: add per-file DAX flag
Quoted 83 lines · Vivek Goyal
Quoted 75 lines · JeffleXu
Quoted 23 lines · Vivek Goyal
Quoted 7 lines · Jeffle Xu
Add one flag for fuse_attr.flags indicating if DAX shall be enabled for this file.
When the per-file DAX flag changes for an *opened* file, the state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>[..]
Quoted 13 lines
include/uapi/linux/fuse.h +3
-0@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_ATTR_DAX */ Quoted 12 lines
include/uapi/linux/fuse.h struct fuse_file_lock { · line 458
Quoted 9 lines
include/uapi/linux/fuse.h +2
-0#ifndef _LINUX_FUSE_H@@ -449,8 +452,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)Generic fuse changes (addition of
FUSE_ATTR_DAX) should probably in a separate patch.Got it.
Quoted 6 lines
I am not clear on one thing. If we are planning to rely on persistent inode attr (
FS_XFLAG_DAXas perDocumentation/filesystems/dax.rst), then why fuse server needs to communicate the state of that attr using a flag? Can client directly query it? I am not sure where at these attrs stored and if fuse protocol currently supports it.There are two issues.
1. FUSE server side: Algorithm of deciding whether DAX is enabled for a file.
As I previously replied in [1], FUSE server must enable DAX if the backend file is flagged with
FS_XFLAG_DAX, to make theFS_XFLAG_DAXpreviously set by FUSE client effective.But I will argue that FUSE server also has the flexibility of the algorithm implementation. Even if guest queries
FS_XFLAG_DAXby GETFLAGS/FSGETXATTR ioctl, FUSE server can still enable DAX when the backend file is notFS_XFLAG_DAXflagged.2. The protocol between server and client.
extending LOOKUP vs. LOOKUP + GETFLAGS/FSGETXATTR ioctl
As I said in [1], client can directly query the
FS_XFLAG_DAXflag, but there will be one more round trip.[1]
https://lore.kernel.org/linux-fsdevel/[email protected]/T/#m3f3407158b2c028694c85d82d0d6bd0387f4e24eQuoted 1 line
Quoted 6 lines
Quoted 3 lines
What about flag
STATX_ATTR_DAX. We probably should report that too in stat if we are using dax on the inode?VFS will automatically report
STATX_ATTR_DAXif inode is in DAX mode, e.g., invfs_getattr_nosec().
Quoted 8 lines
Good to know. Given user will know which files are using dax and which ones are not, it is even more important to define semantics properly. In what cases DAX will be driven by
FS_XFLAGS_DAXattr and in what cases DAX will completely be driven by server.May be we should divide it in two patch series. First patch series implements "-o dax=inode" and server follows
FS_XFLAGS_DAXattr and reports during lookup/getattr/.....
Agreed, '-o dax=inode' and policy upon FS_XFLAG_DAX xattr could be implemented as the first step.
Quoted 4 lines
And once that is merged this can be ehanced with "-o dax=server" where server is free to choose what files dax should be used on. Only if this is still needed.
I also need to discuss with my colleagues about our using case, and if FS_XFLAG_DAX poly is enough.
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerRe: [PATCH v2 3/4] fuse: add per-file DAX flag
In reply to Jeffle XuQuoted 50 lines · Jeffle Xu
Add one flag for fuse_attr.flags indicating if DAX shall be enabled for this file.
When the per-file DAX flag changes for an *opened* file, the state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 21 +++++++++++++++++---- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 5 +++-- fs/fuse/inode.c | 5 ++++- include/uapi/linux/fuse.h | 5 +++++ 5 files changed, 31 insertions(+), 9 deletions(-)fs/fuse/dax.c +8
-4@@ -1341,7 +1341,7 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -static bool fuse_should_enable_dax (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode,unsigned int flags) {struct fuse_conn *fc =get_fuse_conn (inode);unsigned int mode;@@ -1354,18 +1354,31 @@ static bool fuse_should_enable_dax(struct inode *inode)if (mode ==FUSE_DAX_MOUNT_NEVER )return false; -return true; +if (mode ==FUSE_DAX_MOUNT_ALWAYS ) +return true; + +WARN_ON (mode !=FUSE_DAX_MOUNT_INODE ); +return flags &FUSE_ATTR_DAX ; } -void fuse_dax_inode_init (struct inode *inode) +void fuse_dax_inode_init (struct inode *inode,unsigned int flags) { -if (!fuse_should_enable_dax (inode)) +if (!fuse_should_enable_dax (inode, flags))return ; inode->i_flags |=S_DAX ; inode->i_data.a_ops = &fuse_dax_file_aops; }
fs/fuse/dax.c static bool fuse_should_enable_dax(struct inode *inode) · line 1381
Quoted 10 lines
fs/fuse/dax.c +9
-0+void fuse_dax_dontcache (struct inode *inode,bool newdax) +{ +struct fuse_conn *fc =get_fuse_conn (inode); + +if (fc->dax && fc->dax->mode ==FUSE_DAX_MOUNT_INODE && +IS_DAX (inode) != newdax) +d_mark_dontcache (inode); +} +
This capability to mark an inode dontcache should probably be in a separate patch. These seem to logically two functionalities. One is enabling DAX on an inode. And second is making sure how soon you see the effect of that change and hence marking inode dontcache.
Not sure how useful this is. In cache=none mode we should get rid of inode ASAP. In cache=auto mode we will get rid of after 1 second (or after a user specified timeout). So only place this seems to be useful is cache=always.
Vivek
Quoted 98 lines
fs/fuse/dax.c +0
-0bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment) {if (fc->dax && (map_alignment >FUSE_DAX_SHIFT )) {fs/fuse/file.c +2
-2@@ -3142,7 +3142,7 @@ static const struct address_space_operations fuse_file_aops = { .write_end = fuse_write_end, }; -void fuse_init_file_inode (struct inode *inode) +void fuse_init_file_inode (struct inode *inode,struct fuse_attr *attr) {struct fuse_inode *fi =get_fuse_inode (inode);@@ -3156,5 +3156,5 @@ void fuse_init_file_inode(struct inode *inode) fi->writepages =RB_ROOT ;if (IS_ENABLED (CONFIG_FUSE_DAX )) -fuse_dax_inode_init (inode); +fuse_dax_inode_init (inode, attr->flags); }fs/fuse/fuse_i.h +3
-2@@ -1000,7 +1000,7 @@ int fuse_notify_poll_wakeup(struct fuse_conn *fc,/** * Initialize file operations on a regular file */ -void fuse_init_file_inode (struct inode *inode); +void fuse_init_file_inode (struct inode *inode,struct fuse_attr *attr);/** * Initialize inode operations on regular files and special files @@ -1252,8 +1252,9 @@ int fuse_dax_conn_alloc(struct fuse_conn *fc, unsigned int mode, struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi); -void fuse_dax_inode_init (struct inode *inode); +void fuse_dax_inode_init (struct inode *inode,unsigned int flags);void fuse_dax_inode_cleanup (struct inode *inode); +void fuse_dax_dontcache (struct inode *inode,bool newdax);bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment);void fuse_dax_cancel_work (struct fuse_conn *fc);fs/fuse/inode.c +4
-1@@ -269,6 +269,9 @@ void fuse_change_attributes(struct inode *inode, struct fuse_attr *attr,if (inval)invalidate_inode_pages2 (inode->i_mapping); } + +if (IS_ENABLED (CONFIG_FUSE_DAX )) +fuse_dax_dontcache (inode, attr->flags &FUSE_ATTR_DAX ); }static void fuse_init_inode (struct inode *inode,struct fuse_attr *attr)@@ -281,7 +284,7 @@ static void fuse_init_inode(struct inode *inode, struct fuse_attr *attr) inode->i_ctime.tv_nsec = attr->ctimensec;if (S_ISREG (inode->i_mode)) {fuse_init_common (inode); -fuse_init_file_inode (inode); +fuse_init_file_inode (inode, attr); }else if (S_ISDIR (inode->i_mode))fuse_init_dir (inode);else if (S_ISLNK (inode->i_mode))include/uapi/linux/fuse.h +5
-0@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_ATTR_DAX */ #ifndef _LINUX_FUSE_H@@ -449,8 +452,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)/** * Open flags Signature · 1 line
2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v2 3/4] fuse: add per-file DAX flag
Quoted 52 lines · Vivek Goyal
Quoted 51 lines · Jeffle Xu
Add one flag for fuse_attr.flags indicating if DAX shall be enabled for this file.
When the per-file DAX flag changes for an *opened* file, the state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 21 +++++++++++++++++---- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 5 +++-- fs/fuse/inode.c | 5 ++++- include/uapi/linux/fuse.h | 5 +++++ 5 files changed, 31 insertions(+), 9 deletions(-)fs/fuse/dax.c +8
-4@@ -1341,7 +1341,7 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -static bool fuse_should_enable_dax (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode,unsigned int flags) {struct fuse_conn *fc =get_fuse_conn (inode);unsigned int mode;@@ -1354,18 +1354,31 @@ static bool fuse_should_enable_dax(struct inode *inode)if (mode ==FUSE_DAX_MOUNT_NEVER )return false; -return true; +if (mode ==FUSE_DAX_MOUNT_ALWAYS ) +return true; + +WARN_ON (mode !=FUSE_DAX_MOUNT_INODE ); +return flags &FUSE_ATTR_DAX ; } -void fuse_dax_inode_init (struct inode *inode) +void fuse_dax_inode_init (struct inode *inode,unsigned int flags) { -if (!fuse_should_enable_dax (inode)) +if (!fuse_should_enable_dax (inode, flags))return ; inode->i_flags |=S_DAX ; inode->i_data.a_ops = &fuse_dax_file_aops; }
Quoted 14 lines
fs/fuse/dax.c static bool fuse_should_enable_dax(struct inode *inode) · line 1381
Quoted 9 lines
fs/fuse/dax.c +9
-0+void fuse_dax_dontcache (struct inode *inode,bool newdax) +{ +struct fuse_conn *fc =get_fuse_conn (inode); + +if (fc->dax && fc->dax->mode ==FUSE_DAX_MOUNT_INODE && +IS_DAX (inode) != newdax) +d_mark_dontcache (inode); +} +This capability to mark an inode dontcache should probably be in a separate patch. These seem to logically two functionalities. One is enabling DAX on an inode. And second is making sure how soon you see the effect of that change and hence marking inode dontcache.
OK, sounds reasonable.
Quoted 5 lines
Not sure how useful this is. In cache=none mode we should get rid of inode ASAP. In cache=auto mode we will get rid of after 1 second (or after a user specified timeout). So only place this seems to be useful is cache=always.
Actually dontcache here is used to avoid dynamic switching between DAX and non-DAX state while file is opened. The complexity of dynamic switching is that, you have to clear the address_space, since page cache and DAX entry can not coexist in the address space. Besides, inode->a_ops also needs to be changed dynamically.
With dontcache, dynamic switching is no longer needed and the DAX state will be decided only when inode (in memory) is initialized. The downside is that the new DAX state won't be updated until the file is closed and reopened later.
'cache=none' only invalidates dentry, while the inode (in memory) is still there (with address_space uncleared and a_ops unchanged).
The dynamic switching may be done, though it's not such straightforward. Currently, ext4/xfs are all implemented in this dontcache way, i.e., the new DAX state won't be seen until the file is closed and reopened later.
Signature · 2 lines
Thanks, Jeffle
Re: [PATCH v2 3/4] fuse: add per-file DAX flag
Quoted 78 lines · JeffleXu
Quoted 52 lines · Vivek Goyal
Quoted 51 lines · Jeffle Xu
Add one flag for fuse_attr.flags indicating if DAX shall be enabled for this file.
When the per-file DAX flag changes for an *opened* file, the state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 21 +++++++++++++++++---- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 5 +++-- fs/fuse/inode.c | 5 ++++- include/uapi/linux/fuse.h | 5 +++++ 5 files changed, 31 insertions(+), 9 deletions(-)fs/fuse/dax.c +8
-4@@ -1341,7 +1341,7 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -static bool fuse_should_enable_dax (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode,unsigned int flags) {struct fuse_conn *fc =get_fuse_conn (inode);unsigned int mode;@@ -1354,18 +1354,31 @@ static bool fuse_should_enable_dax(struct inode *inode)if (mode ==FUSE_DAX_MOUNT_NEVER )return false; -return true; +if (mode ==FUSE_DAX_MOUNT_ALWAYS ) +return true; + +WARN_ON (mode !=FUSE_DAX_MOUNT_INODE ); +return flags &FUSE_ATTR_DAX ; } -void fuse_dax_inode_init (struct inode *inode) +void fuse_dax_inode_init (struct inode *inode,unsigned int flags) { -if (!fuse_should_enable_dax (inode)) +if (!fuse_should_enable_dax (inode, flags))return ; inode->i_flags |=S_DAX ; inode->i_data.a_ops = &fuse_dax_file_aops; }Quoted 14 lines
fs/fuse/dax.c static bool fuse_should_enable_dax(struct inode *inode) · line 1381
Quoted 9 lines
fs/fuse/dax.c +9
-0+void fuse_dax_dontcache (struct inode *inode,bool newdax) +{ +struct fuse_conn *fc =get_fuse_conn (inode); + +if (fc->dax && fc->dax->mode ==FUSE_DAX_MOUNT_INODE && +IS_DAX (inode) != newdax) +d_mark_dontcache (inode); +} +This capability to mark an inode dontcache should probably be in a separate patch. These seem to logically two functionalities. One is enabling DAX on an inode. And second is making sure how soon you see the effect of that change and hence marking inode dontcache.
OK, sounds reasonable.
Quoted 5 lines
Not sure how useful this is. In cache=none mode we should get rid of inode ASAP. In cache=auto mode we will get rid of after 1 second (or after a user specified timeout). So only place this seems to be useful is cache=always.
Quoted 5 lines
Actually dontcache here is used to avoid dynamic switching between DAX and non-DAX state while file is opened. The complexity of dynamic switching is that, you have to clear the
address_space, since page cache and DAX entry can not coexist in the address space. Besides,inode->a_opsalso needs to be changed dynamically.
FYI some context on the complication of switching DAX state dynamically, when Ira Weiny was working on per-file DAX of ext4/xfs.
Quoted 1 line
At LSF/MM we discussed the difficulties of switching the DAX state of
a file with active mappings / page cache. It was thought the races could be avoided by limiting DAX state flips to 0-length files.
Quoted 2 lines
However, this turns out to not be true.[3][5] This is because address
space operations (a_ops) may be in use at any time the inode is referenced.
Quoted 1 line
from Ira Weiny
[1]
https://patchwork.kernel.org/project/xfs/cover/[email protected]/
[2] https://lore.kernel.org/lkml/[email protected]/
[3] https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/
[4] https://lore.kernel.org/lkml/20200403182904.GP80283@magnolia/
Quoted 13 lines
With dontcache, dynamic switching is no longer needed and the DAX state will be decided only when inode (in memory) is initialized. The downside is that the new DAX state won't be updated until the file is closed and reopened later.
'cache=none' only invalidates dentry, while the inode (in memory) is still there (with
address_spaceuncleared anda_opsunchanged).The dynamic switching may be done, though it's not such straightforward. Currently, ext4/xfs are all implemented in this dontcache way, i.e., the new DAX state won't be seen until the file is closed and reopened later.
Signature · 6 lines
Thanks, Jeffle _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Vivek Goyal
maintainerRe: [PATCH v2 3/4] fuse: add per-file DAX flag
In reply to JeffleXuQuoted 84 lines · JeffleXu
Quoted 52 lines · Vivek Goyal
Quoted 51 lines · Jeffle Xu
Add one flag for fuse_attr.flags indicating if DAX shall be enabled for this file.
When the per-file DAX flag changes for an *opened* file, the state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 21 +++++++++++++++++---- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 5 +++-- fs/fuse/inode.c | 5 ++++- include/uapi/linux/fuse.h | 5 +++++ 5 files changed, 31 insertions(+), 9 deletions(-)fs/fuse/dax.c +8
-4@@ -1341,7 +1341,7 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -static bool fuse_should_enable_dax (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode,unsigned int flags) {struct fuse_conn *fc =get_fuse_conn (inode);unsigned int mode;@@ -1354,18 +1354,31 @@ static bool fuse_should_enable_dax(struct inode *inode)if (mode ==FUSE_DAX_MOUNT_NEVER )return false; -return true; +if (mode ==FUSE_DAX_MOUNT_ALWAYS ) +return true; + +WARN_ON (mode !=FUSE_DAX_MOUNT_INODE ); +return flags &FUSE_ATTR_DAX ; } -void fuse_dax_inode_init (struct inode *inode) +void fuse_dax_inode_init (struct inode *inode,unsigned int flags) { -if (!fuse_should_enable_dax (inode)) +if (!fuse_should_enable_dax (inode, flags))return ; inode->i_flags |=S_DAX ; inode->i_data.a_ops = &fuse_dax_file_aops; }Quoted 14 lines
fs/fuse/dax.c static bool fuse_should_enable_dax(struct inode *inode) · line 1381
Quoted 9 lines
fs/fuse/dax.c +9
-0+void fuse_dax_dontcache (struct inode *inode,bool newdax) +{ +struct fuse_conn *fc =get_fuse_conn (inode); + +if (fc->dax && fc->dax->mode ==FUSE_DAX_MOUNT_INODE && +IS_DAX (inode) != newdax) +d_mark_dontcache (inode); +} +This capability to mark an inode dontcache should probably be in a separate patch. These seem to logically two functionalities. One is enabling DAX on an inode. And second is making sure how soon you see the effect of that change and hence marking inode dontcache.
OK, sounds reasonable.
Quoted 5 lines
Not sure how useful this is. In cache=none mode we should get rid of inode ASAP. In cache=auto mode we will get rid of after 1 second (or after a user specified timeout). So only place this seems to be useful is cache=always.
Actually dontcache here is used to avoid dynamic switching between DAX and non-DAX state while file is opened. The complexity of dynamic switching is that, you have to clear the
address_space, since page cache and DAX entry can not coexist in the address space. Besides,inode->a_opsalso needs to be changed dynamically.
Quoted 7 lines
With dontcache, dynamic switching is no longer needed and the DAX state will be decided only when inode (in memory) is initialized. The downside is that the new DAX state won't be updated until the file is closed and reopened later.
'cache=none' only invalidates dentry, while the inode (in memory) is still there (with
address_spaceuncleared anda_opsunchanged).
Aha.., that's a good point.
Quoted 4 lines
The dynamic switching may be done, though it's not such straightforward. Currently, ext4/xfs are all implemented in this dontcache way, i.e., the new DAX state won't be seen until the file is closed and reopened later.
Got it. Agreed that dontcache seems reasonable if file's DAX state has changed. Keep it in separate patch though with proper commit logs.
Also, please copy virtiofs list ([email protected]) when you post patches next time.
Vivek
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Re: [PATCH v2 3/4] fuse: add per-file DAX flag
Quoted 99 lines · Vivek Goyal
Quoted 84 lines · JeffleXu
Quoted 52 lines · Vivek Goyal
Quoted 51 lines · Jeffle Xu
Add one flag for fuse_attr.flags indicating if DAX shall be enabled for this file.
When the per-file DAX flag changes for an *opened* file, the state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 21 +++++++++++++++++---- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 5 +++-- fs/fuse/inode.c | 5 ++++- include/uapi/linux/fuse.h | 5 +++++ 5 files changed, 31 insertions(+), 9 deletions(-)fs/fuse/dax.c +8
-4@@ -1341,7 +1341,7 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -static bool fuse_should_enable_dax (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode,unsigned int flags) {struct fuse_conn *fc =get_fuse_conn (inode);unsigned int mode;@@ -1354,18 +1354,31 @@ static bool fuse_should_enable_dax(struct inode *inode)if (mode ==FUSE_DAX_MOUNT_NEVER )return false; -return true; +if (mode ==FUSE_DAX_MOUNT_ALWAYS ) +return true; + +WARN_ON (mode !=FUSE_DAX_MOUNT_INODE ); +return flags &FUSE_ATTR_DAX ; } -void fuse_dax_inode_init (struct inode *inode) +void fuse_dax_inode_init (struct inode *inode,unsigned int flags) { -if (!fuse_should_enable_dax (inode)) +if (!fuse_should_enable_dax (inode, flags))return ; inode->i_flags |=S_DAX ; inode->i_data.a_ops = &fuse_dax_file_aops; }Quoted 14 lines
fs/fuse/dax.c static bool fuse_should_enable_dax(struct inode *inode) · line 1381
Quoted 9 lines
fs/fuse/dax.c +9
-0+void fuse_dax_dontcache (struct inode *inode,bool newdax) +{ +struct fuse_conn *fc =get_fuse_conn (inode); + +if (fc->dax && fc->dax->mode ==FUSE_DAX_MOUNT_INODE && +IS_DAX (inode) != newdax) +d_mark_dontcache (inode); +} +This capability to mark an inode dontcache should probably be in a separate patch. These seem to logically two functionalities. One is enabling DAX on an inode. And second is making sure how soon you see the effect of that change and hence marking inode dontcache.
OK, sounds reasonable.
Quoted 5 lines
Not sure how useful this is. In cache=none mode we should get rid of inode ASAP. In cache=auto mode we will get rid of after 1 second (or after a user specified timeout). So only place this seems to be useful is cache=always.
Actually dontcache here is used to avoid dynamic switching between DAX and non-DAX state while file is opened. The complexity of dynamic switching is that, you have to clear the
address_space, since page cache and DAX entry can not coexist in the address space. Besides,inode->a_opsalso needs to be changed dynamically.Quoted 7 lines
With dontcache, dynamic switching is no longer needed and the DAX state will be decided only when inode (in memory) is initialized. The downside is that the new DAX state won't be updated until the file is closed and reopened later.
'cache=none' only invalidates dentry, while the inode (in memory) is still there (with
address_spaceuncleared anda_opsunchanged).Aha.., that's a good point.
Quoted 4 lines
The dynamic switching may be done, though it's not such straightforward. Currently, ext4/xfs are all implemented in this dontcache way, i.e., the new DAX state won't be seen until the file is closed and reopened later.
Quoted 7 lines
Got it. Agreed that dontcache seems reasonable if file's DAX state has changed. Keep it in separate patch though with proper commit logs.
Also, please copy virtiofs list ([email protected]) when you post patches next time.
Got it. By the way, what's the git repository of virtiofsd? AFAIK, virtiofsd included in qemu ([email protected]:qemu/qemu.git) doesn't support DAX yet?
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerRe: [PATCH v2 3/4] fuse: add per-file DAX flag
On Wed, Jul 21, 2021 at 10:14:44PM +0800, JeffleXu wrote:
[..]
Quoted 7 lines
Quoted 3 lines
Also, please copy virtiofs list ([email protected]) when you post patches next time.
Got it. By the way, what's the git repository of virtiofsd? AFAIK, virtiofsd included in qemu ([email protected]:qemu/qemu.git) doesn't support DAX yet?
Yes virtiofsd got merged in qemu upstream. And it does not support dax yet. David is still sorting out couple of issues based on feedback. I think following is the branch where he had pushed his latest patches.
https://gitlab.com/virtio-fs/qemu/-/tree/virtio-fs-dev
David, please correct me if that's not the case.
Vivek
PATCH v2 4/4 fuse: support changing per-file DAX flag inside guest
In reply to Jeffle XuFuse client can enable or disable per-file DAX inside guest by chattr(1). Similarly the new state won't be updated until the file is closed and reopened later.
It is worth nothing that it is a best-effort style, since whether per-file DAX is enabled or not is controlled by fuse_attr.flags retrieved by FUSE LOOKUP routine, while the algorithm constructing fuse_attr.flags is totally fuse server specific, not to mention ioctl may not be supported by fuse server at all.
--- fs/fuse/ioctl.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
fs/fuse/ioctl.c +6 -3
@@ -460,6 +460,7 @@ int fuse_fileattr_set(struct user_namespace *mnt_userns,struct fuse_file *ff;unsigned int flags = fa->flags;struct fsxattr xfa; +bool newdax;int err; ff =fuse_priv_ioctl_prepare (inode);
@@ -467,10 +468,9 @@ int fuse_fileattr_set(struct user_namespace *mnt_userns,return PTR_ERR (ff);if (fa->flags_valid) { + newdax = flags &FS_DAX_FL ; err =fuse_priv_ioctl (inode, ff,FS_IOC_SETFLAGS , &flags,sizeof (flags)); -if (err) - goto cleanup; }else {memset (&xfa,0 ,sizeof (xfa)); xfa.fsx_xflags = fa->fsx_xflags;
@@ -479,11 +479,14 @@ int fuse_fileattr_set(struct user_namespace *mnt_userns, xfa.fsx_projid = fa->fsx_projid; xfa.fsx_cowextsize = fa->fsx_cowextsize; + newdax = fa->fsx_xflags &FS_XFLAG_DAX ; err =fuse_priv_ioctl (inode, ff,FS_IOC_FSSETXATTR , &xfa,sizeof (xfa)); } -cleanup: +if (!err &&IS_ENABLED (CONFIG_FUSE_DAX )) +fuse_dax_dontcache (inode, newdax); +fuse_priv_ioctl_cleanup (inode, ff);return err;
Signature · 1 line
2.27.0
Vivek Goyal
maintainerRe: [PATCH v2 4/4] fuse: support changing per-file DAX flag inside guest
Quoted 41 lines · Jeffle Xu
Fuse client can enable or disable per-file DAX inside guest by chattr(1). Similarly the new state won't be updated until the file is closed and reopened later.
It is worth nothing that it is a best-effort style, since whether per-file DAX is enabled or not is controlled by fuse_attr.flags retrieved by FUSE LOOKUP routine, while the algorithm constructing fuse_attr.flags is totally fuse server specific, not to mention ioctl may not be supported by fuse server at all.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/ioctl.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)fs/fuse/ioctl.c +2
-2@@ -460,6 +460,7 @@ int fuse_fileattr_set(struct user_namespace *mnt_userns,struct fuse_file *ff;unsigned int flags = fa->flags;struct fsxattr xfa; +bool newdax;int err; ff =fuse_priv_ioctl_prepare (inode);@@ -467,10 +468,9 @@ int fuse_fileattr_set(struct user_namespace *mnt_userns,return PTR_ERR (ff);if (fa->flags_valid) { + newdax = flags &FS_DAX_FL ; err =fuse_priv_ioctl (inode, ff,FS_IOC_SETFLAGS , &flags,sizeof (flags)); -if (err) - goto cleanup; }else {memset (&xfa,0 ,sizeof (xfa)); xfa.fsx_xflags = fa->fsx_xflags;@@ -479,11 +479,14 @@ int fuse_fileattr_set(struct user_namespace *mnt_userns, xfa.fsx_projid = fa->fsx_projid;
fs/fuse/ioctl.c int fuse_fileattr_set(struct user_namespace *mnt_userns, · line 488
Quoted 10 lines
fs/fuse/ioctl.c +3
-1xfa.fsx_cowextsize = fa->fsx_cowextsize; + newdax = fa->fsx_xflags &FS_XFLAG_DAX ; err =fuse_priv_ioctl (inode, ff,FS_IOC_FSSETXATTR , &xfa,sizeof (xfa)); } -cleanup: +if (!err &&IS_ENABLED (CONFIG_FUSE_DAX )) +fuse_dax_dontcache (inode, newdax);
This assumes that server will set ATTR_DAX flag for inode based on whether inode has FS_DAX_FL set or not.
So that means server first will have to know that client has DAX enabled so that it can query FS_DAX_FL. And in current framework we don't have a way for server to know if client is using DAX or not.
I think there is little disconnect here. So either client should be checking FS_DAX_FL flag on inode. But we probably don't want to pay extra round trip cost for this.
That means somehow server should return this information as part of inode attrs only if client wants this extra file attr informaiton. So may be GETATTR should be enhanced instead to return file attr information too if client asked for it?
I have not looked what it takes to implement this. If this is too complicated, then alternate approach will be that it is up to the server to decide what inodes should use DAX and there is no guarantee that server will make sue of FS_DAX_FL flag. fuse will still support setting FS_DAX_FL but server could choose to not use it at all. In that case fuse client will not have to query file attrs in GETATTR and just rely on ATTR_DAX flag set by server. I think that's what you are implementing. If that's the case then dontcache does not make much sense because you don't even know if server is looking at FS_DAX_FL to decide whether DAX should be used or not.
Thanks
Vivek
Quoted 7 lines
fs/fuse/ioctl.c +1
-0+fuse_priv_ioctl_cleanup (inode, ff);return err;Signature · 1 line
2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Vivek Goyal
maintainerQuoted 17 lines · Jeffle Xu
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
There are three related scenarios:
1. Alloc inode: get per-file DAX flag from fuse_attr.flags. (patch 3)
2. Per-file DAX flag changes when the file has been opened. (patch 3) In this case, the dentry and inode are all marked asDONT_CACHE, and the DAX state won't be updated until the file is closed and reopened later.
3. Users can change the per-file DAX flag inside the guest by chattr(1). (patch 4)
4. Create new files under directories with DAX enabled. When creating new files in ext4/xfs on host, the new created files will inherit the per-file DAX flag from the directory, and thus the new created files in virtiofs will also inherit the per-file DAX flag if the fuse server derives fuse_attr.flags from the underlying ext4/xfs inode's per-file DAX flag.
Thinking little bit more about this from requirement perspective. I think we are trying to address two use cases here.
A. Client does not know which files DAX should be used on. Only server
knows it and server passes this information to client. I suspect that's your primary use case.
B. Client is driving which files are supposed to be using DAX. This is
exactly same as the model ext4/xfs are using by storing a persistent flag on inode.
Current patches seem to be a hybrid of both approach A and B.
If we were to implement B, then fuse client probably needs to have the capability to query FS_XFLAG_DAX on inode and decide whether to enable DAX or not. (Without extra round trip). Or know it indirectly by extending GETATTR and requesting this explicitly.
If we were only implementing A, then server does not have a way to tell client to enable DAX. Server can either look at FS_XFLAG_DAX and decide to enable DAX or use some other property. Given querying FS_XFLAG_DAX will be an extra ioctl() on every inode lookup/getattr, it probably will be a server option. But enabling on server does not mean client will enable it.
I think my primary concern with this patch right now is trying to figure out which requirement we are trying to cater to first and how to connect server and client well so they both understand what mode they are operating in and interact well.
Vivek
Quoted 19 lines
Any comment is welcome.
[1] commit
9cb20f94afcd("fs/ext4: Make DAX mount option a tri-state")
[2] commit02beb2686ff9("fs/xfs: Make DAX mount option a tri-state")changes since v1:
- add support for changing per-file DAX flags inside guest (patch 4)v1:https://www.spinics.net/lists/linux-virtualization/msg51008.html
Jeffle Xu (4):
fuse: addfuse_should_enable_dax()helper
fuse: Make DAX mount option a tri-state
fuse: add per-file DAX flag
fuse: support changing per-file DAX flag inside guest
Quoted 12 lines
fs/fuse/dax.c | 36 ++++++++++++++++++++++++++++++++++-- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 16 ++++++++++++---- fs/fuse/inode.c | 7 +++++-- fs/fuse/ioctl.c | 9 ++++++--- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- include/uapi/linux/fuse.h | 5 +++++ 7 files changed, 78 insertions(+), 15 deletions(-)Signature · 1 line
2.27.0
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 19 lines · Vivek Goyal
Quoted 17 lines · Jeffle Xu
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
There are three related scenarios:
1. Alloc inode: get per-file DAX flag from fuse_attr.flags. (patch 3)
2. Per-file DAX flag changes when the file has been opened. (patch 3) In this case, the dentry and inode are all marked asDONT_CACHE, and the DAX state won't be updated until the file is closed and reopened later.
3. Users can change the per-file DAX flag inside the guest by chattr(1). (patch 4)
4. Create new files under directories with DAX enabled. When creating new files in ext4/xfs on host, the new created files will inherit the per-file DAX flag from the directory, and thus the new created files in virtiofs will also inherit the per-file DAX flag if the fuse server derives fuse_attr.flags from the underlying ext4/xfs inode's per-file DAX flag.
Quoted 6 lines
Thinking little bit more about this from requirement perspective. I think we are trying to address two use cases here.
A. Client does not know which files DAX should be used on. Only server
knows it and server passes this information to client. I suspect that's your primary use case.
Yes, this is the starting point of this patch set.
Quoted 11 lines
B. Client is driving which files are supposed to be using DAX. This is
exactly same as the model ext4/xfs are using by storing a persistent flag on inode.Current patches seem to be a hybrid of both approach A and B.
If we were to implement B, then fuse client probably needs to have the capability to query
FS_XFLAG_DAXon inode and decide whether to enable DAX or not. (Without extra round trip). Or know it indirectly by extending GETATTR and requesting this explicitly.
If guest queries if the file is DAX capable or not by an extra GETATTR, I'm afraid this will add extra round trip.
Or if we add the DAX flag (ATTR_DAX) by extending LOOKUP, as done by this patch set, then the FUSE server needs to set ATTR_DAX according to the FS_XFLAG_DAX of the backend files, *to make the FS_XFLAG_DAX flag previously set by FUSE client work*. Then it becomes a *mandatory* requirement when implementing FUSE server. i.e., it becomes part of the FUSE protocol rather than implementation specific. FUSE server can still implement some other algorithm deciding whether to set ATTR_DAX or not, though it must set ATTR_DAX once the backend file is flagged with FS_XFLAG_DAX.
Besides, as you said, FUSE server needs to add one extra GETFLAGS/FSGETXATTR ioctl per LOOKUP in this case. To eliminate this cost, we could negotiate the per-file DAX capability during FUSE_INIT. Only when the per-file DAX capability is negotiated, will the FUSE server do extra GETFLAGS/FSGETXATTR ioctl and set ATTR_DAX flag when doing LOOKUP.
Personally, I prefer the second way, i.e., by extending LOOKUP (adding ATTR_DAX), to eliminate the extra round trip.
Quoted 7 lines
If we were only implementing A, then server does not have a way to tell client to enable DAX. Server can either look at
FS_XFLAG_DAXand decide to enable DAX or use some other property. Given queryingFS_XFLAG_DAXwill be an extraioctl()on every inode lookup/getattr, it probably will be a server option. But enabling on server does not mean client will enable it.
As I said previously, it can be negotiated whether this per-file DAX capability is needed. Guest can advertise this capability when '-o dax=inode' is configured.
Quoted 6 lines
I think my primary concern with this patch right now is trying to figure out which requirement we are trying to cater to first and how to connect server and client well so they both understand what mode they are operating in and interact well.
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerQuoted 64 lines · JeffleXu
Quoted 19 lines · Vivek Goyal
Quoted 17 lines · Jeffle Xu
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
There are three related scenarios:
1. Alloc inode: get per-file DAX flag from fuse_attr.flags. (patch 3)
2. Per-file DAX flag changes when the file has been opened. (patch 3) In this case, the dentry and inode are all marked asDONT_CACHE, and the DAX state won't be updated until the file is closed and reopened later.
3. Users can change the per-file DAX flag inside the guest by chattr(1). (patch 4)
4. Create new files under directories with DAX enabled. When creating new files in ext4/xfs on host, the new created files will inherit the per-file DAX flag from the directory, and thus the new created files in virtiofs will also inherit the per-file DAX flag if the fuse server derives fuse_attr.flags from the underlying ext4/xfs inode's per-file DAX flag.Quoted 6 lines
Thinking little bit more about this from requirement perspective. I think we are trying to address two use cases here.
A. Client does not know which files DAX should be used on. Only server
knows it and server passes this information to client. I suspect that's your primary use case.Yes, this is the starting point of this patch set.
Quoted 11 lines
B. Client is driving which files are supposed to be using DAX. This is
exactly same as the model ext4/xfs are using by storing a persistent flag on inode.Current patches seem to be a hybrid of both approach A and B.
If we were to implement B, then fuse client probably needs to have the capability to query
FS_XFLAG_DAXon inode and decide whether to enable DAX or not. (Without extra round trip). Or know it indirectly by extending GETATTR and requesting this explicitly.If guest queries if the file is DAX capable or not by an extra GETATTR, I'm afraid this will add extra round trip.
Or if we add the DAX flag (
ATTR_DAX) by extending LOOKUP, as done by this patch set, then the FUSE server needs to setATTR_DAXaccording to theFS_XFLAG_DAXof the backend files, *to make theFS_XFLAG_DAXflag previously set by FUSE client work*. Then it becomes a *mandatory* requirement when implementing FUSE server. i.e., it becomes part of the FUSE protocol rather than implementation specific. FUSE server can still implement some other algorithm deciding whether to setATTR_DAXor not, though it must setATTR_DAXonce the backend file is flagged withFS_XFLAG_DAX.Besides, as you said, FUSE server needs to add one extra GETFLAGS/FSGETXATTR ioctl per LOOKUP in this case. To eliminate this cost, we could negotiate the per-file DAX capability during
FUSE_INIT. Only when the per-file DAX capability is negotiated, will the FUSE server do extra GETFLAGS/FSGETXATTR ioctl and setATTR_DAXflag when doing LOOKUP.
Quoted 2 lines
Personally, I prefer the second way, i.e., by extending LOOKUP (adding
ATTR_DAX), to eliminate the extra round trip.
Negotiating a fuse feature say FUSE_FS_XFLAG_DAX makes sense. If client is mounted with "-o dax=inode", then client will negotitate this feature and if server does not support it, mount can fail.
But this probably will be binding on server that it needs to return the state of FS_XFLAG_DAX attr on inode upon lookup/getattr. I don't this will allow server to implement its own separate policy which does not follow FS_XFLAG_DAX xattr.
IOW, I don't think server can choose to implement its own policy for enabling dax for "-o dax=inode".
If there is really a need to for something new where server needs to dynamically decide which inodes should use dax (and not use FS_XFLAG_FS), I guess that probably should be a separate mount option say "-o dax=server" and it negotiates a separate feature say FUSE_DAX_SERVER. Once that's negotiated, now both client and server know that DAX will be used on files as determined by server and not necessarily by using file attr FS_XFLAG_DAX.
So is "dax=inode" enough for your needs? What's your requirement, can you give little bit of more details.
Thanks
Vivek
Quoted 14 lines
Quoted 8 lines
If we were only implementing A, then server does not have a way to tell client to enable DAX. Server can either look at
FS_XFLAG_DAXand decide to enable DAX or use some other property. Given queryingFS_XFLAG_DAXwill be an extraioctl()on every inode lookup/getattr, it probably will be a server option. But enabling on server does not mean client will enable it.As I said previously, it can be negotiated whether this per-file DAX capability is needed. Guest can advertise this capability when '-o dax=inode' is configured.
Quoted 1 line
Quoted 11 lines
Quoted 5 lines
I think my primary concern with this patch right now is trying to figure out which requirement we are trying to cater to first and how to connect server and client well so they both understand what mode they are operating in and interact well.
Signature · 2 lines
Thanks, Jeffle
_______________________________________________
Virtualization mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Quoted 68 lines · Vivek Goyal
Quoted 64 lines · JeffleXu
Quoted 19 lines · Vivek Goyal
Quoted 17 lines · Jeffle Xu
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
There are three related scenarios:
1. Alloc inode: get per-file DAX flag from fuse_attr.flags. (patch 3)
2. Per-file DAX flag changes when the file has been opened. (patch 3) In this case, the dentry and inode are all marked asDONT_CACHE, and the DAX state won't be updated until the file is closed and reopened later.
3. Users can change the per-file DAX flag inside the guest by chattr(1). (patch 4)
4. Create new files under directories with DAX enabled. When creating new files in ext4/xfs on host, the new created files will inherit the per-file DAX flag from the directory, and thus the new created files in virtiofs will also inherit the per-file DAX flag if the fuse server derives fuse_attr.flags from the underlying ext4/xfs inode's per-file DAX flag.Quoted 6 lines
Thinking little bit more about this from requirement perspective. I think we are trying to address two use cases here.
A. Client does not know which files DAX should be used on. Only server
knows it and server passes this information to client. I suspect that's your primary use case.Yes, this is the starting point of this patch set.
Quoted 11 lines
B. Client is driving which files are supposed to be using DAX. This is
exactly same as the model ext4/xfs are using by storing a persistent flag on inode.Current patches seem to be a hybrid of both approach A and B.
If we were to implement B, then fuse client probably needs to have the capability to query
FS_XFLAG_DAXon inode and decide whether to enable DAX or not. (Without extra round trip). Or know it indirectly by extending GETATTR and requesting this explicitly.If guest queries if the file is DAX capable or not by an extra GETATTR, I'm afraid this will add extra round trip.
Or if we add the DAX flag (
ATTR_DAX) by extending LOOKUP, as done by this patch set, then the FUSE server needs to setATTR_DAXaccording to theFS_XFLAG_DAXof the backend files, *to make theFS_XFLAG_DAXflag previously set by FUSE client work*. Then it becomes a *mandatory* requirement when implementing FUSE server. i.e., it becomes part of the FUSE protocol rather than implementation specific. FUSE server can still implement some other algorithm deciding whether to setATTR_DAXor not, though it must setATTR_DAXonce the backend file is flagged withFS_XFLAG_DAX.Besides, as you said, FUSE server needs to add one extra GETFLAGS/FSGETXATTR ioctl per LOOKUP in this case. To eliminate this cost, we could negotiate the per-file DAX capability during
FUSE_INIT. Only when the per-file DAX capability is negotiated, will the FUSE server do extra GETFLAGS/FSGETXATTR ioctl and setATTR_DAXflag when doing LOOKUP.Quoted 2 lines
Personally, I prefer the second way, i.e., by extending LOOKUP (adding
ATTR_DAX), to eliminate the extra round trip.
Quoted 8 lines
Negotiating a fuse feature say
FUSE_FS_XFLAG_DAXmakes sense. If client is mounted with "-o dax=inode", then client will negotitate this feature and if server does not support it, mount can fail.But this probably will be binding on server that it needs to return the state of
FS_XFLAG_DAXattr on inode upon lookup/getattr. I don't this will allow server to implement its own separate policy which does not followFS_XFLAG_DAXxattr.
That means the backend fs must be ext4/xfs supporting per-file DAX feature.
If given more right to construct its own policy, FUSE server could support per-file DAX upon other backend fs, though it will always fail when virtiofs wants to set FS_XFLAG_DAX inside guest.
Quoted 12 lines
IOW, I don't think server can choose to implement its own policy for enabling dax for "-o dax=inode".
If there is really a need to for something new where server needs to dynamically decide which inodes should use dax (and not use
FS_XFLAG_FS), I guess that probably should be a separate mount option say "-o dax=server" and it negotiates a separate feature sayFUSE_DAX_SERVER. Once that's negotiated, now both client and server know that DAX will be used on files as determined by server and not necessarily by using file attrFS_XFLAG_DAX.
Quoted 2 lines
So is "dax=inode" enough for your needs? What's your requirement, can you give little bit of more details.
In our use case, the backend fs is something like SquashFS on host. The content of the file on host is downloaded *as needed*. When the file is not completely ready (completely downloaded), the guest will follow the normal IO routine, i.e., by FUSE_READ/FUSE_WRITE request. While the file is completely ready, per-file DAX is enabled for this file. IOW the FUSE server need to dynamically decide if per-file DAX shall be enabled, depending on if the file is completely downloaded.
Our strategy and design is still under discussion and may change. Any comment and discussion is welcomed.
Quoted 26 lines
Quoted 26 lines
Quoted 9 lines
If we were only implementing A, then server does not have a way to tell client to enable DAX. Server can either look at
FS_XFLAG_DAXand decide to enable DAX or use some other property. Given queryingFS_XFLAG_DAXwill be an extraioctl()on every inode lookup/getattr, it probably will be a server option. But enabling on server does not mean client will enable it.As I said previously, it can be negotiated whether this per-file DAX capability is needed. Guest can advertise this capability when '-o dax=inode' is configured.
Quoted 6 lines
I think my primary concern with this patch right now is trying to figure out which requirement we are trying to cater to first and how to connect server and client well so they both understand what mode they are operating in and interact well.
Signature · 2 lines
Thanks, Jeffle
Signature · 6 lines
Thanks, Jeffle _______________________________________________ Virtualization mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/virtualization
Vivek Goyal
maintainerQuoted 75 lines · JeffleXu
Quoted 72 lines · Vivek Goyal
Quoted 64 lines · JeffleXu
Quoted 19 lines · Vivek Goyal
Quoted 17 lines · Jeffle Xu
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
There are three related scenarios:
1. Alloc inode: get per-file DAX flag from fuse_attr.flags. (patch 3)
2. Per-file DAX flag changes when the file has been opened. (patch 3) In this case, the dentry and inode are all marked asDONT_CACHE, and the DAX state won't be updated until the file is closed and reopened later.
3. Users can change the per-file DAX flag inside the guest by chattr(1). (patch 4)
4. Create new files under directories with DAX enabled. When creating new files in ext4/xfs on host, the new created files will inherit the per-file DAX flag from the directory, and thus the new created files in virtiofs will also inherit the per-file DAX flag if the fuse server derives fuse_attr.flags from the underlying ext4/xfs inode's per-file DAX flag.Quoted 6 lines
Thinking little bit more about this from requirement perspective. I think we are trying to address two use cases here.
A. Client does not know which files DAX should be used on. Only server
knows it and server passes this information to client. I suspect that's your primary use case.Yes, this is the starting point of this patch set.
Quoted 11 lines
B. Client is driving which files are supposed to be using DAX. This is
exactly same as the model ext4/xfs are using by storing a persistent flag on inode.Current patches seem to be a hybrid of both approach A and B.
If we were to implement B, then fuse client probably needs to have the capability to query
FS_XFLAG_DAXon inode and decide whether to enable DAX or not. (Without extra round trip). Or know it indirectly by extending GETATTR and requesting this explicitly.If guest queries if the file is DAX capable or not by an extra GETATTR, I'm afraid this will add extra round trip.
Or if we add the DAX flag (
ATTR_DAX) by extending LOOKUP, as done by this patch set, then the FUSE server needs to setATTR_DAXaccording to theFS_XFLAG_DAXof the backend files, *to make theFS_XFLAG_DAXflag previously set by FUSE client work*. Then it becomes a *mandatory* requirement when implementing FUSE server. i.e., it becomes part of the FUSE protocol rather than implementation specific. FUSE server can still implement some other algorithm deciding whether to setATTR_DAXor not, though it must setATTR_DAXonce the backend file is flagged withFS_XFLAG_DAX.Besides, as you said, FUSE server needs to add one extra GETFLAGS/FSGETXATTR ioctl per LOOKUP in this case. To eliminate this cost, we could negotiate the per-file DAX capability during
FUSE_INIT. Only when the per-file DAX capability is negotiated, will the FUSE server do extra GETFLAGS/FSGETXATTR ioctl and setATTR_DAXflag when doing LOOKUP.Quoted 2 lines
Personally, I prefer the second way, i.e., by extending LOOKUP (adding
ATTR_DAX), to eliminate the extra round trip.Negotiating a fuse feature say
FUSE_FS_XFLAG_DAXmakes sense. If client is mounted with "-o dax=inode", then client will negotitate this feature and if server does not support it, mount can fail.
Quoted 6 lines
Quoted 4 lines
But this probably will be binding on server that it needs to return the state of
FS_XFLAG_DAXattr on inode upon lookup/getattr. I don't this will allow server to implement its own separate policy which does not followFS_XFLAG_DAXxattr.That means the backend fs must be ext4/xfs supporting per-file DAX feature.
Yes. This probably will be a requirement because we are dependent on file attr FS_XFLAG_DAX to decide whether to enable DAX or not. And if underlying filesystem does not support this attr, then it will not work.
Quoted 20 lines
If given more right to construct its own policy, FUSE server could support per-file DAX upon other backend fs, though it will always fail when virtiofs wants to set
FS_XFLAG_DAXinside guest.Quoted 12 lines
IOW, I don't think server can choose to implement its own policy for enabling dax for "-o dax=inode".
If there is really a need to for something new where server needs to dynamically decide which inodes should use dax (and not use
FS_XFLAG_FS), I guess that probably should be a separate mount option say "-o dax=server" and it negotiates a separate feature sayFUSE_DAX_SERVER. Once that's negotiated, now both client and server know that DAX will be used on files as determined by server and not necessarily by using file attrFS_XFLAG_DAX.Quoted 2 lines
So is "dax=inode" enough for your needs? What's your requirement, can you give little bit of more details.
Quoted 7 lines
In our use case, the backend fs is something like SquashFS on host. The content of the file on host is downloaded *as needed*. When the file is not completely ready (completely downloaded), the guest will follow the normal IO routine, i.e., by FUSE_READ/FUSE_WRITE request. While the file is completely ready, per-file DAX is enabled for this file. IOW the FUSE server need to dynamically decide if per-file DAX shall be enabled, depending on if the file is completely downloaded.
So you don't want to enable DAX yet because guest might fault on a section of file which has not been downloaded yet?
I am wondering if somehow user fault handling can help with this. If we could handle faults for this file in user space, then you should be able to download that particular page[s] and resolve the fault?
Thanks
Vivek
Vivek Goyal
maintainerOn Wed, Jul 21, 2021 at 08:48:57AM -0400, Vivek Goyal wrote:
[..]
Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
So is "dax=inode" enough for your needs? What's your requirement, can you give little bit of more details.
In our use case, the backend fs is something like SquashFS on host. The content of the file on host is downloaded *as needed*. When the file is not completely ready (completely downloaded), the guest will follow the normal IO routine, i.e., by FUSE_READ/FUSE_WRITE request. While the file is completely ready, per-file DAX is enabled for this file. IOW the FUSE server need to dynamically decide if per-file DAX shall be enabled, depending on if the file is completely downloaded.
Quoted 7 lines
So you don't want to enable DAX yet because guest might fault on a section of file which has not been downloaded yet?
I am wondering if somehow user fault handling can help with this. If we could handle faults for this file in user space, then you should be able to download that particular page[s] and resolve the fault?
Stefan mentioned that can't we block when fuse mmap request comes in and download corresponding section of file. Or do whatever you are doing in FUSE_READ.
IOW, even if you enable dax in your use case on all files, FUSE_SETUPMAPPING request will give you control to make sure file section being mmaped has been downloaded.
Vivek
Quoted 21 lines · Vivek Goyal
On Wed, Jul 21, 2021 at 08:48:57AM -0400, Vivek Goyal wrote:
[..]Quoted 11 lines
Quoted 10 lines
Quoted 2 lines
So is "dax=inode" enough for your needs? What's your requirement, can you give little bit of more details.
In our use case, the backend fs is something like SquashFS on host. The content of the file on host is downloaded *as needed*. When the file is not completely ready (completely downloaded), the guest will follow the normal IO routine, i.e., by FUSE_READ/FUSE_WRITE request. While the file is completely ready, per-file DAX is enabled for this file. IOW the FUSE server need to dynamically decide if per-file DAX shall be enabled, depending on if the file is completely downloaded.
Quoted 7 lines
So you don't want to enable DAX yet because guest might fault on a section of file which has not been downloaded yet?
I am wondering if somehow user fault handling can help with this. If we could handle faults for this file in user space, then you should be able to download that particular page[s] and resolve the fault?
Quoted 8 lines
Stefan mentioned that can't we block when fuse mmap request comes in and download corresponding section of file. Or do whatever you are doing in
FUSE_READ.IOW, even if you enable dax in your use case on all files,
FUSE_SETUPMAPPINGrequest will give you control to make sure file section being mmaped has been downloaded.
Sorry for the late reply. I missed this mail as it is classified into the mailing list folder.
The idea you mentioned may works. Anyway, the implementation details of the FUSE server is not strongly binding to the FUSE protocol changes in kernel. The protocol only requires that FUSE client shall be able to store FS_DAX_FL attr persistently in *some way*. The changes in kernel shall be general, no matter whether the FUSE server is FS_DAX_FL attr based or something else.
Signature · 2 lines
Thanks, Jeffle