fuse,virtiofs: support per-file DAX
Subsystems (4)
v6 cover fuse,virtiofs: support per-file DAX
changes since v5:
Overall Design Changes:
1. virtiofsd now supports ioctl (only FS_IOC_SETFLAGS and FS_IOC_FSSETXATTR), so that users inside guest could set/clear persistent inode flags now. (FUSE kernel module has already supported .ioctl(), virtiofsd need to suuport it.)
2. When FUSE client is mounted with '-o dax=inode', it indicates that whether DAX shall be enabled or not for one specific file is completely determined by FUSE server while FUSE client has no say on it, and the decision whether DAX shall be enabled or not for specific file is communicated through FUSE_ATTR_DAX flag of FUSE protocol. The algorithm used by virtiofsd to determine whether DAX shall be enabled or not is totally implementation specific, and thus the following scenario may exist: users inside guest has already set related persistent inode flag (i.e. FS_XFLAG_DAX) on corresponding file but FUSE server finnaly decides not to enable DAX for this file. This slight semantic difference is documented in patch 7. Also because of this, d_mark_dontcache() is not called when FS_IOC_SETFLAGS/FS_IOC_FSSETXATTR ioctl is done inside guest. It's delayed to be done if the FUSE_ATTR_DAX flag **indeed** changes (as showed in patch 6).
3. patch 1: slightly modify logic of fuse_should_enable_dax()
4. patch 4: add back negotiation during FUSE_INIT. FUSE client shall advertise to FUSE server that it's in per-file DAX mode, so that FUSE server may omit querying persistent inode flags on host if FUSE client is not mounted in per-file DAX mode, giving querying persistent inode flags could be quite expensive.
chanegs since v4:
- drop support for setting/clearing FS_DAX inside guest
- and thus drop the negotiation phase during FUSE_INIT
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
Any comment is welcome.
[1] commit 9cb20f94afcd ("fs/ext4: Make DAX mount option a tri-state")
[2] commit 02beb2686ff9 ("fs/xfs: Make DAX mount option a tri-state")
[Purpose]
DAX may be limited in some specific situation. When the number of usable DAX windows is under watermark, the recalim routine will be triggered to reclaim some DAX windows. It may have a negative impact on the performance, since some processes may need to wait for DAX windows to be recalimed and reused then. To mitigate the performance degradation, the overall DAX window need to be expanded larger.
However, simply expanding the DAX window may not be a good deal in some scenario. To maintain one DAX window chunk (i.e., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest, which is greater than the memory footprint if it uses guest page cache when DAX disabled. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Per-file DAX feature is introduced to address this issue, by offering a finer grained control for dax to users, trying to achieve a balance between performance and memory overhead.
[Note]
When the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state, since dynamic switching needs to switch a_ops atomiclly. Ira Weiny had ever implemented a so called i_aops_sem lock [3] but eventually gave up since the complexity of the implementation [4][5][6][7].
Hence mark the inode and corresponding dentries as DONE_CACHE once the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.
In summary, when the per-file DAX hint changes for an *opened* file, the DAX state of the file won't be updated until this file is closed and reopened later. This is also how ext4/xfs per-file DAX works.
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://patchwork.kernel.org/project/xfs/cover/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/
[6] https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/
[7] https://lore.kernel.org/lkml/20200403182904.GP80283@magnolia/
changes since v3:
- bug fix (patch 6): s/"IS_DAX(inode) != newdax"/"!!IS_DAX(inode) != newdax"
- during FUSE_INIT, advertise capability for per-file DAX only when mounted as "-o dax=inode" (patch 4)
changes since v2:
- modify fuse_show_options() accordingly to make it compatible with new tri-state mount option (patch 2)
- extract FUSE protocol changes into one separate patch (patch 3)
- FUSE server/client need to negotiate if they support per-file DAX (patch 4)
- extract DONT_CACHE logic into patch 6/7
v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/linux-fsdevel/[email protected]/
v3: https://www.spinics.net/lists/linux-fsdevel/msg200852.html
v2: https://www.spinics.net/lists/linux-fsdevel/msg199584.html
v1: https://www.spinics.net/lists/linux-virtualization/msg51008.html
Jeffle Xu (7):
fuse: add fuse_should_enable_dax() helper
fuse: make DAX mount option a tri-state
fuse: support per-file DAX in fuse protocol
fuse: negotiate per-file DAX in FUSE_INIT
fuse: enable per-file DAX
fuse: mark inode DONT_CACHE when per-file DAX hint changes Documentation/filesystem/dax: record DAX on virtiofs
Documentation/filesystems/dax.rst | 20 +++++++++++++++-- fs/fuse/dax.c | 36 ++++++++++++++++++++++++++++--- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 19 ++++++++++++---- fs/fuse/inode.c | 17 +++++++++++---- fs/fuse/virtio_fs.c | 16 ++++++++++++-- include/uapi/linux/fuse.h | 9 +++++++- 7 files changed, 103 insertions(+), 18 deletions(-)
Signature · 1 line
2.27.0
PATCH v6 1/7 fuse: add fuse_should_enable_dax() helper
This is in prep for following per-file DAX checking.
--- fs/fuse/dax.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
fs/fuse/dax.c +10 -2
@@ -1332,11 +1332,19 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -void fuse_dax_inode_init (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); -if (!fc->dax) +if (fc->dax) +return true; + +return false; +} + +void fuse_dax_inode_init (struct inode *inode) +{ +if (!fuse_should_enable_dax (inode))return ; inode->i_flags |=S_DAX ;
Signature · 1 line
2.27.0
PATCH v6 2/7 fuse: make DAX mount option a tri-state
In reply to Jeffle XuWe add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
By the time this patch is applied, 'inode' mode is actually equal to 'always' mode, before the per-file DAX flag is introduced in the following patch.
--- fs/fuse/dax.c | 19 ++++++++++++++++--- fs/fuse/fuse_i.h | 14 ++++++++++++-- fs/fuse/inode.c | 10 +++++++--- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- 4 files changed, 49 insertions(+), 10 deletions(-)
fs/fuse/dax.c +16 -3
@@ -1284,11 +1284,14 @@ static int fuse_dax_mem_range_init(struct fuse_conn_dax *fcd)return ret; } -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev) +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode dax_mode, +struct dax_device *dax_dev) {struct fuse_conn_dax *fcd;int err; + fc->dax_mode = dax_mode; +if (!dax_dev)return 0 ;
@@ -1335,11 +1338,21 @@ static const struct address_space_operations fuse_dax_file_aops = {static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); +unsigned int dax_mode = fc->dax_mode; + +if (dax_mode ==FUSE_DAX_NEVER ) +return false; -if (fc->dax) +/* +* If 'dax=always/inode', fc->dax couldn't be NULL even when fuse +* daemon doesn't support DAX, since the mount routine will fail +* early in this case. +*/ +if (dax_mode ==FUSE_DAX_ALWAYS )return true; -return false; +/* dax_mode == FUSE_DAX_INODE */ +return true; }void fuse_dax_inode_init (struct inode *inode)
fs/fuse/fuse_i.h +12 -2
@@ -480,6 +480,12 @@ struct fuse_dev {struct list_head entry; }; +enum fuse_dax_mode { +FUSE_DAX_INODE , +FUSE_DAX_ALWAYS , +FUSE_DAX_NEVER , +}; +struct fuse_fs_context {int fd;struct file *file;
@@ -497,7 +503,7 @@ struct fuse_fs_context { bool no_control:1; bool no_force_umount:1; bool legacy_opts_show:1; - bool dax:1; +enum fuse_dax_mode dax_mode;unsigned int max_read;unsigned int blksize;const char *subtype;
@@ -802,6 +808,9 @@ struct fuse_conn {struct list_head devices;#ifdef CONFIG_FUSE_DAX +/* dax mode: FUSE_DAX_* (always, never or per-file) */ +enum fuse_dax_mode dax_mode; +/* Dax specific conn data, non-NULL if DAX is enabled */ struct fuse_conn_dax *dax;#endif
@@ -1255,7 +1264,8 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);int fuse_dax_mmap (struct file *file,struct vm_area_struct *vma);int fuse_dax_break_layouts (struct inode *inode,u64 dmap_start,u64 dmap_end); -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev); +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode mode, +struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode);
fs/fuse/inode.c +7 -3
@@ -742,8 +742,12 @@ static int fuse_show_options(struct seq_file *m, struct dentry *root)seq_printf (m,",blksize=%lu" , sb->s_blocksize); }#ifdef CONFIG_FUSE_DAX -if (fc->dax) -seq_puts (m,",dax" ); +if (fc->dax_mode ==FUSE_DAX_ALWAYS ) +seq_puts (m,",dax=always" ); +else if (fc->dax_mode ==FUSE_DAX_NEVER ) +seq_puts (m,",dax=never" ); +else if (fc->dax_mode ==FUSE_DAX_INODE ) +seq_puts (m,",dax=inode" );#endif return 0 ;
@@ -1493,7 +1497,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) sb->s_subtype = ctx->subtype; ctx->subtype =NULL ;if (IS_ENABLED (CONFIG_FUSE_DAX )) { - err =fuse_dax_conn_alloc (fc, ctx->dax_dev); + err =fuse_dax_conn_alloc (fc, ctx->dax_mode, ctx->dax_dev);if (err) goto err; }
fs/fuse/virtio_fs.c +14 -2
@@ -88,12 +88,21 @@ struct virtio_fs_req_work {static int virtio_fs_enqueue_req (struct virtio_fs_vq *fsvq,struct fuse_req *req,bool in_flight); +static const struct constant_table dax_param_enums[] = { + {"inode" ,FUSE_DAX_INODE }, + {"always" ,FUSE_DAX_ALWAYS }, + {"never" ,FUSE_DAX_NEVER }, + {} +}; +enum {OPT_DAX , +OPT_DAX_ENUM , };static const struct fs_parameter_spec virtio_fs_parameters[] = {fsparam_flag ("dax" ,OPT_DAX ), +fsparam_enum ("dax" ,OPT_DAX_ENUM , dax_param_enums), {} };
@@ -110,7 +119,10 @@ static int virtio_fs_parse_param(struct fs_context *fsc,switch (opt) {case OPT_DAX : - ctx->dax =1 ; + ctx->dax_mode =FUSE_DAX_ALWAYS ; +break ; +case OPT_DAX_ENUM : + ctx->dax_mode = result.uint_32;break ;default :return -EINVAL ;
@@ -1326,7 +1338,7 @@ static int virtio_fs_fill_super(struct super_block *sb, struct fs_context *fsc)/* virtiofs allocates and installs its own fuse devices */ ctx->fudptr =NULL ; -if (ctx->dax) { +if (ctx->dax_mode !=FUSE_DAX_NEVER ) {if (!fs->dax_dev) { err = -EINVAL ; pr_err("virtio-fs: dax can't be enabled as filesystem"
Signature · 1 line
2.27.0
Vivek Goyal
maintainerRe: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Hi Jeffle,
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
Quoted 44 lines
By the time this patch is applied, 'inode' mode is actually equal to 'always' mode, before the per-file DAX flag is introduced in the following patch.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 19 ++++++++++++++++--- fs/fuse/fuse_i.h | 14 ++++++++++++-- fs/fuse/inode.c | 10 +++++++--- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- 4 files changed, 49 insertions(+), 10 deletions(-)fs/fuse/dax.c +9
-2@@ -1284,11 +1284,14 @@ static int fuse_dax_mem_range_init(struct fuse_conn_dax *fcd)return ret; } -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev) +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode dax_mode, +struct dax_device *dax_dev) {struct fuse_conn_dax *fcd;int err; + fc->dax_mode = dax_mode; +if (!dax_dev)return 0 ;@@ -1335,11 +1338,21 @@ static const struct address_space_operations fuse_dax_file_aops = {static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); +unsigned int dax_mode = fc->dax_mode; + +if (dax_mode ==FUSE_DAX_NEVER ) +return false; -if (fc->dax) +/*
fs/fuse/dax.c static const struct address_space_operations fuse_dax_file_aops = { · line 1355
Quoted 10 lines
fs/fuse/dax.c +7
-1+* If 'dax=always/inode', fc->dax couldn't be NULL even when fuse +* daemon doesn't support DAX, since the mount routine will fail +* early in this case. +*/ +if (dax_mode ==FUSE_DAX_ALWAYS )return true; -return false; +/* dax_mode == FUSE_DAX_INODE */ +return true;
So as of this patch except FUSE_DAX_NEVER return true and this will change in later patches for FUSE_DAX_INODE? If that's the case, keep it simple in this patch and change it later in the patch series.
fuse_should_enable_dax()
{
if (dax_mode == FUSE_DAX_NEVER)
return false;
return true;
}Quoted 54 lines
fs/fuse/dax.c +0
-0}void fuse_dax_inode_init (struct inode *inode)fs/fuse/fuse_i.h +12
-2@@ -480,6 +480,12 @@ struct fuse_dev {struct list_head entry; }; +enum fuse_dax_mode { +FUSE_DAX_INODE , +FUSE_DAX_ALWAYS , +FUSE_DAX_NEVER , +}; +struct fuse_fs_context {int fd;struct file *file;@@ -497,7 +503,7 @@ struct fuse_fs_context { bool no_control:1; bool no_force_umount:1; bool legacy_opts_show:1; - bool dax:1; +enum fuse_dax_mode dax_mode;unsigned int max_read;unsigned int blksize;const char *subtype;@@ -802,6 +808,9 @@ struct fuse_conn {struct list_head devices;#ifdef CONFIG_FUSE_DAX +/* dax mode: FUSE_DAX_* (always, never or per-file) */ +enum fuse_dax_mode dax_mode; +/* Dax specific conn data, non-NULL if DAX is enabled */ struct fuse_conn_dax *dax;#endif @@ -1255,7 +1264,8 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);int fuse_dax_mmap (struct file *file,struct vm_area_struct *vma);int fuse_dax_break_layouts (struct inode *inode,u64 dmap_start,u64 dmap_end); -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev); +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode mode, +struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode);fs/fuse/inode.c +0
-0
fs/fuse/inode.c static int fuse_show_options(struct seq_file *m, struct dentry *root) · line 746
Quoted 7 lines
fs/fuse/inode.c +2
-2@@ -742,8 +742,12 @@ static int fuse_show_options(struct seq_file *m, struct dentry *root)seq_printf (m,",blksize=%lu" , sb->s_blocksize); }#ifdef CONFIG_FUSE_DAX -if (fc->dax) -seq_puts (m,",dax" ); +if (fc->dax_mode ==FUSE_DAX_ALWAYS ) +seq_puts (m,",dax=always" );
So if somebody mounts with "-o dax" then kernel previous to this change will show "dax" and kernel after this change will show "dax=always"?
How about not change the behavior. Keep a mode say FUSE_DAX_LEGACY which will be set when user specifies "-o dax". Internally FUSE_DAX_LEGACY and FUSE_DAX_ALWAYS will be same.
if (fc->dax_mode == FUSE_DAX_LEGACY) seq_puts(m, ",dax");
Thanks
Vivek
Quoted 66 lines
fs/fuse/inode.c +5
-1+ else if (fc->dax_mode == FUSE_DAX_NEVER) + seq_puts(m, ",dax=never"); + else if (fc->dax_mode == FUSE_DAX_INODE) +seq_puts (m,",dax=inode" );#endif return 0 ;@@ -1493,7 +1497,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) sb->s_subtype = ctx->subtype; ctx->subtype =NULL ;if (IS_ENABLED (CONFIG_FUSE_DAX )) { - err =fuse_dax_conn_alloc (fc, ctx->dax_dev); + err =fuse_dax_conn_alloc (fc, ctx->dax_mode, ctx->dax_dev);if (err) goto err; }fs/fuse/virtio_fs.c +14
-2@@ -88,12 +88,21 @@ struct virtio_fs_req_work {static int virtio_fs_enqueue_req (struct virtio_fs_vq *fsvq,struct fuse_req *req,bool in_flight); +static const struct constant_table dax_param_enums[] = { + {"inode" ,FUSE_DAX_INODE }, + {"always" ,FUSE_DAX_ALWAYS }, + {"never" ,FUSE_DAX_NEVER }, + {} +}; +enum {OPT_DAX , +OPT_DAX_ENUM , };static const struct fs_parameter_spec virtio_fs_parameters[] = {fsparam_flag ("dax" ,OPT_DAX ), +fsparam_enum ("dax" ,OPT_DAX_ENUM , dax_param_enums), {} };@@ -110,7 +119,10 @@ static int virtio_fs_parse_param(struct fs_context *fsc,switch (opt) {case OPT_DAX : - ctx->dax =1 ; + ctx->dax_mode =FUSE_DAX_ALWAYS ; +break ; +case OPT_DAX_ENUM : + ctx->dax_mode = result.uint_32;break ;default :return -EINVAL ;@@ -1326,7 +1338,7 @@ static int virtio_fs_fill_super(struct super_block *sb, struct fs_context *fsc)/* virtiofs allocates and installs its own fuse devices */ ctx->fudptr =NULL ; -if (ctx->dax) { +if (ctx->dax_mode !=FUSE_DAX_NEVER ) {if (!fs->dax_dev) { err = -EINVAL ; pr_err("virtio-fs: dax can't be enabled as filesystem"Signature · 1 line
2.27.0
Re: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 6 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Quoted 6 lines
Hi Jeffle,
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
This default behaviour for local filesystem, e.g. ext4/xfs, may be straightforward, since the disk inode will be read into memory during the inode instantiation, and checking for persistent inode attribute shall be realatively cheap, except that the default behaviour has changed from 'dax=never' to 'dax=inode'.
Come back to virtiofs, when neither '-o dax' nor '-o dax=always/never/inode' is specified, and it actually behaves as '-o dax=inode', as long as '-o dax=server/attr' option is not specified for virtiofsd, virtiofsd will always clear FUSE_ATTR_DAX and thus guest will always disable DAX. IOWs, the guest virtiofs atually behaves as '-o dax=never' when neither '-o dax' nor '-o dax=always/never/inode' is specified, and '-o dax=server/attr' option is not specified for virtiofsd.
But I'm okay if we need to change the default behaviour for virtiofs.
Quoted 123 lines
Quoted 45 lines
By the time this patch is applied, 'inode' mode is actually equal to 'always' mode, before the per-file DAX flag is introduced in the following patch.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 19 ++++++++++++++++--- fs/fuse/fuse_i.h | 14 ++++++++++++-- fs/fuse/inode.c | 10 +++++++--- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- 4 files changed, 49 insertions(+), 10 deletions(-)fs/fuse/dax.c +9
-2@@ -1284,11 +1284,14 @@ static int fuse_dax_mem_range_init(struct fuse_conn_dax *fcd)return ret; } -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev) +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode dax_mode, +struct dax_device *dax_dev) {struct fuse_conn_dax *fcd;int err; + fc->dax_mode = dax_mode; +if (!dax_dev)return 0 ;@@ -1335,11 +1338,21 @@ static const struct address_space_operations fuse_dax_file_aops = {static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); +unsigned int dax_mode = fc->dax_mode; + +if (dax_mode ==FUSE_DAX_NEVER ) +return false; -if (fc->dax) +/* fs/fuse/dax.c static const struct address_space_operations fuse_dax_file_aops = { · line 1355
Quoted 10 lines
fs/fuse/dax.c +7
-1+* If 'dax=always/inode', fc->dax couldn't be NULL even when fuse +* daemon doesn't support DAX, since the mount routine will fail +* early in this case. +*/ +if (dax_mode ==FUSE_DAX_ALWAYS )return true; -return false; +/* dax_mode == FUSE_DAX_INODE */ +return true;So as of this patch except
FUSE_DAX_NEVERreturn true and this will change in later patches forFUSE_DAX_INODE? If that's the case, keep it simple in this patch and change it later in the patch series.
fuse_should_enable_dax(){ if (dax_mode == FUSE_DAX_NEVER) return false; return true; }Quoted 56 lines
fs/fuse/dax.c +0
-0}void fuse_dax_inode_init (struct inode *inode)fs/fuse/fuse_i.h +12
-2@@ -480,6 +480,12 @@ struct fuse_dev {struct list_head entry; }; +enum fuse_dax_mode { +FUSE_DAX_INODE , +FUSE_DAX_ALWAYS , +FUSE_DAX_NEVER , +}; +struct fuse_fs_context {int fd;struct file *file;@@ -497,7 +503,7 @@ struct fuse_fs_context { bool no_control:1; bool no_force_umount:1; bool legacy_opts_show:1; - bool dax:1; +enum fuse_dax_mode dax_mode;unsigned int max_read;unsigned int blksize;const char *subtype;@@ -802,6 +808,9 @@ struct fuse_conn {struct list_head devices;#ifdef CONFIG_FUSE_DAX +/* dax mode: FUSE_DAX_* (always, never or per-file) */ +enum fuse_dax_mode dax_mode; +/* Dax specific conn data, non-NULL if DAX is enabled */ struct fuse_conn_dax *dax;#endif @@ -1255,7 +1264,8 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);int fuse_dax_mmap (struct file *file,struct vm_area_struct *vma);int fuse_dax_break_layouts (struct inode *inode,u64 dmap_start,u64 dmap_end); -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev); +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode mode, +struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode);fs/fuse/inode.c +0
-0@@ -742,8 +742,12 @@ static int fuse_show_options(struct seq_file *m, struct dentry *root)seq_printf (m,",blksize=%lu" , sb->s_blocksize); }
Quoted 8 lines
fs/fuse/inode.c static int fuse_show_options(struct seq_file *m, struct dentry *root) · line 746
Quoted 5 lines
fs/fuse/inode.c +2
-2#ifdef CONFIG_FUSE_DAX -if (fc->dax) -seq_puts (m,",dax" ); +if (fc->dax_mode ==FUSE_DAX_ALWAYS ) +seq_puts (m,",dax=always" );So if somebody mounts with "-o dax" then kernel previous to this change will show "dax" and kernel after this change will show "dax=always"?
Yes. It's actually how per-file DAX on ext4/xfs behaves.
Quoted 79 lines
How about not change the behavior. Keep a mode say
FUSE_DAX_LEGACYwhich will be set when user specifies "-o dax". InternallyFUSE_DAX_LEGACYandFUSE_DAX_ALWAYSwill be same.if (fc->dax_mode == FUSE_DAX_LEGACY) seq_puts(m, ",dax");Quoted 66 lines
fs/fuse/inode.c +5
-1+ else if (fc->dax_mode == FUSE_DAX_NEVER) + seq_puts(m, ",dax=never"); + else if (fc->dax_mode == FUSE_DAX_INODE) +seq_puts (m,",dax=inode" );#endif return 0 ;@@ -1493,7 +1497,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) sb->s_subtype = ctx->subtype; ctx->subtype =NULL ;if (IS_ENABLED (CONFIG_FUSE_DAX )) { - err =fuse_dax_conn_alloc (fc, ctx->dax_dev); + err =fuse_dax_conn_alloc (fc, ctx->dax_mode, ctx->dax_dev);if (err) goto err; }fs/fuse/virtio_fs.c +14
-2@@ -88,12 +88,21 @@ struct virtio_fs_req_work {static int virtio_fs_enqueue_req (struct virtio_fs_vq *fsvq,struct fuse_req *req,bool in_flight); +static const struct constant_table dax_param_enums[] = { + {"inode" ,FUSE_DAX_INODE }, + {"always" ,FUSE_DAX_ALWAYS }, + {"never" ,FUSE_DAX_NEVER }, + {} +}; +enum {OPT_DAX , +OPT_DAX_ENUM , };static const struct fs_parameter_spec virtio_fs_parameters[] = {fsparam_flag ("dax" ,OPT_DAX ), +fsparam_enum ("dax" ,OPT_DAX_ENUM , dax_param_enums), {} };@@ -110,7 +119,10 @@ static int virtio_fs_parse_param(struct fs_context *fsc,switch (opt) {case OPT_DAX : - ctx->dax =1 ; + ctx->dax_mode =FUSE_DAX_ALWAYS ; +break ; +case OPT_DAX_ENUM : + ctx->dax_mode = result.uint_32;break ;default :return -EINVAL ;@@ -1326,7 +1338,7 @@ static int virtio_fs_fill_super(struct super_block *sb, struct fs_context *fsc)/* virtiofs allocates and installs its own fuse devices */ ctx->fudptr =NULL ; -if (ctx->dax) { +if (ctx->dax_mode !=FUSE_DAX_NEVER ) {if (!fs->dax_dev) { err = -EINVAL ; pr_err("virtio-fs: dax can't be enabled as filesystem"Signature · 1 line
2.27.0
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerRe: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 20 lines · JeffleXu
Quoted 6 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Quoted 6 lines
Hi Jeffle,
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
Quoted 5 lines
This default behaviour for local filesystem, e.g. ext4/xfs, may be straightforward, since the disk inode will be read into memory during the inode instantiation, and checking for persistent inode attribute shall be realatively cheap, except that the default behaviour has changed from 'dax=never' to 'dax=inode'.
Interesting that ext4/xfs allowed for this behavior change.
Quoted 10 lines
Come back to virtiofs, when neither '-o dax' nor '-o dax=always/never/inode' is specified, and it actually behaves as '-o dax=inode', as long as '-o dax=server/attr' option is not specified for virtiofsd, virtiofsd will always clear
FUSE_ATTR_DAXand thus guest will always disable DAX. IOWs, the guest virtiofs atually behaves as '-o dax=never' when neither '-o dax' nor '-o dax=always/never/inode' is specified, and '-o dax=server/attr' option is not specified for virtiofsd.But I'm okay if we need to change the default behaviour for virtiofs.
This is change of behavior from client's perspective. Even if client did not opt-in for DAX, DAX can be enabled based on server's setting. Not that there is anything wrong with it, but change of behavior part concerns me.
In case of virtiofs, lot of features we are controlling from server. Client typically just calls "mount" and there are not many options users can specify for mount.
Given we already allowed to make client a choice about DAX behavior, I will feel more comfortable that we don't change it and let client request a specific DAX mode and if client does not specify anything, then DAX is not enabled.
Vivek
Quoted 220 lines
Quoted 125 lines
Quoted 47 lines
By the time this patch is applied, 'inode' mode is actually equal to 'always' mode, before the per-file DAX flag is introduced in the following patch.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 19 ++++++++++++++++--- fs/fuse/fuse_i.h | 14 ++++++++++++-- fs/fuse/inode.c | 10 +++++++--- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- 4 files changed, 49 insertions(+), 10 deletions(-)fs/fuse/dax.c +9
-2@@ -1284,11 +1284,14 @@ static int fuse_dax_mem_range_init(struct fuse_conn_dax *fcd)return ret; } -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev) +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode dax_mode, +struct dax_device *dax_dev) {struct fuse_conn_dax *fcd;int err; + fc->dax_mode = dax_mode; +if (!dax_dev)return 0 ;@@ -1335,11 +1338,21 @@ static const struct address_space_operations fuse_dax_file_aops = {static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); +unsigned int dax_mode = fc->dax_mode; + +if (dax_mode ==FUSE_DAX_NEVER ) +return false; -if (fc->dax) +/* fs/fuse/dax.c static const struct address_space_operations fuse_dax_file_aops = { · line 1355
Quoted 10 lines
fs/fuse/dax.c +7
-1+* If 'dax=always/inode', fc->dax couldn't be NULL even when fuse +* daemon doesn't support DAX, since the mount routine will fail +* early in this case. +*/ +if (dax_mode ==FUSE_DAX_ALWAYS )return true; -return false; +/* dax_mode == FUSE_DAX_INODE */ +return true;So as of this patch except
FUSE_DAX_NEVERreturn true and this will change in later patches forFUSE_DAX_INODE? If that's the case, keep it simple in this patch and change it later in the patch series.
fuse_should_enable_dax(){ if (dax_mode == FUSE_DAX_NEVER) return false; return true; }Quoted 56 lines
fs/fuse/dax.c +0
-0}void fuse_dax_inode_init (struct inode *inode)fs/fuse/fuse_i.h +12
-2@@ -480,6 +480,12 @@ struct fuse_dev {struct list_head entry; }; +enum fuse_dax_mode { +FUSE_DAX_INODE , +FUSE_DAX_ALWAYS , +FUSE_DAX_NEVER , +}; +struct fuse_fs_context {int fd;struct file *file;@@ -497,7 +503,7 @@ struct fuse_fs_context { bool no_control:1; bool no_force_umount:1; bool legacy_opts_show:1; - bool dax:1; +enum fuse_dax_mode dax_mode;unsigned int max_read;unsigned int blksize;const char *subtype;@@ -802,6 +808,9 @@ struct fuse_conn {struct list_head devices;#ifdef CONFIG_FUSE_DAX +/* dax mode: FUSE_DAX_* (always, never or per-file) */ +enum fuse_dax_mode dax_mode; +/* Dax specific conn data, non-NULL if DAX is enabled */ struct fuse_conn_dax *dax;#endif @@ -1255,7 +1264,8 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);int fuse_dax_mmap (struct file *file,struct vm_area_struct *vma);int fuse_dax_break_layouts (struct inode *inode,u64 dmap_start,u64 dmap_end); -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev); +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode mode, +struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode);fs/fuse/inode.c +0
-0@@ -742,8 +742,12 @@ static int fuse_show_options(struct seq_file *m, struct dentry *root)seq_printf (m,",blksize=%lu" , sb->s_blocksize); }Quoted 8 lines
fs/fuse/inode.c static int fuse_show_options(struct seq_file *m, struct dentry *root) · line 746
Quoted 5 lines
fs/fuse/inode.c +2
-2#ifdef CONFIG_FUSE_DAX -if (fc->dax) -seq_puts (m,",dax" ); +if (fc->dax_mode ==FUSE_DAX_ALWAYS ) +seq_puts (m,",dax=always" );So if somebody mounts with "-o dax" then kernel previous to this change will show "dax" and kernel after this change will show "dax=always"?
Yes. It's actually how per-file DAX on ext4/xfs behaves.
Quoted 79 lines
How about not change the behavior. Keep a mode say
FUSE_DAX_LEGACYwhich will be set when user specifies "-o dax". InternallyFUSE_DAX_LEGACYandFUSE_DAX_ALWAYSwill be same.if (fc->dax_mode == FUSE_DAX_LEGACY) seq_puts(m, ",dax");Quoted 66 lines
fs/fuse/inode.c +5
-1+ else if (fc->dax_mode == FUSE_DAX_NEVER) + seq_puts(m, ",dax=never"); + else if (fc->dax_mode == FUSE_DAX_INODE) +seq_puts (m,",dax=inode" );#endif return 0 ;@@ -1493,7 +1497,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) sb->s_subtype = ctx->subtype; ctx->subtype =NULL ;if (IS_ENABLED (CONFIG_FUSE_DAX )) { - err =fuse_dax_conn_alloc (fc, ctx->dax_dev); + err =fuse_dax_conn_alloc (fc, ctx->dax_mode, ctx->dax_dev);if (err) goto err; }fs/fuse/virtio_fs.c +14
-2@@ -88,12 +88,21 @@ struct virtio_fs_req_work {static int virtio_fs_enqueue_req (struct virtio_fs_vq *fsvq,struct fuse_req *req,bool in_flight); +static const struct constant_table dax_param_enums[] = { + {"inode" ,FUSE_DAX_INODE }, + {"always" ,FUSE_DAX_ALWAYS }, + {"never" ,FUSE_DAX_NEVER }, + {} +}; +enum {OPT_DAX , +OPT_DAX_ENUM , };static const struct fs_parameter_spec virtio_fs_parameters[] = {fsparam_flag ("dax" ,OPT_DAX ), +fsparam_enum ("dax" ,OPT_DAX_ENUM , dax_param_enums), {} };@@ -110,7 +119,10 @@ static int virtio_fs_parse_param(struct fs_context *fsc,switch (opt) {case OPT_DAX : - ctx->dax =1 ; + ctx->dax_mode =FUSE_DAX_ALWAYS ; +break ; +case OPT_DAX_ENUM : + ctx->dax_mode = result.uint_32;break ;default :return -EINVAL ;@@ -1326,7 +1338,7 @@ static int virtio_fs_fill_super(struct super_block *sb, struct fs_context *fsc)/* virtiofs allocates and installs its own fuse devices */ ctx->fudptr =NULL ; -if (ctx->dax) { +if (ctx->dax_mode !=FUSE_DAX_NEVER ) {if (!fs->dax_dev) { err = -EINVAL ; pr_err("virtio-fs: dax can't be enabled as filesystem"Signature · 1 line
2.27.0Signature · 2 lines
Thanks, Jeffle
Re: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 45 lines · Vivek Goyal
Quoted 20 lines · JeffleXu
Quoted 6 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Quoted 6 lines
Hi Jeffle,
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
Quoted 5 lines
This default behaviour for local filesystem, e.g. ext4/xfs, may be straightforward, since the disk inode will be read into memory during the inode instantiation, and checking for persistent inode attribute shall be realatively cheap, except that the default behaviour has changed from 'dax=never' to 'dax=inode'.
Interesting that ext4/xfs allowed for this behavior change.
Quoted 10 lines
Come back to virtiofs, when neither '-o dax' nor '-o dax=always/never/inode' is specified, and it actually behaves as '-o dax=inode', as long as '-o dax=server/attr' option is not specified for virtiofsd, virtiofsd will always clear
FUSE_ATTR_DAXand thus guest will always disable DAX. IOWs, the guest virtiofs atually behaves as '-o dax=never' when neither '-o dax' nor '-o dax=always/never/inode' is specified, and '-o dax=server/attr' option is not specified for virtiofsd.But I'm okay if we need to change the default behaviour for virtiofs.
This is change of behavior from client's perspective. Even if client did not opt-in for DAX, DAX can be enabled based on server's setting. Not that there is anything wrong with it, but change of behavior part concerns me.
Quoted 9 lines
In case of virtiofs, lot of features we are controlling from server. Client typically just calls "mount" and there are not many options users can specify for mount.
Given we already allowed to make client a choice about DAX behavior, I will feel more comfortable that we don't change it and let client request a specific DAX mode and if client does not specify anything, then DAX is not enabled.
Hi Vivek,
How about the following design about the default behavior to move this patchset forward, considering the discussion on another thread [1]?
- guest side: '-o dax=inode' acts as the default, keeping consistent with xfs/ext4
- virtiofsd: the default behavior is like, neither file size based policy ('dax=server') nor persistent inode flags based policy ('dax=attr') is used, though virtiofsd indeed advertises that it supports per inode DAX feature (so that it won't fail FUSE_INIT negotiation phase when guest advertises dax=inode by default)... In fact, it acts like 'dax=never' in this case.
Then when guest opts-in and specifies '-o dax=inode' manually, then it shall realize that proper configuration for virtiofsd is also needed (-o dax=server|attr).
[1] https://www.spinics.net/lists/linux-xfs/msg56642.html
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerRe: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 58 lines · JeffleXu
Quoted 45 lines · Vivek Goyal
Quoted 20 lines · JeffleXu
Quoted 6 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Quoted 6 lines
Hi Jeffle,
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
Quoted 5 lines
This default behaviour for local filesystem, e.g. ext4/xfs, may be straightforward, since the disk inode will be read into memory during the inode instantiation, and checking for persistent inode attribute shall be realatively cheap, except that the default behaviour has changed from 'dax=never' to 'dax=inode'.
Interesting that ext4/xfs allowed for this behavior change.
Quoted 10 lines
Come back to virtiofs, when neither '-o dax' nor '-o dax=always/never/inode' is specified, and it actually behaves as '-o dax=inode', as long as '-o dax=server/attr' option is not specified for virtiofsd, virtiofsd will always clear
FUSE_ATTR_DAXand thus guest will always disable DAX. IOWs, the guest virtiofs atually behaves as '-o dax=never' when neither '-o dax' nor '-o dax=always/never/inode' is specified, and '-o dax=server/attr' option is not specified for virtiofsd.But I'm okay if we need to change the default behaviour for virtiofs.
This is change of behavior from client's perspective. Even if client did not opt-in for DAX, DAX can be enabled based on server's setting. Not that there is anything wrong with it, but change of behavior part concerns me.
Quoted 9 lines
In case of virtiofs, lot of features we are controlling from server. Client typically just calls "mount" and there are not many options users can specify for mount.
Given we already allowed to make client a choice about DAX behavior, I will feel more comfortable that we don't change it and let client request a specific DAX mode and if client does not specify anything, then DAX is not enabled.
Quoted 7 lines
Hi Vivek,
How about the following design about the default behavior to move this patchset forward, considering the discussion on another thread [1]?
- guest side: '-o dax=inode' acts as the default, keeping consistent with xfs/ext4
This sounds good.
Quoted 6 lines
- virtiofsd: the default behavior is like, neither file size based policy ('dax=server') nor persistent inode flags based policy ('dax=attr') is used, though virtiofsd indeed advertises that it supports per inode DAX feature (so that it won't fail
FUSE_INITnegotiation phase when guest advertises dax=inode by default)... In fact, it acts like 'dax=never' in this case.
Not sure why virtiofsd needs to advertise that it supports per inode DAX even if no per inode dax policy is in affect. Guest will know that server is not supporting per inode DAX. But it will not return an error to user space (because dax=inode seems to be advisory).
IOW, this is very similar to the case of using dax=inode on a block device which does not support DAX. No errors and no warnings.
Quoted 4 lines
Then when guest opts-in and specifies '-o dax=inode' manually, then it shall realize that proper configuration for virtiofsd is also needed (-o dax=server|attr).
I gave some comments w.r.t dax=server naming in your posting. Not sure if you got a chance to look at it.
Thanks
Vivek
Quoted 7 lines
[1] https://www.spinics.net/lists/linux-xfs/msg56642.html
Signature · 2 lines
Thanks, Jeffle
Re: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 76 lines · Vivek Goyal
Quoted 58 lines · JeffleXu
Quoted 45 lines · Vivek Goyal
Quoted 20 lines · JeffleXu
Quoted 6 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Quoted 6 lines
Hi Jeffle,
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
Quoted 5 lines
This default behaviour for local filesystem, e.g. ext4/xfs, may be straightforward, since the disk inode will be read into memory during the inode instantiation, and checking for persistent inode attribute shall be realatively cheap, except that the default behaviour has changed from 'dax=never' to 'dax=inode'.
Interesting that ext4/xfs allowed for this behavior change.
Quoted 10 lines
Come back to virtiofs, when neither '-o dax' nor '-o dax=always/never/inode' is specified, and it actually behaves as '-o dax=inode', as long as '-o dax=server/attr' option is not specified for virtiofsd, virtiofsd will always clear
FUSE_ATTR_DAXand thus guest will always disable DAX. IOWs, the guest virtiofs atually behaves as '-o dax=never' when neither '-o dax' nor '-o dax=always/never/inode' is specified, and '-o dax=server/attr' option is not specified for virtiofsd.But I'm okay if we need to change the default behaviour for virtiofs.
This is change of behavior from client's perspective. Even if client did not opt-in for DAX, DAX can be enabled based on server's setting. Not that there is anything wrong with it, but change of behavior part concerns me.
Quoted 9 lines
In case of virtiofs, lot of features we are controlling from server. Client typically just calls "mount" and there are not many options users can specify for mount.
Given we already allowed to make client a choice about DAX behavior, I will feel more comfortable that we don't change it and let client request a specific DAX mode and if client does not specify anything, then DAX is not enabled.
Quoted 7 lines
Hi Vivek,
How about the following design about the default behavior to move this patchset forward, considering the discussion on another thread [1]?
- guest side: '-o dax=inode' acts as the default, keeping consistent with xfs/ext4
This sounds good.
Quoted 6 lines
- virtiofsd: the default behavior is like, neither file size based policy ('dax=server') nor persistent inode flags based policy ('dax=attr') is used, though virtiofsd indeed advertises that it supports per inode DAX feature (so that it won't fail
FUSE_INITnegotiation phase when guest advertises dax=inode by default)... In fact, it acts like 'dax=never' in this case.
Quoted 7 lines
Not sure why virtiofsd needs to advertise that it supports per inode DAX even if no per inode dax policy is in affect. Guest will know that server is not supporting per inode DAX. But it will not return an error to user space (because dax=inode seems to be advisory).
IOW, this is very similar to the case of using dax=inode on a block device which does not support DAX. No errors and no warnings.
OK. I will adopt this behavior. That is, if virtiofsd is not specified with 'dax=server|attr' option, it won't advertise support for per inode DAX in FUSE_INIT either. And then client will fallback to 'dax=never' even if it is mounted with 'dax=inode'.
Quoted 19 lines
Quoted 5 lines
Then when guest opts-in and specifies '-o dax=inode' manually, then it shall realize that proper configuration for virtiofsd is also needed (-o dax=server|attr).
I gave some comments w.r.t dax=server naming in your posting. Not sure if you got a chance to look at it.
Thanks
VivekQuoted 7 lines
[1] https://www.spinics.net/lists/linux-xfs/msg56642.html
Signature · 2 lines
Thanks, Jeffle
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerRe: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
In reply to JeffleXuQuoted 11 lines · JeffleXu
Quoted 8 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Hi Jeffle,
Quoted 8 lines
Quoted 4 lines
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
[ CC dave chinner]
Is it not change of default behavior for ext4/xfs as well. My understanding is that prior to this new dax options, "-o dax" enabled dax on filesystem and if user did not specify it, DAX is disbaled by default.
Now after introduction of "-o dax=always/never/inode", if suddenly "-o dax=inode" became the default if user did not specify anything, that's change of behavior. Is that intentional. If given a choice, I would rather not change default and ask user to opt-in for appropriate dax functionality.
Dave, you might have thoughts on this. It makes me uncomfortable to change virtiofs dax default now just because other filesytems did it.
Thanks
Vivek
Quoted 236 lines
This default behaviour for local filesystem, e.g. ext4/xfs, may be straightforward, since the disk inode will be read into memory during the inode instantiation, and checking for persistent inode attribute shall be realatively cheap, except that the default behaviour has changed from 'dax=never' to 'dax=inode'.
Come back to virtiofs, when neither '-o dax' nor '-o dax=always/never/inode' is specified, and it actually behaves as '-o dax=inode', as long as '-o dax=server/attr' option is not specified for virtiofsd, virtiofsd will always clear
FUSE_ATTR_DAXand thus guest will always disable DAX. IOWs, the guest virtiofs atually behaves as '-o dax=never' when neither '-o dax' nor '-o dax=always/never/inode' is specified, and '-o dax=server/attr' option is not specified for virtiofsd.But I'm okay if we need to change the default behaviour for virtiofs.
Quoted 123 lines
Quoted 45 lines
By the time this patch is applied, 'inode' mode is actually equal to 'always' mode, before the per-file DAX flag is introduced in the following patch.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 19 ++++++++++++++++--- fs/fuse/fuse_i.h | 14 ++++++++++++-- fs/fuse/inode.c | 10 +++++++--- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- 4 files changed, 49 insertions(+), 10 deletions(-)fs/fuse/dax.c +9
-2@@ -1284,11 +1284,14 @@ static int fuse_dax_mem_range_init(struct fuse_conn_dax *fcd)return ret; } -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev) +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode dax_mode, +struct dax_device *dax_dev) {struct fuse_conn_dax *fcd;int err; + fc->dax_mode = dax_mode; +if (!dax_dev)return 0 ;@@ -1335,11 +1338,21 @@ static const struct address_space_operations fuse_dax_file_aops = {static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); +unsigned int dax_mode = fc->dax_mode; + +if (dax_mode ==FUSE_DAX_NEVER ) +return false; -if (fc->dax) +/* fs/fuse/dax.c static const struct address_space_operations fuse_dax_file_aops = { · line 1355
Quoted 10 lines
fs/fuse/dax.c +7
-1+* If 'dax=always/inode', fc->dax couldn't be NULL even when fuse +* daemon doesn't support DAX, since the mount routine will fail +* early in this case. +*/ +if (dax_mode ==FUSE_DAX_ALWAYS )return true; -return false; +/* dax_mode == FUSE_DAX_INODE */ +return true;So as of this patch except
FUSE_DAX_NEVERreturn true and this will change in later patches forFUSE_DAX_INODE? If that's the case, keep it simple in this patch and change it later in the patch series.
fuse_should_enable_dax(){ if (dax_mode == FUSE_DAX_NEVER) return false; return true; }Quoted 56 lines
fs/fuse/dax.c +0
-0}void fuse_dax_inode_init (struct inode *inode)fs/fuse/fuse_i.h +12
-2@@ -480,6 +480,12 @@ struct fuse_dev {struct list_head entry; }; +enum fuse_dax_mode { +FUSE_DAX_INODE , +FUSE_DAX_ALWAYS , +FUSE_DAX_NEVER , +}; +struct fuse_fs_context {int fd;struct file *file;@@ -497,7 +503,7 @@ struct fuse_fs_context { bool no_control:1; bool no_force_umount:1; bool legacy_opts_show:1; - bool dax:1; +enum fuse_dax_mode dax_mode;unsigned int max_read;unsigned int blksize;const char *subtype;@@ -802,6 +808,9 @@ struct fuse_conn {struct list_head devices;#ifdef CONFIG_FUSE_DAX +/* dax mode: FUSE_DAX_* (always, never or per-file) */ +enum fuse_dax_mode dax_mode; +/* Dax specific conn data, non-NULL if DAX is enabled */ struct fuse_conn_dax *dax;#endif @@ -1255,7 +1264,8 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);int fuse_dax_mmap (struct file *file,struct vm_area_struct *vma);int fuse_dax_break_layouts (struct inode *inode,u64 dmap_start,u64 dmap_end); -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev); +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode mode, +struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode);fs/fuse/inode.c +0
-0@@ -742,8 +742,12 @@ static int fuse_show_options(struct seq_file *m, struct dentry *root)seq_printf (m,",blksize=%lu" , sb->s_blocksize); }Quoted 8 lines
fs/fuse/inode.c static int fuse_show_options(struct seq_file *m, struct dentry *root) · line 746
Quoted 5 lines
fs/fuse/inode.c +2
-2#ifdef CONFIG_FUSE_DAX -if (fc->dax) -seq_puts (m,",dax" ); +if (fc->dax_mode ==FUSE_DAX_ALWAYS ) +seq_puts (m,",dax=always" );So if somebody mounts with "-o dax" then kernel previous to this change will show "dax" and kernel after this change will show "dax=always"?
Yes. It's actually how per-file DAX on ext4/xfs behaves.
Quoted 79 lines
How about not change the behavior. Keep a mode say
FUSE_DAX_LEGACYwhich will be set when user specifies "-o dax". InternallyFUSE_DAX_LEGACYandFUSE_DAX_ALWAYSwill be same.if (fc->dax_mode == FUSE_DAX_LEGACY) seq_puts(m, ",dax");Quoted 66 lines
fs/fuse/inode.c +5
-1+ else if (fc->dax_mode == FUSE_DAX_NEVER) + seq_puts(m, ",dax=never"); + else if (fc->dax_mode == FUSE_DAX_INODE) +seq_puts (m,",dax=inode" );#endif return 0 ;@@ -1493,7 +1497,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) sb->s_subtype = ctx->subtype; ctx->subtype =NULL ;if (IS_ENABLED (CONFIG_FUSE_DAX )) { - err =fuse_dax_conn_alloc (fc, ctx->dax_dev); + err =fuse_dax_conn_alloc (fc, ctx->dax_mode, ctx->dax_dev);if (err) goto err; }fs/fuse/virtio_fs.c +14
-2@@ -88,12 +88,21 @@ struct virtio_fs_req_work {static int virtio_fs_enqueue_req (struct virtio_fs_vq *fsvq,struct fuse_req *req,bool in_flight); +static const struct constant_table dax_param_enums[] = { + {"inode" ,FUSE_DAX_INODE }, + {"always" ,FUSE_DAX_ALWAYS }, + {"never" ,FUSE_DAX_NEVER }, + {} +}; +enum {OPT_DAX , +OPT_DAX_ENUM , };static const struct fs_parameter_spec virtio_fs_parameters[] = {fsparam_flag ("dax" ,OPT_DAX ), +fsparam_enum ("dax" ,OPT_DAX_ENUM , dax_param_enums), {} };@@ -110,7 +119,10 @@ static int virtio_fs_parse_param(struct fs_context *fsc,switch (opt) {case OPT_DAX : - ctx->dax =1 ; + ctx->dax_mode =FUSE_DAX_ALWAYS ; +break ; +case OPT_DAX_ENUM : + ctx->dax_mode = result.uint_32;break ;default :return -EINVAL ;@@ -1326,7 +1338,7 @@ static int virtio_fs_fill_super(struct super_block *sb, struct fs_context *fsc)/* virtiofs allocates and installs its own fuse devices */ ctx->fudptr =NULL ; -if (ctx->dax) { +if (ctx->dax_mode !=FUSE_DAX_NEVER ) {if (!fs->dax_dev) { err = -EINVAL ; pr_err("virtio-fs: dax can't be enabled as filesystem"Signature · 1 line
2.27.0Signature · 2 lines
Thanks, Jeffle
Re: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
cc [Ira Weiny], author of per inode DAX on xfs/ext4
Quoted 28 lines · Vivek Goyal
Quoted 11 lines · JeffleXu
Quoted 8 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Hi Jeffle,
Quoted 8 lines
Quoted 4 lines
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
[ CC dave chinner]
Is it not change of default behavior for ext4/xfs as well. My understanding is that prior to this new dax options, "-o dax" enabled dax on filesystem and if user did not specify it, DAX is disbaled by default.
Quoted 9 lines
Now after introduction of "-o dax=always/never/inode", if suddenly "-o dax=inode" became the default if user did not specify anything, that's change of behavior. Is that intentional. If given a choice, I would rather not change default and ask user to opt-in for appropriate dax functionality.
Dave, you might have thoughts on this. It makes me uncomfortable to change virtiofs dax default now just because other filesytems did it.
I can only find the following discussions about the earliest record on this tri-state mount option:
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/
Hi, Ira Weiny,
Do you have any thought on this, i.e. why the default behavior has changed after introduction of per inode dax?
Signature · 2 lines
Thanks, Jeffle
Re: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 24 lines · JeffleXu
cc [Ira Weiny], author of per inode DAX on xfs/ext4
Quoted 21 lines · Vivek Goyal
Quoted 19 lines · JeffleXu
Quoted 6 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Quoted 6 lines
Hi Jeffle,
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
It's been a while so I'm fuzzy on the details of the discussions but yes that is the way things are now in the code.
Quoted 6 lines
Quoted 6 lines
[ CC dave chinner]
Is it not change of default behavior for ext4/xfs as well. My understanding is that prior to this new dax options, "-o dax" enabled dax on filesystem and if user did not specify it, DAX is disbaled by default.
Technically it does change default behavior... However, NOT in a way which breaks anything. See below.
Quoted 4 lines
Quoted 4 lines
Now after introduction of "-o dax=always/never/inode", if suddenly "-o dax=inode" became the default if user did not specify anything, that's change of behavior.
Technically yes but not in a broken way.
Quoted 4 lines
Quoted 4 lines
Is that intentional. If given a choice,
I would rather not change default and ask user to opt-in for appropriate dax functionality.
There is no need for this.
Quoted 8 lines
Quoted 4 lines
Dave, you might have thoughts on this. It makes me uncomfortable to change virtiofs dax default now just because other filesytems did it.
I can only find the following discussions about the earliest record on this tri-state mount option:
Quoted 8 lines
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/Hi, Ira Weiny,
Do you have any thought on this, i.e. why the default behavior has changed after introduction of per inode dax?
While this is 'technically' different behavior the end user does not see any difference in behavior if they continue without software changes. Specifically specifying nothing continues to operate with all the files on the FS to be '_not_ DAX'. While specifying '-o dax' forces DAX on all files.
This expands the default behavior in a backwards compatible manner. The user can now enable DAX on some files. But this is an opt-in on the part of the user of the FS and again does not change with existing software/scripts/etc.
Does that make sense?
Ira
Vivek Goyal
maintainerRe: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 70 lines · Ira Weiny
Quoted 24 lines · JeffleXu
cc [Ira Weiny], author of per inode DAX on xfs/ext4
Quoted 21 lines · Vivek Goyal
Quoted 19 lines · JeffleXu
Quoted 6 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Quoted 6 lines
Hi Jeffle,
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
It's been a while so I'm fuzzy on the details of the discussions but yes that is the way things are now in the code.
Quoted 6 lines
Quoted 6 lines
[ CC dave chinner]
Is it not change of default behavior for ext4/xfs as well. My understanding is that prior to this new dax options, "-o dax" enabled dax on filesystem and if user did not specify it, DAX is disbaled by default.
Technically it does change default behavior... However, NOT in a way which breaks anything. See below.
Quoted 4 lines
Quoted 4 lines
Now after introduction of "-o dax=always/never/inode", if suddenly "-o dax=inode" became the default if user did not specify anything, that's change of behavior.
Technically yes but not in a broken way.
Quoted 4 lines
Quoted 4 lines
Is that intentional. If given a choice,
I would rather not change default and ask user to opt-in for appropriate dax functionality.There is no need for this.
Quoted 8 lines
Quoted 4 lines
Dave, you might have thoughts on this. It makes me uncomfortable to change virtiofs dax default now just because other filesytems did it.
I can only find the following discussions about the earliest record on this tri-state mount option:
Quoted 8 lines
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/Hi, Ira Weiny,
Do you have any thought on this, i.e. why the default behavior has changed after introduction of per inode dax?
Quoted 6 lines
While this is 'technically' different behavior the end user does not see any difference in behavior if they continue without software changes. Specifically specifying nothing continues to operate with all the files on the FS to be '_not_ DAX'. While specifying '-o dax' forces DAX on all files.
This expands the default behavior in a backwards compatible manner.
This is backward compatible in a sense that if somebody upgrades to new kernel, things will still be same.
I think little problematic change is that say I bring in persistent memory from another system (which has FS_XFLAGS_DAX set on some inodes) and then mount it without andy of the dax mount options, then per inode dax will be enabled unexpectedly if I boot with newer kernels but it will be disable if I mount with older kernels. Do I understand it right.
Quoted 3 lines
The user
can now enable DAX on some files. But this is an opt-in on the part of the user of the FS and again does not change with existing software/scripts/etc.
Don't understand this "opt-in" bit. If user mounts an fs without specifying any of the dax options, then per inode dax will still be enabled if inode has the correct flag set. So is setting of flag being considered as opt-in (insted of mount option).
If setting of flag is being considered as opt-in, that probably will not work very well with virtiofs. Because server can enforce a different policy for enabling per file dax (instead of FS_XFLAG_DAX).
And given there are two entities here (client and server), I think it will be good if if we give client a chance as well to decide whether it wants to enable per file dax or not. I know it can alwasy do "dax=never" but it can still be broken if client software remains same but host/server software is upgraded or commnad line changed.
So for virtiofs, I think better behavior is to continue to not enable any dax until and unless user opts-in using "-o dax=foo" options.
Thanks
Vivek
Quoted 5 lines
Does that make sense?
Ira
Re: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 81 lines · Vivek Goyal
Quoted 70 lines · Ira Weiny
Quoted 24 lines · JeffleXu
cc [Ira Weiny], author of per inode DAX on xfs/ext4
Quoted 21 lines · Vivek Goyal
Quoted 19 lines · JeffleXu
Quoted 6 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Quoted 6 lines
Hi Jeffle,
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
It's been a while so I'm fuzzy on the details of the discussions but yes that is the way things are now in the code.
Quoted 6 lines
Quoted 6 lines
[ CC dave chinner]
Is it not change of default behavior for ext4/xfs as well. My understanding is that prior to this new dax options, "-o dax" enabled dax on filesystem and if user did not specify it, DAX is disbaled by default.
Technically it does change default behavior... However, NOT in a way which breaks anything. See below.
Quoted 4 lines
Quoted 4 lines
Now after introduction of "-o dax=always/never/inode", if suddenly "-o dax=inode" became the default if user did not specify anything, that's change of behavior.
Technically yes but not in a broken way.
Quoted 4 lines
Quoted 4 lines
Is that intentional. If given a choice,
I would rather not change default and ask user to opt-in for appropriate dax functionality.There is no need for this.
Quoted 8 lines
Quoted 4 lines
Dave, you might have thoughts on this. It makes me uncomfortable to change virtiofs dax default now just because other filesytems did it.
I can only find the following discussions about the earliest record on this tri-state mount option:
Quoted 8 lines
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/Hi, Ira Weiny,
Do you have any thought on this, i.e. why the default behavior has changed after introduction of per inode dax?
Quoted 6 lines
While this is 'technically' different behavior the end user does not see any difference in behavior if they continue without software changes. Specifically specifying nothing continues to operate with all the files on the FS to be '_not_ DAX'. While specifying '-o dax' forces DAX on all files.
This expands the default behavior in a backwards compatible manner.
This is backward compatible in a sense that if somebody upgrades to new kernel, things will still be same.
Quoted 6 lines
I think little problematic change is that say I bring in persistent memory from another system (which has
FS_XFLAGS_DAXset on some inodes) and then mount it without andy of the dax mount options, then per inode dax will be enabled unexpectedly if I boot with newer kernels but it will be disable if I mount with older kernels. Do I understand it right.
Indeed that will happen. However, wouldn't the users (software) of those files have knowledge that those files were DAX and want to continue with them in that mode?
Quoted 8 lines
Quoted 4 lines
The user
can now enable DAX on some files. But this is an opt-in on the part of the user of the FS and again does not change with existing software/scripts/etc.Don't understand this "opt-in" bit. If user mounts an fs without specifying any of the dax options, then per inode dax will still be enabled if inode has the correct flag set.
But only users who actually set that flag 'opt-in'.
Quoted 2 lines
So is setting of flag being
considered as opt-in (insted of mount option).
Yes.
Quoted 4 lines
If setting of flag is being considered as opt-in, that probably will not work very well with virtiofs. Because server can enforce a different policy for enabling per file dax (instead of
FS_XFLAG_DAX).
I'm not sure I understand how this happens? I think the server probably has to enable per INODE by default to allow the client to do what the end users wants.
I agree that if the end user is expecting DAX and the server disables it then that is a problem but couldn't that happen before? Maybe I'm getting confused because I'm not familiar enough with virtiofs.
Quoted 6 lines
And given there are two entities here (client and server), I think it will be good if if we give client a chance as well to decide whether it wants to enable per file dax or not. I know it can alwasy do "dax=never" but it can still be broken if client software remains same but host/server software is upgraded or commnad line changed.
But the files are 'owned' by a single user or group of users who must have placed the file in DAX mode at some point right?
Quoted 3 lines
So for virtiofs, I think better behavior is to continue to not enable any dax until and unless user opts-in using "-o dax=foo" options.
I'm not sure, maybe.
Ira
Vivek Goyal
maintainerRe: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 89 lines · Ira Weiny
Quoted 81 lines · Vivek Goyal
Quoted 70 lines · Ira Weiny
Quoted 24 lines · JeffleXu
cc [Ira Weiny], author of per inode DAX on xfs/ext4
Quoted 21 lines · Vivek Goyal
Quoted 19 lines · JeffleXu
Quoted 6 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
Quoted 6 lines
Hi Jeffle,
I am not sure when -o "dax=inode" is used as a default? If user specifies, "-o dax" then it is equal to "-o dax=always", otherwise user will explicitly specify "-o dax=always/never/inode". So when is dax=inode is used as default?
That means when neither '-o dax' nor '-o dax=always/never/inode' is specified, it is actually equal to '-o dax=inode', which is also how per-file DAX on ext4/xfs works.
It's been a while so I'm fuzzy on the details of the discussions but yes that is the way things are now in the code.
Quoted 6 lines
Quoted 6 lines
[ CC dave chinner]
Is it not change of default behavior for ext4/xfs as well. My understanding is that prior to this new dax options, "-o dax" enabled dax on filesystem and if user did not specify it, DAX is disbaled by default.
Technically it does change default behavior... However, NOT in a way which breaks anything. See below.
Quoted 4 lines
Quoted 4 lines
Now after introduction of "-o dax=always/never/inode", if suddenly "-o dax=inode" became the default if user did not specify anything, that's change of behavior.
Technically yes but not in a broken way.
Quoted 4 lines
Quoted 4 lines
Is that intentional. If given a choice,
I would rather not change default and ask user to opt-in for appropriate dax functionality.There is no need for this.
Quoted 8 lines
Quoted 4 lines
Dave, you might have thoughts on this. It makes me uncomfortable to change virtiofs dax default now just because other filesytems did it.
I can only find the following discussions about the earliest record on this tri-state mount option:
Quoted 8 lines
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/Hi, Ira Weiny,
Do you have any thought on this, i.e. why the default behavior has changed after introduction of per inode dax?
Quoted 6 lines
While this is 'technically' different behavior the end user does not see any difference in behavior if they continue without software changes. Specifically specifying nothing continues to operate with all the files on the FS to be '_not_ DAX'. While specifying '-o dax' forces DAX on all files.
This expands the default behavior in a backwards compatible manner.
This is backward compatible in a sense that if somebody upgrades to new kernel, things will still be same.
Quoted 6 lines
I think little problematic change is that say I bring in persistent memory from another system (which has
FS_XFLAGS_DAXset on some inodes) and then mount it without andy of the dax mount options, then per inode dax will be enabled unexpectedly if I boot with newer kernels but it will be disable if I mount with older kernels. Do I understand it right.
Quoted 3 lines
Indeed that will happen. However, wouldn't the users (software) of those files have knowledge that those files were DAX and want to continue with them in that mode?
I am not sure. Say before per-inode dax feature, I had written a script which walks though all the mount points and figure out if dax is enabled or not. I could simply look at mount options and tell if dax could be enabled or not.
But now same script will give false results as per inode dax could still be enabled.
Quoted 18 lines
Quoted 9 lines
Quoted 5 lines
The user
can now enable DAX on some files. But this is an opt-in on the part of the user of the FS and again does not change with existing software/scripts/etc.Don't understand this "opt-in" bit. If user mounts an fs without specifying any of the dax options, then per inode dax will still be enabled if inode has the correct flag set.
But only users who actually set that flag 'opt-in'.
Quoted 2 lines
So is setting of flag being
considered as opt-in (insted of mount option).Yes.
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
If setting of flag is being considered as opt-in, that probably will not work very well with virtiofs. Because server can enforce a different policy for enabling per file dax (instead of
FS_XFLAG_DAX).I'm not sure I understand how this happens? I think the server probably has to enable per INODE by default to allow the client to do what the end users wants.
Server can have either per inode disabled or enabled. If enabled, it could determine DAX status of file based on FS_XFLAG_DAX or based on something else depending on server policy. Users want to be able to determine DAX status of file based on say file size.
Quoted 2 lines
I agree that if the end user is expecting DAX and the server disables it then that is a problem but couldn't that happen before?
If end user expects to enable DAX and sever can't enable it, then mount fails. So currently if you mount "-o dax" and server does not support DAX, mount will fail.
I think same should happen when per inode DAX is introduced for virtiofs. If sever does not support per inode dax and user mounts with "-o dax=inode", then mount should fail.
In fact, this is another reason that probably "dax=inode" should not be default. Say client is new and server is old and does not support per inode dax, then client might start failing mount after client upgrade, and that's not good.
More I think about it, more it feels like that "dax=never" should be the default if user has not specified any of the dax options. This probably will introduce least amount of surprise. Atleast for virtiofs. IMHO, it probably would have made sense even for ext4/xfs but that ship has already sailed.
Quoted 4 lines
Maybe I'm getting confused
because I'm not familiar enough with virtiofs.Quoted 1 line
Quoted 8 lines
Quoted 5 lines
And given there are two entities here (client and server), I think it will be good if if we give client a chance as well to decide whether it wants to enable per file dax or not. I know it can alwasy do "dax=never" but it can still be broken if client software remains same but host/server software is upgraded or commnad line changed.
But the files are 'owned' by a single user or group of users who must have placed the file in DAX mode at some point right?
Yes, either users/groups/admin might have set FS_XFLAG_DAX on inodes. But now there is another controller (virtiofs server) which determines whether that flag takes affect or not (based on server settings).
We did not have this server scenario in case of local filesystems.
Thanks
Vivek
Quoted 9 lines
Quoted 4 lines
So for virtiofs, I think better behavior is to continue to not enable any dax until and unless user opts-in using "-o dax=foo" options.
I'm not sure, maybe.
Ira
Re: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
On Mon, Oct 25, 2021 at 03:33:31PM -0400, Vivek Goyal wrote:
[snip]
Quoted 35 lines
Quoted 31 lines
Quoted 24 lines
Quoted 14 lines
Quoted 5 lines
Quoted 1 line
I can only find the following discussions about the earliest record on this tri-state mount option:
Quoted 8 lines
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/Hi, Ira Weiny,
Do you have any thought on this, i.e. why the default behavior has changed after introduction of per inode dax?
Quoted 6 lines
While this is 'technically' different behavior the end user does not see any difference in behavior if they continue without software changes. Specifically specifying nothing continues to operate with all the files on the FS to be '_not_ DAX'. While specifying '-o dax' forces DAX on all files.
This expands the default behavior in a backwards compatible manner.
This is backward compatible in a sense that if somebody upgrades to new kernel, things will still be same.
Quoted 6 lines
I think little problematic change is that say I bring in persistent memory from another system (which has
FS_XFLAGS_DAXset on some inodes) and then mount it without andy of the dax mount options, then per inode dax will be enabled unexpectedly if I boot with newer kernels but it will be disable if I mount with older kernels. Do I understand it right.Quoted 3 lines
Indeed that will happen. However, wouldn't the users (software) of those files have knowledge that those files were DAX and want to continue with them in that mode?
Quoted 7 lines
I am not sure. Say before per-inode dax feature, I had written a script which walks though all the mount points and figure out if dax is enabled or not. I could simply look at mount options and tell if dax could be enabled or not.
But now same script will give false results as per inode dax could still be enabled.
The mount option is being deprecated. So it is best to start to phase out scripts like that.
Quoted 23 lines
Quoted 23 lines
Quoted 10 lines
Quoted 6 lines
The user
can now enable DAX on some files. But this is an opt-in on the part of the user of the FS and again does not change with existing software/scripts/etc.Don't understand this "opt-in" bit. If user mounts an fs without specifying any of the dax options, then per inode dax will still be enabled if inode has the correct flag set.
But only users who actually set that flag 'opt-in'.
Quoted 2 lines
So is setting of flag being
considered as opt-in (insted of mount option).Yes.
Quoted 4 lines
If setting of flag is being considered as opt-in, that probably will not work very well with virtiofs. Because server can enforce a different policy for enabling per file dax (instead of
FS_XFLAG_DAX).
Quoted 8 lines
Quoted 3 lines
I'm not sure I understand how this happens? I think the server probably has to enable per INODE by default to allow the client to do what the end users wants.
Server can have either per inode disabled or enabled. If enabled, it could determine DAX status of file based on
FS_XFLAG_DAXor based on something else depending on server policy. Users want to be able to determine DAX status of file based on say file size.
'file size'? I'm not sure how that would work. Did you mean something else?
Quoted 7 lines
Quoted 3 lines
I agree that if the end user is expecting DAX and the server disables it then that is a problem but couldn't that happen before?
If end user expects to enable DAX and sever can't enable it, then mount fails. So currently if you mount "-o dax" and server does not support DAX, mount will fail.
The same could happen on a server where the underlying device does not support DAX. What if the server was mounted without '-o dax'? Wouldn't a client mount with '-o dax' fail now? So why can't the same be true with the new set of options?
Quoted 4 lines
I think same should happen when per inode DAX is introduced for virtiofs. If sever does not support per inode dax and user mounts with "-o dax=inode", then mount should fail.
I think that is reasonable. The client can't mount with something the server can't support.
Quoted 5 lines
In fact, this is another reason that probably "dax=inode" should not be default. Say client is new and server is old and does not support per inode dax, then client might start failing mount after client upgrade, and that's not good.
Shouldn't the client fall back to whatever the server supports? It is the same as the client wanting DAX now without server and/or device support. It just can't get it. Right?
Quoted 6 lines
More I think about it, more it feels like that "dax=never" should be the default if user has not specified any of the dax options. This probably will introduce least amount of surprise. Atleast for virtiofs. IMHO, it probably would have made sense even for ext4/xfs but that ship has already sailed.
I disagree because dax=never is backwards from what we really want for the future. 'dax=inode' is the most flexible setting. In fact that setting is best for the server by default which allows more control to be in the clients hands. Would you agree?
Quoted 11 lines
Quoted 11 lines
Maybe I'm getting confused
because I'm not familiar enough with virtiofs.Quoted 6 lines
And given there are two entities here (client and server), I think it will be good if if we give client a chance as well to decide whether it wants to enable per file dax or not. I know it can alwasy do "dax=never" but it can still be broken if client software remains same but host/server software is upgraded or commnad line changed.
Quoted 6 lines
Quoted 2 lines
But the files are 'owned' by a single user or group of users who must have placed the file in DAX mode at some point right?
Yes, either users/groups/admin might have set
FS_XFLAG_DAXon inodes. But now there is another controller (virtiofs server) which determines whether that flag takes affect or not (based on server settings).
I think this is just like the file being on a device which does not support DAX. The file inode flag can be set but the file will not be in DAX mode on a non-dax device. So in this case the server is a non-dax device.
Ira
Quoted 15 lines
We did not have this server scenario in case of local filesystems.
Thanks
VivekQuoted 9 lines
Quoted 4 lines
So for virtiofs, I think better behavior is to continue to not enable any dax until and unless user opts-in using "-o dax=foo" options.
I'm not sure, maybe.
Ira
Vivek Goyal
maintainerRe: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
Quoted 44 lines · Ira Weiny
On Mon, Oct 25, 2021 at 03:33:31PM -0400, Vivek Goyal wrote:
[snip]
Quoted 40 lines
Quoted 31 lines
Quoted 24 lines
Quoted 14 lines
Quoted 5 lines
Quoted 1 line
I can only find the following discussions about the earliest record on this tri-state mount option:
Quoted 8 lines
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/Hi, Ira Weiny,
Do you have any thought on this, i.e. why the default behavior has changed after introduction of per inode dax?
Quoted 6 lines
While this is 'technically' different behavior the end user does not see any difference in behavior if they continue without software changes. Specifically specifying nothing continues to operate with all the files on the FS to be '_not_ DAX'. While specifying '-o dax' forces DAX on all files.
This expands the default behavior in a backwards compatible manner.
This is backward compatible in a sense that if somebody upgrades to new kernel, things will still be same.
Quoted 6 lines
I think little problematic change is that say I bring in persistent memory from another system (which has
FS_XFLAGS_DAXset on some inodes) and then mount it without andy of the dax mount options, then per inode dax will be enabled unexpectedly if I boot with newer kernels but it will be disable if I mount with older kernels. Do I understand it right.Quoted 3 lines
Indeed that will happen. However, wouldn't the users (software) of those files have knowledge that those files were DAX and want to continue with them in that mode?
I am not sure. Say before per-inode dax feature, I had written a script which walks though all the mount points and figure out if dax is enabled or not. I could simply look at mount options and tell if dax could be enabled or not.
Quoted 5 lines
Quoted 2 lines
But now same script will give false results as per inode dax could still be enabled.
The mount option is being deprecated. So it is best to start to phase out scripts like that.
Sure. But this change does break such scripts (if there is any). I am just responding to previous comments that existing software/scripts should not be broken.
Quoted 28 lines
Quoted 28 lines
Quoted 27 lines
Quoted 11 lines
Quoted 7 lines
The user
can now enable DAX on some files. But this is an opt-in on the part of the user of the FS and again does not change with existing software/scripts/etc.Don't understand this "opt-in" bit. If user mounts an fs without specifying any of the dax options, then per inode dax will still be enabled if inode has the correct flag set.
But only users who actually set that flag 'opt-in'.
Quoted 2 lines
So is setting of flag being
considered as opt-in (insted of mount option).Yes.
Quoted 4 lines
If setting of flag is being considered as opt-in, that probably will not work very well with virtiofs. Because server can enforce a different policy for enabling per file dax (instead of
FS_XFLAG_DAX).I'm not sure I understand how this happens? I think the server probably has to enable per INODE by default to allow the client to do what the end users wants.
Quoted 6 lines
Quoted 4 lines
Server can have either per inode disabled or enabled. If enabled, it could determine DAX status of file based on
FS_XFLAG_DAXor based on something else depending on server policy. Users want to be able to determine DAX status of file based on say file size.'file size'? I'm not sure how that would work. Did you mean something else?
So virtiofs uses DAX only to bypass page cache in guest. virtiofs pci device advertizes a range of memory which is directly accessed using dax. We use a chunk size of 2MB. That means for every 2MB chunk, there will be around 512 pages. Each struct page will consume around 64 bytes of RAM in guest. So for every 2MB chunk of file, RAM usage in guest is around 512 * 64 = 32768 (32Kib).
So there are users who claim that for smaller files say 4K or 8K in size, it is probably better to not use DAX at all. In that case we will use say 4K of page cache and leave DAX memory to be used for larger files. (This will be useful only if virtiofs cache memory is in short supply).
Hence the idea that why not use per inode dax and enable dax selectively on files as needed. Given we have a remote server running, it gives extra capability that we can take this DAX decision dynamically based on some server policy (and not necessarily rely on FS_XFLAG_DAX stuff).
So once such policy is file size based policy. Where if a file size is small, server might not want to use DAX on that file. There could be many more such policies depending on where DAX is most useful in the context of virtiofs.
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
I agree that if the end user is expecting DAX and the server disables it then that is a problem but couldn't that happen before?
If end user expects to enable DAX and sever can't enable it, then mount fails. So currently if you mount "-o dax" and server does not support DAX, mount will fail.
The same could happen on a server where the underlying device does not support DAX. What if the server was mounted without '-o dax'?
In general, there is no connection between DAX in guest and device on host enabling DAX. We can very well enable DAX in guest without having any DAX enabled on host device. From virtiofs perspective, we are just mmapping host files in qemu address space and that works both with dax enabled/disabled devices on host.
Quoted 3 lines
Wouldn't a client mount
with '-o dax' fail now? So why can't the same be true with the new set of options?
So yes, if server does not support DAX and client asks for DAX, mount will fail. (As it should fail).
Problem with enabling "dax=inode" by default is that if a client is mounted without any dax option, then dax is disabled. Now if a server is upgraded and restarted with some dax policy enabled, suddenly dax will be enabled in client without it opting in for anything and client might be surprised.
Now one argument can be hey, we have FS_XFLAG_DAX set on inode, so it is ok to turn on dax. May be. But virtiofs serever can have its own dax policies (like file size based policy), and it can ignore FS_XFLAG_DAX completely. In that case enabling per inode dax by default (without client opting in), seems contrary to what we are doing now.
Hence, I think not having "dax=inode" as default, is path of least surprise for an existing user. A user can easily tell whether dax is being used or not just by looking at filesystem mount optins.
Quoted 10 lines
Quoted 5 lines
I think same should happen when per inode DAX is introduced for virtiofs. If sever does not support per inode dax and user mounts with "-o dax=inode", then mount should fail.
I think that is reasonable. The client can't mount with something the server can't support.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
In fact, this is another reason that probably "dax=inode" should not be default. Say client is new and server is old and does not support per inode dax, then client might start failing mount after client upgrade, and that's not good.
Shouldn't the client fall back to whatever the server supports? It is the same as the client wanting DAX now without server and/or device support. It just can't get it. Right?
Well, current model is that fail the operation and let user try mount again without DAX.
If we were to design fallback, then question will be how will user know that server does not support DAX and we fallback to non-dax. Also it will be change of behavior as well from exsiting non-fallback semantics.
I guess one could argue that if you are moving to new dax options (-o dax=inode/always/never), then this is an opportunity to move to fallback model. My concern remains thought that if user specified "dax=inode or dax=always" and server does not support, how will user know we are not using dax.
Not sure there is a good answer here. In some cases users like to see explicit failure if some option can't be supported. IIRC, in case of overalayfs, if users passed in "-o metacopy=on" and if overlayfs can't enable it, then users expected a failure (instead of a ignoring metacopy silently).
So choosing not to fallback seems ok to be. Nobody has complained so far.
Quoted 10 lines
Quoted 7 lines
More I think about it, more it feels like that "dax=never" should be the default if user has not specified any of the dax options. This probably will introduce least amount of surprise. Atleast for virtiofs. IMHO, it probably would have made sense even for ext4/xfs but that ship has already sailed.
I disagree because dax=never is backwards from what we really want for the future. 'dax=inode' is the most flexible setting.
If your goal is to enable dax by default if FS_XFLAG_DAX is set, then' yes dax=inode default makes sense. I was only complaining about change of behavior in some cases. I mean one coule argue same thing for dax=always. If block device supports dax, then enable dax by default until and unless user specifies "-o dax=never". But previous options were not designed that way. A user had to opt-in for DAX behavior even if device had the capability to support DAX.
And in the same line, I am arguing a user should opt-in for per inode DAX, even if inode has the capability to be used as DAX inode.
And I don't mind "dax=inode" being default if that's deemed more useful. My concern there is only change of behavior by default.
Quoted 3 lines
In fact that setting is
best for the server by default which allows more control to be in the clients hands. Would you agree?
"dax=inode" on server default makes sense to me (as long as client asked for dax=inode). Should it be enabled by default in client, I am still afraid of change of behavior from existing dax mount options and having to explain and justify change of behavior to users.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Maybe I'm getting confused
because I'm not familiar enough with virtiofs.Quoted 6 lines
And given there are two entities here (client and server), I think it will be good if if we give client a chance as well to decide whether it wants to enable per file dax or not. I know it can alwasy do "dax=never" but it can still be broken if client software remains same but host/server software is upgraded or commnad line changed.
But the files are 'owned' by a single user or group of users who must have placed the file in DAX mode at some point right?
Quoted 7 lines
Quoted 3 lines
Yes, either users/groups/admin might have set
FS_XFLAG_DAXon inodes. But now there is another controller (virtiofs server) which determines whether that flag takes affect or not (based on server settings).I think this is just like the file being on a device which does not support DAX. The file inode flag can be set but the file will not be in DAX mode on a non-dax device. So in this case the server is a non-dax device.
So if I mount with "dax=inode or dax=always" and block device does not support DAX, what happens. Mount fails or it fallsback siliently to non-dax mode?
I suspect that in new dax options it falls back to non-dax mode. And your argument seems to be that user should stat every file and query for STATX_ATTR_DAX to determine if dax is enabled on file or not.
One one hand, I am not too fond of this new semantics of automatic fallback and dax=inode default, and on the other hand, I want to be as close as possible to ext4/xfs semantics so that there is less confusion for users.
Vivek
Re: [PATCH v6 2/7] fuse: make DAX mount option a tri-state
In reply to Ira WeinyThanks for your replying, Ira Weiny.
Quoted 56 lines · Ira Weiny
[snippet]
Quoted 15 lines
Quoted 11 lines
Quoted 4 lines
Hi, Ira Weiny,
Do you have any thought on this, i.e. why the default behavior has changed after introduction of per inode dax?
While this is 'technically' different behavior the end user does not see any difference in behavior if they continue without software changes. Specifically specifying nothing continues to operate with all the files on the FS to be '_not_ DAX'. While specifying '-o dax' forces DAX on all files.
This expands the default behavior in a backwards compatible manner.
This is backward compatible in a sense that if somebody upgrades to new kernel, things will still be same.
Quoted 6 lines
I think little problematic change is that say I bring in persistent memory from another system (which has
FS_XFLAGS_DAXset on some inodes) and then mount it without andy of the dax mount options, then per inode dax will be enabled unexpectedly if I boot with newer kernels but it will be disable if I mount with older kernels. Do I understand it right.Indeed that will happen. However, wouldn't the users (software) of those files have knowledge that those files were DAX and want to continue with them in that mode?
Quoted 8 lines
Quoted 4 lines
The user
can now enable DAX on some files. But this is an opt-in on the part of the user of the FS and again does not change with existing software/scripts/etc.Don't understand this "opt-in" bit. If user mounts an fs without specifying any of the dax options, then per inode dax will still be enabled if inode has the correct flag set.
But only users who actually set that flag 'opt-in'.
Quoted 2 lines
So is setting of flag being
considered as opt-in (insted of mount option).Yes.
Quoted 4 lines
If setting of flag is being considered as opt-in, that probably will not work very well with virtiofs. Because server can enforce a different policy for enabling per file dax (instead of
FS_XFLAG_DAX).I'm not sure I understand how this happens? I think the server probably has to enable per INODE by default to allow the client to do what the end users wants.
I agree that if the end user is expecting DAX and the server disables it then that is a problem but couldn't that happen before? Maybe I'm getting confused because I'm not familiar enough with virtiofs.
Quoted 1 line
Quoted 8 lines
Quoted 5 lines
And given there are two entities here (client and server), I think it will be good if if we give client a chance as well to decide whether it wants to enable per file dax or not. I know it can alwasy do "dax=never" but it can still be broken if client software remains same but host/server software is upgraded or commnad line changed.
But the files are 'owned' by a single user or group of users who must have placed the file in DAX mode at some point right?
So this is the essence of this issue, i.e. whether those who mount the filesystem (responsible for specifying mount options) and those who set the persistent inode flag are one same group people.
For local filesystem like ext4/xfs, these two entities are most likely one group people, so we can say that 'the default behavior is still backward compatible'.
However this semantic can be challenged a little by the example exposed by Vivek, that these two entities may not be one group even in local filesystem. Though this case may be rare in real world.
But for remote filesystem like virtiofs, the deviation between these two entities can be larger. For example, if the exported directory on host is shared by two guest and guest A sets the persistent inode flag for one file, then guest B will also see that DAX is enabled for this file when the virtiofs is mounted with the default option inside guest B. In this case, the persistent indoe flag is not set by guest B itself nor the server, and it may break the expectation of guest B.
Quoted 5 lines
Quoted 4 lines
So for virtiofs, I think better behavior is to continue to not enable any dax until and unless user opts-in using "-o dax=foo" options.
I also prefer keeping the 'dax=never' default behavior for virtiofs.
Signature · 2 lines
Thanks, Jeffle
PATCH v6 3/7 fuse: support per-file DAX in fuse protocol
In reply to Jeffle XuExpand the fuse protocol to support per-file DAX.
FUSE_PERFILE_DAX flag is added indicating if fuse server/client supporting per-file DAX. It can be conveyed in both FUSE_INIT request and reply.
FUSE_ATTR_DAX flag is added indicating if DAX shall be enabled for corresponding file. It is conveyed in FUSE_LOOKUP reply.
--- include/uapi/linux/fuse.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
include/uapi/linux/fuse.h +8 -1
@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_PERFILE_DAX, FUSE_ATTR_DAX */ #ifndef _LINUX_FUSE_H
@@ -219,7 +222,7 @@#define FUSE_KERNEL_VERSION 7/** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 34 +#define FUSE_KERNEL_MINOR_VERSION 35/** The node ID of the root inode */ #define FUSE_ROOT_ID 1
@@ -336,6 +339,7 @@ struct fuse_file_lock {* write/truncate sgid is killed only if file has group * execute permission. (Same as Linux VFS behavior). * FUSE_SETXATTR_EXT: Server supports extended struct fuse_setxattr_in +* FUSE_PERFILE_DAX: kernel supports per-file DAX */ #define FUSE_ASYNC_READ (1 << 0)#define FUSE_POSIX_LOCKS (1 << 1)
@@ -367,6 +371,7 @@ struct fuse_file_lock {#define FUSE_SUBMOUNTS (1 << 27)#define FUSE_HANDLE_KILLPRIV_V2 (1 << 28)#define FUSE_SETXATTR_EXT (1 << 29) +#define FUSE_PERFILE_DAX (1 << 30)/** * CUSE INIT request/reply flags
@@ -449,8 +454,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)/** * Open flags
Signature · 1 line
2.27.0
Vivek Goyal
maintainerRe: [PATCH v6 3/7] fuse: support per-file DAX in fuse protocol
Quoted 3 lines · Jeffle Xu
Expand the fuse protocol to support per-file DAX.
FUSE_PERFILE_DAXflag is added indicating if fuse server/client
Should we call this flag FUSE_INODE_DAX instead? It is per inode property?
Vivek
Quoted 64 lines
supporting per-file DAX. It can be conveyed in both
FUSE_INITrequest and reply.
FUSE_ATTR_DAXflag is added indicating if DAX shall be enabled for corresponding file. It is conveyed inFUSE_LOOKUPreply.Signed-off-by Jeffle Xu <[email protected]>--- include/uapi/linux/fuse.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)include/uapi/linux/fuse.h +8
-1@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_PERFILE_DAX, FUSE_ATTR_DAX */ #ifndef _LINUX_FUSE_H@@ -219,7 +222,7 @@#define FUSE_KERNEL_VERSION 7/** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 34 +#define FUSE_KERNEL_MINOR_VERSION 35/** The node ID of the root inode */ #define FUSE_ROOT_ID 1@@ -336,6 +339,7 @@ struct fuse_file_lock {* write/truncate sgid is killed only if file has group * execute permission. (Same as Linux VFS behavior). * FUSE_SETXATTR_EXT: Server supports extended struct fuse_setxattr_in +* FUSE_PERFILE_DAX: kernel supports per-file DAX */ #define FUSE_ASYNC_READ (1 << 0)#define FUSE_POSIX_LOCKS (1 << 1)@@ -367,6 +371,7 @@ struct fuse_file_lock {#define FUSE_SUBMOUNTS (1 << 27)#define FUSE_HANDLE_KILLPRIV_V2 (1 << 28)#define FUSE_SETXATTR_EXT (1 << 29) +#define FUSE_PERFILE_DAX (1 << 30)/** * CUSE INIT request/reply flags @@ -449,8 +454,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)/** * Open flags Signature · 1 line
2.27.0
Vivek Goyal
maintainerRe: [PATCH v6 3/7] fuse: support per-file DAX in fuse protocol
Quoted 7 lines · Vivek Goyal
Quoted 3 lines · Jeffle Xu
Expand the fuse protocol to support per-file DAX.
FUSE_PERFILE_DAXflag is added indicating if fuse server/clientShould we call this flag
FUSE_INODE_DAXinstead? It is per inode property?
I realized that you are using FUSE_DAX_INODE to represent dax mode. So it will be confusing to use FUSE_INODE_DAX as protocol flag. How about FUSE_INODE_DAX_STATE instead?
Vivek
Quoted 66 lines
Vivek
Quoted 64 lines
supporting per-file DAX. It can be conveyed in both
FUSE_INITrequest and reply.
FUSE_ATTR_DAXflag is added indicating if DAX shall be enabled for corresponding file. It is conveyed inFUSE_LOOKUPreply.Signed-off-by Jeffle Xu <[email protected]>--- include/uapi/linux/fuse.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)include/uapi/linux/fuse.h +8
-1@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_PERFILE_DAX, FUSE_ATTR_DAX */ #ifndef _LINUX_FUSE_H@@ -219,7 +222,7 @@#define FUSE_KERNEL_VERSION 7/** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 34 +#define FUSE_KERNEL_MINOR_VERSION 35/** The node ID of the root inode */ #define FUSE_ROOT_ID 1@@ -336,6 +339,7 @@ struct fuse_file_lock {* write/truncate sgid is killed only if file has group * execute permission. (Same as Linux VFS behavior). * FUSE_SETXATTR_EXT: Server supports extended struct fuse_setxattr_in +* FUSE_PERFILE_DAX: kernel supports per-file DAX */ #define FUSE_ASYNC_READ (1 << 0)#define FUSE_POSIX_LOCKS (1 << 1)@@ -367,6 +371,7 @@ struct fuse_file_lock {#define FUSE_SUBMOUNTS (1 << 27)#define FUSE_HANDLE_KILLPRIV_V2 (1 << 28)#define FUSE_SETXATTR_EXT (1 << 29) +#define FUSE_PERFILE_DAX (1 << 30)/** * CUSE INIT request/reply flags @@ -449,8 +454,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)/** * Open flags Signature · 1 line
2.27.0
Re: [PATCH v6 3/7] fuse: support per-file DAX in fuse protocol
Quoted 8 lines · Vivek Goyal
Quoted 7 lines · Vivek Goyal
Quoted 3 lines · Jeffle Xu
Expand the fuse protocol to support per-file DAX.
FUSE_PERFILE_DAXflag is added indicating if fuse server/clientShould we call this flag
FUSE_INODE_DAXinstead? It is per inode property?
Yes, strictly specking, 'per-file' is not correct.
Quoted 5 lines
I realized that you are using
FUSE_DAX_INODEto represent dax mode. So it will be confusing to useFUSE_INODE_DAXas protocol flag. How aboutFUSE_INODE_DAX_STATEinstead?
Emmm, the "_STATE" suffix is not straightforward and clear to me. How about FUSE_HAS_INODE_DAX or FUSE_DO_INODE_DAX, referring to the existing 'FUSE_HAS_IOCTL_DIR' and 'FUSE_DO_READDIRPLUS'?
Quoted 65 lines
Quoted 65 lines
Quoted 65 lines
supporting per-file DAX. It can be conveyed in both
FUSE_INITrequest and reply.
FUSE_ATTR_DAXflag is added indicating if DAX shall be enabled for corresponding file. It is conveyed inFUSE_LOOKUPreply.Signed-off-by Jeffle Xu <[email protected]>--- include/uapi/linux/fuse.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)include/uapi/linux/fuse.h +8
-1@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_PERFILE_DAX, FUSE_ATTR_DAX */ #ifndef _LINUX_FUSE_H@@ -219,7 +222,7 @@#define FUSE_KERNEL_VERSION 7/** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 34 +#define FUSE_KERNEL_MINOR_VERSION 35/** The node ID of the root inode */ #define FUSE_ROOT_ID 1@@ -336,6 +339,7 @@ struct fuse_file_lock {* write/truncate sgid is killed only if file has group * execute permission. (Same as Linux VFS behavior). * FUSE_SETXATTR_EXT: Server supports extended struct fuse_setxattr_in +* FUSE_PERFILE_DAX: kernel supports per-file DAX */ #define FUSE_ASYNC_READ (1 << 0)#define FUSE_POSIX_LOCKS (1 << 1)@@ -367,6 +371,7 @@ struct fuse_file_lock {#define FUSE_SUBMOUNTS (1 << 27)#define FUSE_HANDLE_KILLPRIV_V2 (1 << 28)#define FUSE_SETXATTR_EXT (1 << 29) +#define FUSE_PERFILE_DAX (1 << 30)/** * CUSE INIT request/reply flags @@ -449,8 +454,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)/** * Open flags Signature · 1 line
2.27.0
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerRe: [PATCH v6 3/7] fuse: support per-file DAX in fuse protocol
Quoted 15 lines · JeffleXu
Quoted 8 lines · Vivek Goyal
Quoted 7 lines · Vivek Goyal
Quoted 3 lines · Jeffle Xu
Expand the fuse protocol to support per-file DAX.
FUSE_PERFILE_DAXflag is added indicating if fuse server/clientShould we call this flag
FUSE_INODE_DAXinstead? It is per inode property?Yes, strictly specking, 'per-file' is not correct.
Quoted 1 line
Quoted 8 lines
Quoted 4 lines
I realized that you are using
FUSE_DAX_INODEto represent dax mode. So it will be confusing to useFUSE_INODE_DAXas protocol flag. How aboutFUSE_INODE_DAX_STATEinstead?Emmm, the "
_STATE" suffix is not straightforward and clear to me. How aboutFUSE_HAS_INODE_DAXorFUSE_DO_INODE_DAX, referring to the existing 'FUSE_HAS_IOCTL_DIR' and 'FUSE_DO_READDIRPLUS'?
FUSE_HAS_INODE_DAX or FUSE_DO_INODE_DAX are fine.
Vivek
Quoted 72 lines
Quoted 67 lines
Quoted 67 lines
Quoted 67 lines
supporting per-file DAX. It can be conveyed in both
FUSE_INITrequest and reply.
FUSE_ATTR_DAXflag is added indicating if DAX shall be enabled for corresponding file. It is conveyed inFUSE_LOOKUPreply.Signed-off-by Jeffle Xu <[email protected]>--- include/uapi/linux/fuse.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)include/uapi/linux/fuse.h +8
-1@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_PERFILE_DAX, FUSE_ATTR_DAX */ #ifndef _LINUX_FUSE_H@@ -219,7 +222,7 @@#define FUSE_KERNEL_VERSION 7/** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 34 +#define FUSE_KERNEL_MINOR_VERSION 35/** The node ID of the root inode */ #define FUSE_ROOT_ID 1@@ -336,6 +339,7 @@ struct fuse_file_lock {* write/truncate sgid is killed only if file has group * execute permission. (Same as Linux VFS behavior). * FUSE_SETXATTR_EXT: Server supports extended struct fuse_setxattr_in +* FUSE_PERFILE_DAX: kernel supports per-file DAX */ #define FUSE_ASYNC_READ (1 << 0)#define FUSE_POSIX_LOCKS (1 << 1)@@ -367,6 +371,7 @@ struct fuse_file_lock {#define FUSE_SUBMOUNTS (1 << 27)#define FUSE_HANDLE_KILLPRIV_V2 (1 << 28)#define FUSE_SETXATTR_EXT (1 << 29) +#define FUSE_PERFILE_DAX (1 << 30)/** * CUSE INIT request/reply flags @@ -449,8 +454,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)/** * Open flags Signature · 1 line
2.27.0Signature · 2 lines
Thanks, Jeffle
PATCH v6 4/7 fuse: negotiate per-file DAX in FUSE_INIT
In reply to Jeffle XuAmong the FUSE_INIT phase, client shall advertise per-file DAX if it's mounted with "-o dax=inode". Then server is aware that client is in per-file DAX mode, and will construct per-inode DAX attribute accordingly.
--- fs/fuse/inode.c | 2 ++ 1 file changed, 2 insertions(+)
fs/fuse/inode.c +2 -0
@@ -1203,6 +1203,8 @@ void fuse_send_init(struct fuse_mount *fm)#ifdef CONFIG_FUSE_DAX if (fm->fc->dax) ia->in.flags |=FUSE_MAP_ALIGNMENT ; +if (fm->fc->dax_mode ==FUSE_DAX_INODE ) + ia->in.flags |=FUSE_PERFILE_DAX ;#endif if (fm->fc->auto_submounts) ia->in.flags |=FUSE_SUBMOUNTS ;
Signature · 1 line
2.27.0
Vivek Goyal
maintainerRe: [PATCH v6 4/7] fuse: negotiate per-file DAX in FUSE_INIT
Quoted 15 lines · Jeffle Xu
Among the
FUSE_INITphase, client shall advertise per-file DAX if it's mounted with "-o dax=inode". Then server is aware that client is in per-file DAX mode, and will construct per-inode DAX attribute accordingly.Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/inode.c | 2 ++ 1 file changed, 2 insertions(+)fs/fuse/inode.c +0
-0
fs/fuse/inode.c void fuse_send_init(struct fuse_mount *fm) · line 1207
Quoted 5 lines
fs/fuse/inode.c +2
-0@@ -1203,6 +1203,8 @@ void fuse_send_init(struct fuse_mount *fm)#ifdef CONFIG_FUSE_DAX if (fm->fc->dax) ia->in.flags |=FUSE_MAP_ALIGNMENT ; +if (fm->fc->dax_mode ==FUSE_DAX_INODE ) + ia->in.flags |=FUSE_PERFILE_DAX ;
Are you not keeping track of server's response whether server supports per inode dax or not. Client might be new and server might be old and server might not support per inode dax. In that case, we probably should error out if user mounted with "-o dax=inode".
Vivek
Quoted 6 lines
fs/fuse/inode.c +0
-0#endif if (fm->fc->auto_submounts) ia->in.flags |=FUSE_SUBMOUNTS ;Signature · 1 line
2.27.0
Re: [PATCH v6 4/7] fuse: negotiate per-file DAX in FUSE_INIT
Quoted 11 lines · Vivek Goyal
Quoted 10 lines · Jeffle Xu
Among the
FUSE_INITphase, client shall advertise per-file DAX if it's mounted with "-o dax=inode". Then server is aware that client is in per-file DAX mode, and will construct per-inode DAX attribute accordingly.Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/inode.c | 2 ++ 1 file changed, 2 insertions(+)
Quoted 16 lines
fs/fuse/inode.c void fuse_send_init(struct fuse_mount *fm) · line 1207
Quoted 10 lines
fs/fuse/inode.c +2
-0@@ -1203,6 +1203,8 @@ void fuse_send_init(struct fuse_mount *fm)#ifdef CONFIG_FUSE_DAX if (fm->fc->dax) ia->in.flags |=FUSE_MAP_ALIGNMENT ; +if (fm->fc->dax_mode ==FUSE_DAX_INODE ) + ia->in.flags |=FUSE_PERFILE_DAX ;Are you not keeping track of server's response whether server supports per inode dax or not. Client might be new and server might be old and server might not support per inode dax. In that case, we probably should error out if user mounted with "-o dax=inode".
Yes, if guest virtiofs is mounted with '-o dax=inode' while virtiofsd is old and doesn't support per inode dax, then guest virtiofs will never receive FUSE_ATTR_DAX and actually behaves as '-o dax=never'. So the whole system works in this case, though the behavior may be beyond the expectation of users ....
If the behavior really matters, I could change the behavior and fail directly if virtiofsd doesn't advertise supporting per inode DAX.
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerRe: [PATCH v6 4/7] fuse: negotiate per-file DAX in FUSE_INIT
Quoted 31 lines · JeffleXu
Quoted 11 lines · Vivek Goyal
Quoted 10 lines · Jeffle Xu
Among the
FUSE_INITphase, client shall advertise per-file DAX if it's mounted with "-o dax=inode". Then server is aware that client is in per-file DAX mode, and will construct per-inode DAX attribute accordingly.Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/inode.c | 2 ++ 1 file changed, 2 insertions(+)Quoted 16 lines
fs/fuse/inode.c void fuse_send_init(struct fuse_mount *fm) · line 1207
Quoted 10 lines
fs/fuse/inode.c +2
-0@@ -1203,6 +1203,8 @@ void fuse_send_init(struct fuse_mount *fm)#ifdef CONFIG_FUSE_DAX if (fm->fc->dax) ia->in.flags |=FUSE_MAP_ALIGNMENT ; +if (fm->fc->dax_mode ==FUSE_DAX_INODE ) + ia->in.flags |=FUSE_PERFILE_DAX ;Are you not keeping track of server's response whether server supports per inode dax or not. Client might be new and server might be old and server might not support per inode dax. In that case, we probably should error out if user mounted with "-o dax=inode".
Quoted 8 lines
Yes, if guest virtiofs is mounted with '-o dax=inode' while virtiofsd is old and doesn't support per inode dax, then guest virtiofs will never receive
FUSE_ATTR_DAXand actually behaves as '-o dax=never'. So the whole system works in this case, though the behavior may be beyond the expectation of users ....If the behavior really matters, I could change the behavior and fail directly if virtiofsd doesn't advertise supporting per inode DAX.
I think it probably is better to error out if client asked for per-inode DAX and server does not support it.
Vivek
Quoted 5 lines
Signature · 3 lines
-- Thanks, Jeffle
PATCH v6 5/7 fuse: enable per-file DAX
In reply to Jeffle XuDAX may be limited in some specific situation. When the number of usable DAX windows is under watermark, the recalim routine will be triggered to reclaim some DAX windows. It may have a negative impact on the performance, since some processes may need to wait for DAX windows to be recalimed and reused then. To mitigate the performance degradation, the overall DAX window need to be expanded larger.
However, simply expanding the DAX window may not be a good deal in some scenario. To maintain one DAX window chunk (i.e., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest, which is greater than the memory footprint if it uses guest page cache when DAX disabled. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Per-file DAX feature is introduced to address this issue, by offering a finer grained control for dax to users, trying to achieve a balance between performance and memory overhead.
The FUSE_ATTR_DAX flag in FUSE_LOOKUP reply is used to indicate whether DAX should be enabled or not for corresponding file. Currently the state whether DAX is enabled or not for the file is initialized only when inode is instantiated.
--- fs/fuse/dax.c | 8 ++++---- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 4 ++-- fs/fuse/inode.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)
fs/fuse/dax.c +4 -4
@@ -1335,7 +1335,7 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -static bool fuse_should_enable_dax (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode,unsigned int flags) {struct fuse_conn *fc =get_fuse_conn (inode);unsigned int dax_mode = fc->dax_mode;
@@ -1352,12 +1352,12 @@ static bool fuse_should_enable_dax(struct inode *inode)return true;/* dax_mode == FUSE_DAX_INODE */ -return true; +return flags &FUSE_ATTR_DAX ; } -void fuse_dax_inode_init (struct inode *inode) +void fuse_dax_inode_init (struct inode *inode,unsigned int flags) { -if (!fuse_should_enable_dax (inode)) +if (!fuse_should_enable_dax (inode, flags))return ; inode->i_flags |=S_DAX ;
fs/fuse/file.c +2 -2
@@ -3163,7 +3163,7 @@ static const struct address_space_operations fuse_file_aops = { .write_end = fuse_write_end, }; -void fuse_init_file_inode (struct inode *inode) +void fuse_init_file_inode (struct inode *inode,unsigned int flags) {struct fuse_inode *fi =get_fuse_inode (inode);
@@ -3177,5 +3177,5 @@ void fuse_init_file_inode(struct inode *inode) fi->writepages =RB_ROOT ;if (IS_ENABLED (CONFIG_FUSE_DAX )) -fuse_dax_inode_init (inode); +fuse_dax_inode_init (inode, flags); }
fs/fuse/fuse_i.h +2 -2
@@ -1016,7 +1016,7 @@ int fuse_notify_poll_wakeup(struct fuse_conn *fc,/** * Initialize file operations on a regular file */ -void fuse_init_file_inode (struct inode *inode); +void fuse_init_file_inode (struct inode *inode,unsigned int flags);/** * Initialize inode operations on regular files and special files
@@ -1268,7 +1268,7 @@ int fuse_dax_conn_alloc(struct fuse_conn *fc, enum fuse_dax_mode mode, struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi); -void fuse_dax_inode_init (struct inode *inode); +void fuse_dax_inode_init (struct inode *inode,unsigned int flags);void fuse_dax_inode_cleanup (struct inode *inode);bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment);void fuse_dax_cancel_work (struct fuse_conn *fc);
fs/fuse/inode.c +1 -1
@@ -280,7 +280,7 @@ static void fuse_init_inode(struct inode *inode, struct fuse_attr *attr) inode->i_ctime.tv_nsec = attr->ctimensec;if (S_ISREG (inode->i_mode)) {fuse_init_common (inode); -fuse_init_file_inode (inode); +fuse_init_file_inode (inode, attr->flags); }else if (S_ISDIR (inode->i_mode))fuse_init_dir (inode);else if (S_ISLNK (inode->i_mode))
Signature · 1 line
2.27.0
Vivek Goyal
maintainerRe: [PATCH v6 5/7] fuse: enable per-file DAX
Quoted 46 lines · Jeffle Xu
DAX may be limited in some specific situation. When the number of usable DAX windows is under watermark, the recalim routine will be triggered to reclaim some DAX windows. It may have a negative impact on the performance, since some processes may need to wait for DAX windows to be recalimed and reused then. To mitigate the performance degradation, the overall DAX window need to be expanded larger.
However, simply expanding the DAX window may not be a good deal in some scenario. To maintain one DAX window chunk (i.e., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest, which is greater than the memory footprint if it uses guest page cache when DAX disabled. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Per-file DAX feature is introduced to address this issue, by offering a finer grained control for dax to users, trying to achieve a balance between performance and memory overhead.
The
FUSE_ATTR_DAXflag inFUSE_LOOKUPreply is used to indicate whether DAX should be enabled or not for corresponding file. Currently the state whether DAX is enabled or not for the file is initialized only when inode is instantiated.Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 8 ++++---- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 4 ++-- fs/fuse/inode.c | 2 +- 4 files changed, 9 insertions(+), 9 deletions(-)fs/fuse/dax.c +1
-1@@ -1335,7 +1335,7 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -static bool fuse_should_enable_dax (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode,unsigned int flags) {struct fuse_conn *fc =get_fuse_conn (inode);unsigned int dax_mode = fc->dax_mode;
fs/fuse/dax.c static bool fuse_should_enable_dax(struct inode *inode) · line 1358
Quoted 9 lines
fs/fuse/dax.c +2
-2@@ -1352,12 +1352,12 @@ static bool fuse_should_enable_dax(struct inode *inode)return true;/* dax_mode == FUSE_DAX_INODE */ -return true; +return flags &FUSE_ATTR_DAX ; } -void fuse_dax_inode_init (struct inode *inode) +void fuse_dax_inode_init (struct inode *inode,unsigned int flags)
^^^^
These are attr->flags. May be call this variable attr_flags so that it is more clear.
Vivek
Quoted 64 lines
fs/fuse/dax.c +1
-1{ -if (!fuse_should_enable_dax (inode)) +if (!fuse_should_enable_dax (inode, flags))return ; inode->i_flags |=S_DAX ;fs/fuse/file.c +2
-2@@ -3163,7 +3163,7 @@ static const struct address_space_operations fuse_file_aops = { .write_end = fuse_write_end, }; -void fuse_init_file_inode (struct inode *inode) +void fuse_init_file_inode (struct inode *inode,unsigned int flags) {struct fuse_inode *fi =get_fuse_inode (inode);@@ -3177,5 +3177,5 @@ void fuse_init_file_inode(struct inode *inode) fi->writepages =RB_ROOT ;if (IS_ENABLED (CONFIG_FUSE_DAX )) -fuse_dax_inode_init (inode); +fuse_dax_inode_init (inode, flags); }fs/fuse/fuse_i.h +2
-2@@ -1016,7 +1016,7 @@ int fuse_notify_poll_wakeup(struct fuse_conn *fc,/** * Initialize file operations on a regular file */ -void fuse_init_file_inode (struct inode *inode); +void fuse_init_file_inode (struct inode *inode,unsigned int flags);/** * Initialize inode operations on regular files and special files @@ -1268,7 +1268,7 @@ int fuse_dax_conn_alloc(struct fuse_conn *fc, enum fuse_dax_mode mode, struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi); -void fuse_dax_inode_init (struct inode *inode); +void fuse_dax_inode_init (struct inode *inode,unsigned int flags);void fuse_dax_inode_cleanup (struct inode *inode);bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment);void fuse_dax_cancel_work (struct fuse_conn *fc);fs/fuse/inode.c +1
-1@@ -280,7 +280,7 @@ static void fuse_init_inode(struct inode *inode, struct fuse_attr *attr) inode->i_ctime.tv_nsec = attr->ctimensec;if (S_ISREG (inode->i_mode)) {fuse_init_common (inode); -fuse_init_file_inode (inode); +fuse_init_file_inode (inode, attr->flags); }else if (S_ISDIR (inode->i_mode))fuse_init_dir (inode);else if (S_ISLNK (inode->i_mode))Signature · 1 line
2.27.0
PATCH v6 6/7 fuse: mark inode DONT_CACHE when per-file DAX hint changes
In reply to Jeffle XuWhen the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state.
Hence mark the inode and corresponding dentries as DONE_CACHE once the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.
In summary, when the per-file DAX hint changes for an *opened* file, the DAX state of the file won't be updated until this file is closed and reopened later.
--- fs/fuse/dax.c | 9 +++++++++ fs/fuse/fuse_i.h | 1 + fs/fuse/inode.c | 3 +++ 3 files changed, 13 insertions(+)
fs/fuse/dax.c +9 -0
@@ -1364,6 +1364,15 @@ void fuse_dax_inode_init(struct inode *inode, unsigned int flags) inode->i_data.a_ops = &fuse_dax_file_aops; } +void fuse_dax_dontcache (struct inode *inode,unsigned int flags) +{ +struct fuse_conn *fc =get_fuse_conn (inode); + +if (fc->dax_mode ==FUSE_DAX_INODE && + (!!IS_DAX (inode) != !!(flags &FUSE_ATTR_DAX ))) +d_mark_dontcache (inode); +} +bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment) {if (fc->dax && (map_alignment >FUSE_DAX_SHIFT )) {
fs/fuse/fuse_i.h +1 -0
@@ -1270,6 +1270,7 @@ void fuse_dax_conn_free(struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode,unsigned int flags);void fuse_dax_inode_cleanup (struct inode *inode); +void fuse_dax_dontcache (struct inode *inode,unsigned int flags);bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment);void fuse_dax_cancel_work (struct fuse_conn *fc);
fs/fuse/inode.c +3 -0
@@ -268,6 +268,9 @@ void fuse_change_attributes(struct inode *inode, struct fuse_attr *attr,if (inval)invalidate_inode_pages2 (inode->i_mapping); } + +if (IS_ENABLED (CONFIG_FUSE_DAX )) +fuse_dax_dontcache (inode, attr->flags); }static void fuse_init_inode (struct inode *inode,struct fuse_attr *attr)
Signature · 1 line
2.27.0
Vivek Goyal
maintainerRe: [PATCH v6 6/7] fuse: mark inode DONT_CACHE when per-file DAX hint changes
Quoted 58 lines · Jeffle Xu
When the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state.
Hence mark the inode and corresponding dentries as
DONE_CACHEonce the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.In summary, when the per-file DAX hint changes for an *opened* file, the DAX state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 9 +++++++++ fs/fuse/fuse_i.h | 1 + fs/fuse/inode.c | 3 +++ 3 files changed, 13 insertions(+)fs/fuse/dax.c +9
-0@@ -1364,6 +1364,15 @@ void fuse_dax_inode_init(struct inode *inode, unsigned int flags) inode->i_data.a_ops = &fuse_dax_file_aops; } +void fuse_dax_dontcache (struct inode *inode,unsigned int flags) +{ +struct fuse_conn *fc =get_fuse_conn (inode); + +if (fc->dax_mode ==FUSE_DAX_INODE && + (!!IS_DAX (inode) != !!(flags &FUSE_ATTR_DAX ))) +d_mark_dontcache (inode); +} +bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment) {if (fc->dax && (map_alignment >FUSE_DAX_SHIFT )) {fs/fuse/fuse_i.h +1
-0@@ -1270,6 +1270,7 @@ void fuse_dax_conn_free(struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode,unsigned int flags);void fuse_dax_inode_cleanup (struct inode *inode); +void fuse_dax_dontcache (struct inode *inode,unsigned int flags);bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment);void fuse_dax_cancel_work (struct fuse_conn *fc);fs/fuse/inode.c +0
-0
fs/fuse/inode.c void fuse_change_attributes(struct inode *inode, struct fuse_attr *attr, · line 273
Quoted 6 lines
fs/fuse/inode.c +3
-0@@ -268,6 +268,9 @@ void fuse_change_attributes(struct inode *inode, struct fuse_attr *attr,if (inval)invalidate_inode_pages2 (inode->i_mapping); } + +if (IS_ENABLED (CONFIG_FUSE_DAX )) +fuse_dax_dontcache (inode, attr->flags);
Should we give this function more generic name. Say fuse_dax_change_attributes(). And let that function decide what attributes have changed and does it need to take any action.
Vivek
Quoted 6 lines
fs/fuse/inode.c +0
-0}static void fuse_init_inode (struct inode *inode,struct fuse_attr *attr)Signature · 1 line
2.27.0
Re: [PATCH v6 6/7] fuse: mark inode DONT_CACHE when per-file DAX hint changes
Sorry for the late reply, as your previous reply was moved to junk box by the algorithm...
Quoted 54 lines · Vivek Goyal
Quoted 53 lines · Jeffle Xu
When the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state.
Hence mark the inode and corresponding dentries as
DONE_CACHEonce the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.In summary, when the per-file DAX hint changes for an *opened* file, the DAX state of the file won't be updated until this file is closed and reopened later.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 9 +++++++++ fs/fuse/fuse_i.h | 1 + fs/fuse/inode.c | 3 +++ 3 files changed, 13 insertions(+)fs/fuse/dax.c +9
-0@@ -1364,6 +1364,15 @@ void fuse_dax_inode_init(struct inode *inode, unsigned int flags) inode->i_data.a_ops = &fuse_dax_file_aops; } +void fuse_dax_dontcache (struct inode *inode,unsigned int flags) +{ +struct fuse_conn *fc =get_fuse_conn (inode); + +if (fc->dax_mode ==FUSE_DAX_INODE && + (!!IS_DAX (inode) != !!(flags &FUSE_ATTR_DAX ))) +d_mark_dontcache (inode); +} +bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment) {if (fc->dax && (map_alignment >FUSE_DAX_SHIFT )) {fs/fuse/fuse_i.h +1
-0@@ -1270,6 +1270,7 @@ void fuse_dax_conn_free(struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode,unsigned int flags);void fuse_dax_inode_cleanup (struct inode *inode); +void fuse_dax_dontcache (struct inode *inode,unsigned int flags);bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment);void fuse_dax_cancel_work (struct fuse_conn *fc);
Quoted 16 lines
fs/fuse/inode.c void fuse_change_attributes(struct inode *inode, struct fuse_attr *attr, · line 273
Quoted 11 lines
fs/fuse/inode.c +3
-0@@ -268,6 +268,9 @@ void fuse_change_attributes(struct inode *inode, struct fuse_attr *attr,if (inval)invalidate_inode_pages2 (inode->i_mapping); } + +if (IS_ENABLED (CONFIG_FUSE_DAX )) +fuse_dax_dontcache (inode, attr->flags);Should we give this function more generic name. Say
fuse_dax_change_attributes(). And let that function decide what attributes have changed and does it need to take any action.
But currently we only need to handle 'attr->flags & FUSE_ATTR_DAX'. If other attributes need to be handled later, then we can expand this function and give it a more generic name. But as for now, I prefer to keep it simple.
Signature · 2 lines
Thanks, Jeffle
PATCH v6 7/7 Documentation/filesystem/dax: record DAX on virtiofs
In reply to Jeffle XuRecord DAX on virtiofs and the semantic difference with that on ext4 and xfs.
--- Documentation/filesystems/dax.rst | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
Documentation/filesystems/dax.rst +18 -2
@@ -23,8 +23,8 @@ on it as usual. The `DAX` code currently only supports files with a block size equal to your kernel's `PAGE_SIZE`, so you may need to specify a block size when creating the filesystem. -Currently 3 filesystems support `DAX`: ext2, ext4 and xfs. Enabling `DAX` on them -is different. +Currently 4 filesystems support `DAX`: ext2, ext4, xfs and virtiofs. +Enabling `DAX` on them is different. Enabling DAX on ext2 --------------------
@@ -168,6 +168,22 @@ if the underlying media does not support dax and/or the filesystem is overridden with a mount option. +Enabling DAX on virtiofs +---------------------------- +The semantic of DAX on virtiofs is basically equal to that on ext4 and xfs, +except that when '-o dax=inode' is specified, virtiofs client derives the hint +whether DAX shall be enabled or not from virtiofs server through FUSE protocol, +rather than the persistent `FS_XFLAG_DAX` flag. That is, whether DAX shall be +enabled or not is completely determined by virtiofs server, while virtiofs +server itself may deploy various algorithm making this decision, e.g. depending +on the persistent `FS_XFLAG_DAX` flag on the host. + +It is still supported to set or clear persistent `FS_XFLAG_DAX` flag inside +guest, but it is not guaranteed that DAX will be enabled or disabled for +corresponding file then. Users inside guest still need to call statx(2) and +check the statx flag `STATX_ATTR_DAX` to see if DAX is enabled for this file. + + Implementation Tips for Block Driver Writers --------------------------------------------
Signature · 1 line
2.27.0
Hi, any comment?
Quoted 124 lines · Jeffle Xu
changes since v5:
Overall Design Changes:
1. virtiofsd now supports ioctl (onlyFS_IOC_SETFLAGSandFS_IOC_FSSETXATTR), so that users inside guest could set/clear persistent inode flags now. (FUSE kernel module has already supported .ioctl(), virtiofsd need to suuport it.)
2. When FUSE client is mounted with '-o dax=inode', it indicates that whether DAX shall be enabled or not for one specific file is completely determined by FUSE server while FUSE client has no say on it, and the decision whether DAX shall be enabled or not for specific file is communicated throughFUSE_ATTR_DAXflag of FUSE protocol. The algorithm used by virtiofsd to determine whether DAX shall be enabled or not is totally implementation specific, and thus the following scenario may exist: users inside guest has already set related persistent inode flag (i.e.FS_XFLAG_DAX) on corresponding file but FUSE server finnaly decides not to enable DAX for this file. This slight semantic difference is documented in patch 7. Also because of this,d_mark_dontcache()is not called when FS_IOC_SETFLAGS/FS_IOC_FSSETXATTR ioctl is done inside guest. It's delayed to be done if theFUSE_ATTR_DAXflag **indeed** changes (as showed in patch 6).
3. patch 1: slightly modify logic offuse_should_enable_dax()
4. patch 4: add back negotiation duringFUSE_INIT. FUSE client shall advertise to FUSE server that it's in per-file DAX mode, so that FUSE server may omit querying persistent inode flags on host if FUSE client is not mounted in per-file DAX mode, giving querying persistent inode flags could be quite expensive.chanegs since v4:
- drop support for setting/clearingFS_DAXinside guest
- and thus drop the negotiation phase duringFUSE_INITThis patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
Any comment is welcome.
[1] commit
9cb20f94afcd("fs/ext4: Make DAX mount option a tri-state")
[2] commit02beb2686ff9("fs/xfs: Make DAX mount option a tri-state")[Purpose]
DAX may be limited in some specific situation. When the number of usable DAX windows is under watermark, the recalim routine will be triggered to reclaim some DAX windows. It may have a negative impact on the performance, since some processes may need to wait for DAX windows to be recalimed and reused then. To mitigate the performance degradation, the overall DAX window need to be expanded larger.However, simply expanding the DAX window may not be a good deal in some scenario. To maintain one DAX window chunk (i.e., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest, which is greater than the memory footprint if it uses guest page cache when DAX disabled. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Per-file DAX feature is introduced to address this issue, by offering a finer grained control for dax to users, trying to achieve a balance between performance and memory overhead.
[Note]
When the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state, since dynamic switching needs to switcha_opsatomiclly. Ira Weiny had ever implemented a so calledi_aops_semlock [3] but eventually gave up since the complexity of the implementation [4][5][6][7].Hence mark the inode and corresponding dentries as
DONE_CACHEonce the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.In summary, when the per-file DAX hint changes for an *opened* file, the DAX state of the file won't be updated until this file is closed and reopened later. This is also how ext4/xfs per-file DAX works.
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://patchwork.kernel.org/project/xfs/cover/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/
[6] https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/
[7] https://lore.kernel.org/lkml/20200403182904.GP80283@magnolia/changes since v3:
- bug fix (patch 6): s/"IS_DAX(inode) != newdax"/"!!IS_DAX(inode) != newdax"
- duringFUSE_INIT, advertise capability for per-file DAX only when mounted as "-o dax=inode" (patch 4)changes since v2:
- modifyfuse_show_options()accordingly to make it compatible with new tri-state mount option (patch 2)
- extract FUSE protocol changes into one separate patch (patch 3)
- FUSE server/client need to negotiate if they support per-file DAX (patch 4)
- extractDONT_CACHElogic into patch 6/7v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/linux-fsdevel/[email protected]/
v3: https://www.spinics.net/lists/linux-fsdevel/msg200852.html
v2: https://www.spinics.net/lists/linux-fsdevel/msg199584.html
v1: https://www.spinics.net/lists/linux-virtualization/msg51008.htmlJeffle Xu (7):
fuse: addfuse_should_enable_dax()helper
fuse: make DAX mount option a tri-state
fuse: support per-file DAX in fuse protocol
fuse: negotiate per-file DAX inFUSE_INIT
fuse: enable per-file DAX
fuse: mark inodeDONT_CACHEwhen per-file DAX hint changesDocumentation/filesystem/dax: record DAX on virtiofsDocumentation/filesystems/dax.rst | 20 +++++++++++++++-- fs/fuse/dax.c | 36 ++++++++++++++++++++++++++++--- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 19 ++++++++++++---- fs/fuse/inode.c | 17 +++++++++++---- fs/fuse/virtio_fs.c | 16 ++++++++++++-- include/uapi/linux/fuse.h | 9 +++++++- 7 files changed, 103 insertions(+), 18 deletions(-)
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerQuoted 6 lines · Jeffle Xu
changes since v5:
Overall Design Changes:
1. virtiofsd now supports ioctl (onlyFS_IOC_SETFLAGSandFS_IOC_FSSETXATTR), so that users inside guest could set/clear persistent inode flags now. (FUSE kernel module has already supported .ioctl(), virtiofsd need to suuport it.)
So no changes needed in fuse side (kernel) to support FS_IOC_FSSETXATTR? Only virtiofsd needs to be changed. That sounds good.
Vivek
Quoted 121 lines
2. When FUSE client is mounted with '-o dax=inode', it indicates that whether DAX shall be enabled or not for one specific file is completely determined by FUSE server while FUSE client has no say on it, and the decision whether DAX shall be enabled or not for specific file is communicated through
FUSE_ATTR_DAXflag of FUSE protocol. The algorithm used by virtiofsd to determine whether DAX shall be enabled or not is totally implementation specific, and thus the following scenario may exist: users inside guest has already set related persistent inode flag (i.e.FS_XFLAG_DAX) on corresponding file but FUSE server finnaly decides not to enable DAX for this file. This slight semantic difference is documented in patch 7. Also because of this,d_mark_dontcache()is not called when FS_IOC_SETFLAGS/FS_IOC_FSSETXATTR ioctl is done inside guest. It's delayed to be done if theFUSE_ATTR_DAXflag **indeed** changes (as showed in patch 6).
3. patch 1: slightly modify logic offuse_should_enable_dax()
4. patch 4: add back negotiation duringFUSE_INIT. FUSE client shall advertise to FUSE server that it's in per-file DAX mode, so that FUSE server may omit querying persistent inode flags on host if FUSE client is not mounted in per-file DAX mode, giving querying persistent inode flags could be quite expensive.chanegs since v4:
- drop support for setting/clearingFS_DAXinside guest
- and thus drop the negotiation phase duringFUSE_INITThis patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
Any comment is welcome.
[1] commit
9cb20f94afcd("fs/ext4: Make DAX mount option a tri-state")
[2] commit02beb2686ff9("fs/xfs: Make DAX mount option a tri-state")[Purpose]
DAX may be limited in some specific situation. When the number of usable DAX windows is under watermark, the recalim routine will be triggered to reclaim some DAX windows. It may have a negative impact on the performance, since some processes may need to wait for DAX windows to be recalimed and reused then. To mitigate the performance degradation, the overall DAX window need to be expanded larger.However, simply expanding the DAX window may not be a good deal in some scenario. To maintain one DAX window chunk (i.e., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest, which is greater than the memory footprint if it uses guest page cache when DAX disabled. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Per-file DAX feature is introduced to address this issue, by offering a finer grained control for dax to users, trying to achieve a balance between performance and memory overhead.
[Note]
When the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state, since dynamic switching needs to switcha_opsatomiclly. Ira Weiny had ever implemented a so calledi_aops_semlock [3] but eventually gave up since the complexity of the implementation [4][5][6][7].Hence mark the inode and corresponding dentries as
DONE_CACHEonce the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.In summary, when the per-file DAX hint changes for an *opened* file, the DAX state of the file won't be updated until this file is closed and reopened later. This is also how ext4/xfs per-file DAX works.
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://patchwork.kernel.org/project/xfs/cover/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/
[6] https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/
[7] https://lore.kernel.org/lkml/20200403182904.GP80283@magnolia/changes since v3:
- bug fix (patch 6): s/"IS_DAX(inode) != newdax"/"!!IS_DAX(inode) != newdax"
- duringFUSE_INIT, advertise capability for per-file DAX only when mounted as "-o dax=inode" (patch 4)changes since v2:
- modifyfuse_show_options()accordingly to make it compatible with new tri-state mount option (patch 2)
- extract FUSE protocol changes into one separate patch (patch 3)
- FUSE server/client need to negotiate if they support per-file DAX (patch 4)
- extractDONT_CACHElogic into patch 6/7v5: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/linux-fsdevel/[email protected]/
v3: https://www.spinics.net/lists/linux-fsdevel/msg200852.html
v2: https://www.spinics.net/lists/linux-fsdevel/msg199584.html
v1: https://www.spinics.net/lists/linux-virtualization/msg51008.htmlJeffle Xu (7):
fuse: addfuse_should_enable_dax()helper
fuse: make DAX mount option a tri-state
fuse: support per-file DAX in fuse protocol
fuse: negotiate per-file DAX inFUSE_INIT
fuse: enable per-file DAX
fuse: mark inodeDONT_CACHEwhen per-file DAX hint changesDocumentation/filesystem/dax: record DAX on virtiofsDocumentation/filesystems/dax.rst | 20 +++++++++++++++-- fs/fuse/dax.c | 36 ++++++++++++++++++++++++++++--- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 19 ++++++++++++---- fs/fuse/inode.c | 17 +++++++++++---- fs/fuse/virtio_fs.c | 16 ++++++++++++-- include/uapi/linux/fuse.h | 9 +++++++- 7 files changed, 103 insertions(+), 18 deletions(-)Signature · 1 line
2.27.0
Quoted 11 lines · Vivek Goyal
Quoted 6 lines · Jeffle Xu
changes since v5:
Overall Design Changes:
1. virtiofsd now supports ioctl (onlyFS_IOC_SETFLAGSandFS_IOC_FSSETXATTR), so that users inside guest could set/clear persistent inode flags now. (FUSE kernel module has already supported .ioctl(), virtiofsd need to suuport it.)So no changes needed in fuse side (kernel) to support
FS_IOC_FSSETXATTR? Only virtiofsd needs to be changed. That sounds good.
Yes, the fuse kernel modules has already supported FUSE_IOCTL.
Per inode DAX on ext4/xfs will also call d_mark_dontcache() and try to evict this inode as soon as possible when the persistent (DAX) inode attribute has changed, just like [1].
But because of following reason:
Quoted 7 lines
Quoted 7 lines
2. The
algorithm used by virtiofsd to determine whether DAX shall be enabled or not is totally implementation specific, and thus the following scenario may exist: users inside guest has already set related persistent inode flag (i.e.FS_XFLAG_DAX) on corresponding file but FUSE server finnaly decides not to enable DAX for this file.
If we always call d_mark_dontcache() and try to evict this inode when the persistent (DAX) inode attribute has changed, the DAX state returned by virtiofsd may sustain the same, and thus the previous eviction is totally wasted and unnecessary.
So, as the following said,
Quoted 4 lines
Quoted 4 lines
Also because of this,
d_mark_dontcache()is
not called when FS_IOC_SETFLAGS/FS_IOC_FSSETXATTR ioctl is done inside guest. It's delayed to be done if theFUSE_ATTR_DAXflag **indeed** changes (as showed in patch 6).
the call for d_mark_dontcache() and inode eviction is delayed when the DAX state returned by virtiofsd **indeed** changed (when dentry is timed out and a new FUSE_LOOKUP is requested). But the defect is that, if '-o cache=always' is set for virtiofsd, then the DAX state won't be updated for a long time, after users have changed the persistent (DAX) inode attribute inside guest via FS_IOC_FSSETXATTR ioctl.
[1] https://www.spinics.net/lists/linux-fsdevel/msg200851.html
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerQuoted 42 lines · JeffleXu
Quoted 11 lines · Vivek Goyal
Quoted 6 lines · Jeffle Xu
changes since v5:
Overall Design Changes:
1. virtiofsd now supports ioctl (onlyFS_IOC_SETFLAGSandFS_IOC_FSSETXATTR), so that users inside guest could set/clear persistent inode flags now. (FUSE kernel module has already supported .ioctl(), virtiofsd need to suuport it.)So no changes needed in fuse side (kernel) to support
FS_IOC_FSSETXATTR? Only virtiofsd needs to be changed. That sounds good.Yes, the fuse kernel modules has already supported
FUSE_IOCTL.Per inode DAX on ext4/xfs will also call
d_mark_dontcache()and try to evict this inode as soon as possible when the persistent (DAX) inode attribute has changed, just like [1].But because of following reason:
Quoted 7 lines
Quoted 7 lines
2. The
algorithm used by virtiofsd to determine whether DAX shall be enabled or not is totally implementation specific, and thus the following scenario may exist: users inside guest has already set related persistent inode flag (i.e.FS_XFLAG_DAX) on corresponding file but FUSE server finnaly decides not to enable DAX for this file.If we always call
d_mark_dontcache()and try to evict this inode when the persistent (DAX) inode attribute has changed, the DAX state returned by virtiofsd may sustain the same, and thus the previous eviction is totally wasted and unnecessary.So, as the following said,
Quoted 4 lines
Quoted 4 lines
Also because of this,
d_mark_dontcache()is
not called when FS_IOC_SETFLAGS/FS_IOC_FSSETXATTR ioctl is done inside guest. It's delayed to be done if theFUSE_ATTR_DAXflag **indeed** changes (as showed in patch 6).
Quoted 6 lines
the call for
d_mark_dontcache()and inode eviction is delayed when the DAX state returned by virtiofsd **indeed** changed (when dentry is timed out and a newFUSE_LOOKUPis requested). But the defect is that, if '-o cache=always' is set for virtiofsd, then the DAX state won't be updated for a long time, after users have changed the persistent (DAX) inode attribute inside guest viaFS_IOC_FSSETXATTRioctl.
Good point. I guess this probably is not too bad. If it becomes a concern, we can always mark inode don't cache whenever client changes persistent DAX flag.
Vivek
Quoted 9 lines
[1] https://www.spinics.net/lists/linux-fsdevel/msg200851.html
Signature · 2 lines
Thanks, Jeffle