fuse,virtiofs: support per-file DAX
Subsystems (3)
v5 cover fuse,virtiofs: support per-file DAX
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
Any comment is welcome.
[1] commit 9cb20f94afcd ("fs/ext4: Make DAX mount option a tri-state")
[2] commit 02beb2686ff9 ("fs/xfs: Make DAX mount option a tri-state")
[Purpose]
DAX may be limited in some specific situation. When the number of usable DAX windows is under watermark, the recalim routine will be triggered to reclaim some DAX windows. It may have a negative impact on the performance, since some processes may need to wait for DAX windows to be recalimed and reused then. To mitigate the performance degradation, the overall DAX window need to be expanded larger.
However, simply expanding the DAX window may not be a good deal in some scenario. To maintain one DAX window chunk (i.e., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest, which is greater than the memory footprint if it uses guest page cache when DAX disabled. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Per-file DAX feature is introduced to address this issue, by offering a finer grained control for dax to users, trying to achieve a balance between performance and memory overhead.
[Note]
When the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state, since dynamic switching needs to switch a_ops atomiclly. Ira Weiny had ever implemented a so called i_aops_sem lock [3] but eventually gave up since the complexity of the implementation [4][5][6][7].
Hence mark the inode and corresponding dentries as DONE_CACHE once the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.
In summary, when the per-file DAX hint changes for an *opened* file, the DAX state of the file won't be updated until this file is closed and reopened later. This is also how ext4/xfs per-file DAX works.
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://patchwork.kernel.org/project/xfs/cover/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/
[6] https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/
[7] https://lore.kernel.org/lkml/20200403182904.GP80283@magnolia/
chanegs since v4:
- drop support for setting/clearing FS_DAX inside guest
- and thus drop the negotiation phase during FUSE_INIT
changes since v3:
- bug fix (patch 6): s/"IS_DAX(inode) != newdax"/"!!IS_DAX(inode) != newdax"
- during FUSE_INIT, advertise capability for per-file DAX only when mounted as "-o dax=inode" (patch 4)
changes since v2:
- modify fuse_show_options() accordingly to make it compatible with new tri-state mount option (patch 2)
- extract FUSE protocol changes into one seperate patch (patch 3)
- FUSE server/client need to negotiate if they support per-file DAX (patch 4)
- extract DONT_CACHE logic into patch 6/7
v4: https://lore.kernel.org/linux-fsdevel/[email protected]/
v3: https://www.spinics.net/lists/linux-fsdevel/msg200852.html
v2: https://www.spinics.net/lists/linux-fsdevel/msg199584.html
v1: https://www.spinics.net/lists/linux-virtualization/msg51008.html
Jeffle Xu (5):
fuse: add fuse_should_enable_dax() helper
fuse: make DAX mount option a tri-state
fuse: support per-file DAX
fuse: enable per-file DAX
fuse: mark inode DONT_CACHE when per-file DAX hint changes
fs/fuse/dax.c | 32 +++++++++++++++++++++++++++++--- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 19 +++++++++++++++---- fs/fuse/inode.c | 15 +++++++++++---- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- include/uapi/linux/fuse.h | 7 ++++++- 6 files changed, 77 insertions(+), 16 deletions(-)
Signature · 1 line
2.27.0
PATCH v5 1/5 fuse: add fuse_should_enable_dax() helper
This is in prep for following per-file DAX checking.
--- fs/fuse/dax.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
fs/fuse/dax.c +9 -1
@@ -1330,11 +1330,19 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -void fuse_dax_inode_init (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode);if (!fc->dax) +return false; + +return true; +} + +void fuse_dax_inode_init (struct inode *inode) +{ +if (!fuse_should_enable_dax (inode))return ; inode->i_flags |=S_DAX ;
Signature · 1 line
2.27.0
Vivek Goyal
maintainerRe: [PATCH v5 1/5] fuse: add fuse_should_enable_dax() helper
Quoted 15 lines · Jeffle Xu
This is in prep for following per-file DAX checking.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)fs/fuse/dax.c +0
-0@@ -1330,11 +1330,19 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, };
fs/fuse/dax.c static const struct address_space_operations fuse_dax_file_aops = { · line 1341
Quoted 10 lines
fs/fuse/dax.c +5
-1-void fuse_dax_inode_init (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode);if (!fc->dax) +return false; + +return true; +}
It probably is easier to read if we flip the logic.
{
if (fc->dax)
return true;
return false;
}Quoted 10 lines
fs/fuse/dax.c +4
-0+ +void fuse_dax_inode_init (struct inode *inode) +{ +if (!fuse_should_enable_dax (inode))return ; inode->i_flags |=S_DAX ;Signature · 1 line
2.27.0
PATCH v5 2/5 fuse: make DAX mount option a tri-state
In reply to Jeffle XuWe add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
By the time this patch is applied, 'inode' mode is actually equal to 'always' mode, before the per-file DAX flag is introduced in the following patch.
--- fs/fuse/dax.c | 9 +++++++-- fs/fuse/fuse_i.h | 14 ++++++++++++-- fs/fuse/inode.c | 10 +++++++--- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- 4 files changed, 40 insertions(+), 9 deletions(-)
fs/fuse/dax.c +7 -2
@@ -1282,11 +1282,14 @@ static int fuse_dax_mem_range_init(struct fuse_conn_dax *fcd)return ret; } -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev) +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode dax_mode, +struct dax_device *dax_dev) {struct fuse_conn_dax *fcd;int err; + fc->dax_mode = dax_mode; +if (!dax_dev)return 0 ;
@@ -1333,8 +1336,10 @@ static const struct address_space_operations fuse_dax_file_aops = {static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); +unsigned int dax_mode = fc->dax_mode; -if (!fc->dax) +/* If 'dax=always/inode', fc->dax couldn't be NULL */ +if (dax_mode ==FUSE_DAX_NEVER )return false;return true;
fs/fuse/fuse_i.h +12 -2
@@ -480,6 +480,12 @@ struct fuse_dev {struct list_head entry; }; +enum fuse_dax_mode { +FUSE_DAX_INODE , +FUSE_DAX_ALWAYS , +FUSE_DAX_NEVER , +}; +struct fuse_fs_context {int fd;struct file *file;
@@ -497,7 +503,7 @@ struct fuse_fs_context { bool no_control:1; bool no_force_umount:1; bool legacy_opts_show:1; - bool dax:1; +enum fuse_dax_mode dax_mode;unsigned int max_read;unsigned int blksize;const char *subtype;
@@ -802,6 +808,9 @@ struct fuse_conn {struct list_head devices;#ifdef CONFIG_FUSE_DAX +/* dax mode: FUSE_DAX_* (always, never or per-file) */ +enum fuse_dax_mode dax_mode; +/* Dax specific conn data, non-NULL if DAX is enabled */ struct fuse_conn_dax *dax;#endif
@@ -1255,7 +1264,8 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);int fuse_dax_mmap (struct file *file,struct vm_area_struct *vma);int fuse_dax_break_layouts (struct inode *inode,u64 dmap_start,u64 dmap_end); -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev); +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode mode, +struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode);
fs/fuse/inode.c +7 -3
@@ -742,8 +742,12 @@ static int fuse_show_options(struct seq_file *m, struct dentry *root)seq_printf (m,",blksize=%lu" , sb->s_blocksize); }#ifdef CONFIG_FUSE_DAX -if (fc->dax) -seq_puts (m,",dax" ); +if (fc->dax_mode ==FUSE_DAX_ALWAYS ) +seq_puts (m,",dax=always" ); +else if (fc->dax_mode ==FUSE_DAX_NEVER ) +seq_puts (m,",dax=never" ); +else if (fc->dax_mode ==FUSE_DAX_INODE ) +seq_puts (m,",dax=inode" );#endif return 0 ;
@@ -1493,7 +1497,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) sb->s_subtype = ctx->subtype; ctx->subtype =NULL ;if (IS_ENABLED (CONFIG_FUSE_DAX )) { - err =fuse_dax_conn_alloc (fc, ctx->dax_dev); + err =fuse_dax_conn_alloc (fc, ctx->dax_mode, ctx->dax_dev);if (err) goto err; }
fs/fuse/virtio_fs.c +14 -2
@@ -88,12 +88,21 @@ struct virtio_fs_req_work {static int virtio_fs_enqueue_req (struct virtio_fs_vq *fsvq,struct fuse_req *req,bool in_flight); +static const struct constant_table dax_param_enums[] = { + {"inode" ,FUSE_DAX_INODE }, + {"always" ,FUSE_DAX_ALWAYS }, + {"never" ,FUSE_DAX_NEVER }, + {} +}; +enum {OPT_DAX , +OPT_DAX_ENUM , };static const struct fs_parameter_spec virtio_fs_parameters[] = {fsparam_flag ("dax" ,OPT_DAX ), +fsparam_enum ("dax" ,OPT_DAX_ENUM , dax_param_enums), {} };
@@ -110,7 +119,10 @@ static int virtio_fs_parse_param(struct fs_context *fsc,switch (opt) {case OPT_DAX : - ctx->dax =1 ; + ctx->dax_mode =FUSE_DAX_ALWAYS ; +break ; +case OPT_DAX_ENUM : + ctx->dax_mode = result.uint_32;break ;default :return -EINVAL ;
@@ -1326,7 +1338,7 @@ static int virtio_fs_fill_super(struct super_block *sb, struct fs_context *fsc)/* virtiofs allocates and installs its own fuse devices */ ctx->fudptr =NULL ; -if (ctx->dax) { +if (ctx->dax_mode !=FUSE_DAX_NEVER ) {if (!fs->dax_dev) { err = -EINVAL ; pr_err("virtio-fs: dax can't be enabled as filesystem"
Signature · 1 line
2.27.0
Vivek Goyal
maintainerRe: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
So will "-o dax=inode" be used for per file DAX where dax mode comes from server?
I think we discussed this. It will be better to leave "-o dax=inode" alone. It should be used when we are reading dax status from file attr (like ext4 and xfs).
And probably create separate option say "-o dax=server" where server specifies which inode should use dax.
Otherwise it will be very confusing. People familiar with "-o dax=inode" on ext4/xfs will expect file attr to work and that's not what we are implementing, IIUC.
Vivek
Quoted 170 lines
By the time this patch is applied, 'inode' mode is actually equal to 'always' mode, before the per-file DAX flag is introduced in the following patch.
Signed-off-by Jeffle Xu <[email protected]>--- fs/fuse/dax.c | 9 +++++++-- fs/fuse/fuse_i.h | 14 ++++++++++++-- fs/fuse/inode.c | 10 +++++++--- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- 4 files changed, 40 insertions(+), 9 deletions(-)fs/fuse/dax.c +7
-2@@ -1282,11 +1282,14 @@ static int fuse_dax_mem_range_init(struct fuse_conn_dax *fcd)return ret; } -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev) +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode dax_mode, +struct dax_device *dax_dev) {struct fuse_conn_dax *fcd;int err; + fc->dax_mode = dax_mode; +if (!dax_dev)return 0 ;@@ -1333,8 +1336,10 @@ static const struct address_space_operations fuse_dax_file_aops = {static bool fuse_should_enable_dax (struct inode *inode) {struct fuse_conn *fc =get_fuse_conn (inode); +unsigned int dax_mode = fc->dax_mode; -if (!fc->dax) +/* If 'dax=always/inode', fc->dax couldn't be NULL */ +if (dax_mode ==FUSE_DAX_NEVER )return false;return true;fs/fuse/fuse_i.h +12
-2@@ -480,6 +480,12 @@ struct fuse_dev {struct list_head entry; }; +enum fuse_dax_mode { +FUSE_DAX_INODE , +FUSE_DAX_ALWAYS , +FUSE_DAX_NEVER , +}; +struct fuse_fs_context {int fd;struct file *file;@@ -497,7 +503,7 @@ struct fuse_fs_context { bool no_control:1; bool no_force_umount:1; bool legacy_opts_show:1; - bool dax:1; +enum fuse_dax_mode dax_mode;unsigned int max_read;unsigned int blksize;const char *subtype;@@ -802,6 +808,9 @@ struct fuse_conn {struct list_head devices;#ifdef CONFIG_FUSE_DAX +/* dax mode: FUSE_DAX_* (always, never or per-file) */ +enum fuse_dax_mode dax_mode; +/* Dax specific conn data, non-NULL if DAX is enabled */ struct fuse_conn_dax *dax;#endif @@ -1255,7 +1264,8 @@ ssize_t fuse_dax_read_iter(struct kiocb *iocb, struct iov_iter *to);ssize_t fuse_dax_write_iter (struct kiocb *iocb,struct iov_iter *from);int fuse_dax_mmap (struct file *file,struct vm_area_struct *vma);int fuse_dax_break_layouts (struct inode *inode,u64 dmap_start,u64 dmap_end); -int fuse_dax_conn_alloc (struct fuse_conn *fc,struct dax_device *dax_dev); +int fuse_dax_conn_alloc (struct fuse_conn *fc,enum fuse_dax_mode mode, +struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode);fs/fuse/inode.c +7
-3@@ -742,8 +742,12 @@ static int fuse_show_options(struct seq_file *m, struct dentry *root)seq_printf (m,",blksize=%lu" , sb->s_blocksize); }#ifdef CONFIG_FUSE_DAX -if (fc->dax) -seq_puts (m,",dax" ); +if (fc->dax_mode ==FUSE_DAX_ALWAYS ) +seq_puts (m,",dax=always" ); +else if (fc->dax_mode ==FUSE_DAX_NEVER ) +seq_puts (m,",dax=never" ); +else if (fc->dax_mode ==FUSE_DAX_INODE ) +seq_puts (m,",dax=inode" );#endif return 0 ;@@ -1493,7 +1497,7 @@ int fuse_fill_super_common(struct super_block *sb, struct fuse_fs_context *ctx) sb->s_subtype = ctx->subtype; ctx->subtype =NULL ;if (IS_ENABLED (CONFIG_FUSE_DAX )) { - err =fuse_dax_conn_alloc (fc, ctx->dax_dev); + err =fuse_dax_conn_alloc (fc, ctx->dax_mode, ctx->dax_dev);if (err) goto err; }fs/fuse/virtio_fs.c +14
-2@@ -88,12 +88,21 @@ struct virtio_fs_req_work {static int virtio_fs_enqueue_req (struct virtio_fs_vq *fsvq,struct fuse_req *req,bool in_flight); +static const struct constant_table dax_param_enums[] = { + {"inode" ,FUSE_DAX_INODE }, + {"always" ,FUSE_DAX_ALWAYS }, + {"never" ,FUSE_DAX_NEVER }, + {} +}; +enum {OPT_DAX , +OPT_DAX_ENUM , };static const struct fs_parameter_spec virtio_fs_parameters[] = {fsparam_flag ("dax" ,OPT_DAX ), +fsparam_enum ("dax" ,OPT_DAX_ENUM , dax_param_enums), {} };@@ -110,7 +119,10 @@ static int virtio_fs_parse_param(struct fs_context *fsc,switch (opt) {case OPT_DAX : - ctx->dax =1 ; + ctx->dax_mode =FUSE_DAX_ALWAYS ; +break ; +case OPT_DAX_ENUM : + ctx->dax_mode = result.uint_32;break ;default :return -EINVAL ;@@ -1326,7 +1338,7 @@ static int virtio_fs_fill_super(struct super_block *sb, struct fs_context *fsc)/* virtiofs allocates and installs its own fuse devices */ ctx->fudptr =NULL ; -if (ctx->dax) { +if (ctx->dax_mode !=FUSE_DAX_NEVER ) {if (!fs->dax_dev) { err = -EINVAL ; pr_err("virtio-fs: dax can't be enabled as filesystem"Signature · 1 line
2.27.0
Re: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 9 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
So will "-o dax=inode" be used for per file DAX where dax mode comes from server?
Quoted 6 lines
I think we discussed this. It will be better to leave "-o dax=inode" alone. It should be used when we are reading dax status from file attr (like ext4 and xfs).
And probably create separate option say "-o dax=server" where server specifies which inode should use dax.
That seems like a poor idea to me.
The server side already controls what the client side does by controlling the inode attributes that the client side sees. That is, if the server is going to specify whether the client side data access is going to use dax, then the server presents the client with an inode that has the DAX attribute flag set on it.
In that case, turning off dax on the guest side should be communicated to the fuse server so the server turns off the DAX flag on the server side iff server side policy allows it. When the guest then purges it's local inode and reloads it from the server then it will get an inode with the DAX flag set according to server side policy.
Hence if the server side is configured with "dax=always" or dax="never" semantics it means the client side inode flag state cannot control DAX mode. That means, regardless of the client side mount options, DAX is operating under always or never policy, enforced by the server side by direct control of the client inode DAX attribute flag. If dax=inode is in use on both sides, the the server honours the requests of the client to set/clear the inode flags and presents the inode flag according to the state the client side has requested.
This policy state probably should be communicated to
the fuse client from the server at mount time so policy conflicts can be be resolved at mount time (e.g. reject mount if policy conflict occurs, default to guest overrides server or vice versa, etc). This then means that that the client side mount policies will default to server side policy when they set "dax=inode" but also provide a local override for always or never local behaviour.
Hence, AFAICT, there is no need for a 'dax=server' option - this seems to be exactly what 'dax=inode' behaviour means on the client side - it behaves according to how the server side propagates the DAX attribute to the client for each inode.
Quoted 3 lines
Otherwise it will be very confusing. People familiar with "-o dax=inode" on ext4/xfs will expect file attr to work and that's not what we are implementing, IIUC.
The dax mount option behaviour is already confusing enough without adding yet another weird, poorly documented, easily misunderstood mode that behaves subtly different to existing modes.
Please try to make the virtiofs behaviour compatible with existing modes - it's not that hard to make the client dax=inode behaviour be controlled by the server side without any special client side mount modes.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Vivek Goyal
maintainerRe: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 17 lines · Dave Chinner
Quoted 9 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
So will "-o dax=inode" be used for per file DAX where dax mode comes from server?
Quoted 6 lines
I think we discussed this. It will be better to leave "-o dax=inode" alone. It should be used when we are reading dax status from file attr (like ext4 and xfs).
And probably create separate option say "-o dax=server" where server specifies which inode should use dax.
Quoted 7 lines
That seems like a poor idea to me.
The server side already controls what the client side does by controlling the inode attributes that the client side sees. That is, if the server is going to specify whether the client side data access is going to use dax, then the server presents the client with an inode that has the DAX attribute flag set on it.
Hi Dave,
Currently in fuse/virtiofs, DAX is compltely controlled by client. Server has no say in it. If client is mounted with "-o dax", dax is enabled on all files otherwise dax is disabled on all files. One could think of implementing an option on server so that server could deny mmap() requests that come from client, but so far nobody asked for such an option on server side.
When you say "inode that has DAX attribute flag set on it", are you referring to "S_DAX (in inode->i_flags)" or persistent attr "FS_XFLAG_DAX"?
As of now S_DAX on client side inode is set by fuse client whenever client mounted filesystem with "-o dax". And I think you are assuming that DAX attribute of inode is already coming from server. That's not the case. In fact that seems to be the proposal. Provide capability so that server can specify which inode should be using DAX and which inode should not be.
Quoted 4 lines
In that case, turning off dax on the guest side should be communicated to the fuse server so the server turns off the DAX flag on the server side iff server side policy allows it.
Not sure what do you mean by server turns of DAX flag based on client turning off DAX. Server does not have to do anything. If client decides to not use DAX (in guest), it will not send FUSE_SETUPMAPPING requests to server and that's it.
Quoted 4 lines
When the guest
then purges it's local inode and reloads it from the server then it will get an inode with the DAX flag set according to server side policy.
So right now we don't have a mechanism for server to specify DAX flag. And that's what this patch series is trying to do.
Quoted 5 lines
Hence if the server side is configured with "dax=always" or dax="never" semantics it means the client side inode flag state cannot control DAX mode. That means, regardless of the client side mount options, DAX is operating under always or never policy,
Hmm..., When you say "server side is configured with "dax=always", do you mean shared directory on host is mounted with "-o dax=always", or you mean some virtiofs server specific option which enables dax on all inodes from server side.
In general, DAX on host and DAX inside guest are completely independent. Host filesystem could very well be mounted with dax or without dax and that has no affect on guests's capability to be able to enable DAX or not.
Quoted 14 lines
enforced by the server side by direct control of the client inode DAX attribute flag. If dax=inode is in use on both sides, the the server honours the requests of the client to set/clear the inode flags and presents the inode flag according to the state the client side has requested.
This policy state probably should be communicated to
the fuse client from the server at mount time so policy conflicts can be be resolved at mount time (e.g. reject mount if policy conflict occurs, default to guest overrides server or vice versa, etc). This then means that that the client side mount policies will default to server side policy when they set "dax=inode" but also provide a local override for always or never local behaviour.
Quoted 4 lines
Hence, AFAICT, there is no need for a 'dax=server' option - this seems to be exactly what 'dax=inode' behaviour means on the client side - it behaves according to how the server side propagates the DAX attribute to the client for each inode.
Ok. So "-o dax=inode" in fuse will have a different meaning as opposed to ext4/xfs. This will mean that server will pass DAX state of inode when inode is instantiated and client should honor that.
But what about FS_XFLAG_DAX flag then. Say host file system does support this att and fuse/virtiofs allows querying and setting this attribute (I don't think it is allowed now). So will we not create a future conflict where in case of fuse/virtiofs "-o dax=inode" means something different and it does look at FS_XFLAG_DAX file attr.
Quoted 5 lines
Quoted 4 lines
Otherwise it will be very confusing. People familiar with "-o dax=inode" on ext4/xfs will expect file attr to work and that's not what we are implementing, IIUC.
Quoted 8 lines
The dax mount option behaviour is already confusing enough without adding yet another weird, poorly documented, easily misunderstood mode that behaves subtly different to existing modes.
Please try to make the virtiofs behaviour compatible with existing modes - it's not that hard to make the client dax=inode behaviour be controlled by the server side without any special client side mount modes.
Given I want to keep the option of similar behavior for "dax=inode" across ext4/xfs and virtiofs, I suggested "dax=server". Because I assumed that "dax=inode" means that dax is per inode property AND this per inode property is specified by persistent file attr FS_XFLAG_DAX.
But fuse/virtiofs will not be specifying dax property of inode using FS_XFLAG_DAX (atleast as of now). And server will set DAX property using some bit in protocol.
So these seem little different. If we use "dax=inode" for server specifying DAX property of inode, then in future if client can query/set FS_XFLAG_DAX on inode, it will be a problem. There will be a conflict.
Use case I was imagining was, say on host, user might set FS_XFLAG_DAX attr on relevant files and then mount virtiofs in guest with "-o dax=inode". Guest will query state of FS_XFLAG_DAX on inode and enable DAX accordingly. (And server is not necessarily playing an active role in determining which files should use DAX).
In summary, there seem to be two use cases.
A. virtiofsd/fuse-server wants do be able to come up with its own policy
to decide which inodes should use guest.
B. guest client decides which inode use DAX based on FS_XFLAG_DAX attr
on inode (and server does not play a role).
To be able to different between these two cases, I was suggesting using "-o dax=inode" for B and "-o dax=server" for A.
Thanks
Vivek
Re: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 75 lines · Vivek Goyal
Quoted 17 lines · Dave Chinner
Quoted 9 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
So will "-o dax=inode" be used for per file DAX where dax mode comes from server?
Quoted 6 lines
I think we discussed this. It will be better to leave "-o dax=inode" alone. It should be used when we are reading dax status from file attr (like ext4 and xfs).
And probably create separate option say "-o dax=server" where server specifies which inode should use dax.
Quoted 7 lines
That seems like a poor idea to me.
The server side already controls what the client side does by controlling the inode attributes that the client side sees. That is, if the server is going to specify whether the client side data access is going to use dax, then the server presents the client with an inode that has the DAX attribute flag set on it.
Hi Dave,
Currently in fuse/virtiofs, DAX is compltely controlled by client. Server has no say in it. If client is mounted with "-o dax", dax is enabled on all files otherwise dax is disabled on all files. One could think of implementing an option on server so that server could deny
mmap()requests that come from client, but so far nobody asked for such an option on server side.When you say "inode that has DAX attribute flag set on it", are you referring to "
S_DAX(ininode->i_flags)" or persistent attr "FS_XFLAG_DAX"?As of now
S_DAXon client side inode is set by fuse client whenever client mounted filesystem with "-o dax". And I think you are assuming that DAX attribute of inode is already coming from server. That's not the case. In fact that seems to be the proposal. Provide capability so that server can specify which inode should be using DAX and which inode should not be.Quoted 4 lines
In that case, turning off dax on the guest side should be communicated to the fuse server so the server turns off the DAX flag on the server side iff server side policy allows it.
Not sure what do you mean by server turns of DAX flag based on client turning off DAX. Server does not have to do anything. If client decides to not use DAX (in guest), it will not send
FUSE_SETUPMAPPINGrequests to server and that's it.Quoted 4 lines
When the guest
then purges it's local inode and reloads it from the server then it will get an inode with the DAX flag set according to server side policy.So right now we don't have a mechanism for server to specify DAX flag. And that's what this patch series is trying to do.
Quoted 5 lines
Hence if the server side is configured with "dax=always" or dax="never" semantics it means the client side inode flag state cannot control DAX mode. That means, regardless of the client side mount options, DAX is operating under always or never policy,
Hmm..., When you say "server side is configured with "dax=always", do you mean shared directory on host is mounted with "-o dax=always", or you mean some virtiofs server specific option which enables dax on all inodes from server side.
Quoted 4 lines
In general, DAX on host and DAX inside guest are completely independent. Host filesystem could very well be mounted with dax or without dax and that has no affect on guests's capability to be able to enable DAX or not.
Hi Dave, I think you are referring to "shared directory on host is mounted with "-o dax=always"" when you are saying "server side is configured with "dax=always". And just as Vivek said, there's no necessary relationship between the DAX mode in host and that in guest, technically.
Quoted 55 lines
Quoted 15 lines
enforced by the server side by direct control of the client inode DAX attribute flag. If dax=inode is in use on both sides, the the server honours the requests of the client to set/clear the inode flags and presents the inode flag according to the state the client side has requested.
This policy state probably should be communicated to
the fuse client from the server at mount time so policy conflicts can be be resolved at mount time (e.g. reject mount if policy conflict occurs, default to guest overrides server or vice versa, etc). This then means that that the client side mount policies will default to server side policy when they set "dax=inode" but also provide a local override for always or never local behaviour.Quoted 4 lines
Hence, AFAICT, there is no need for a 'dax=server' option - this seems to be exactly what 'dax=inode' behaviour means on the client side - it behaves according to how the server side propagates the DAX attribute to the client for each inode.
Ok. So "-o dax=inode" in fuse will have a different meaning as opposed to ext4/xfs. This will mean that server will pass DAX state of inode when inode is instantiated and client should honor that.
But what about
FS_XFLAG_DAXflag then. Say host file system does support this att and fuse/virtiofs allows querying and setting this attribute (I don't think it is allowed now). So will we not create a future conflict where in case of fuse/virtiofs "-o dax=inode" means something different and it does look atFS_XFLAG_DAXfile attr.Quoted 5 lines
Quoted 4 lines
Otherwise it will be very confusing. People familiar with "-o dax=inode" on ext4/xfs will expect file attr to work and that's not what we are implementing, IIUC.
Quoted 8 lines
The dax mount option behaviour is already confusing enough without adding yet another weird, poorly documented, easily misunderstood mode that behaves subtly different to existing modes.
Please try to make the virtiofs behaviour compatible with existing modes - it's not that hard to make the client dax=inode behaviour be controlled by the server side without any special client side mount modes.
Given I want to keep the option of similar behavior for "dax=inode" across ext4/xfs and virtiofs, I suggested "dax=server". Because I assumed that "dax=inode" means that dax is per inode property AND this per inode property is specified by persistent file attr
FS_XFLAG_DAX.But fuse/virtiofs will not be specifying dax property of inode using
FS_XFLAG_DAX(atleast as of now). And server will set DAX property using some bit in protocol.
Quoted 8 lines
So these seem little different. If we use "dax=inode" for server specifying DAX property of inode, then in future if client can query/set
FS_XFLAG_DAXon inode, it will be a problem. There will be a conflict.Use case I was imagining was, say on host, user might set
FS_XFLAG_DAXattr on relevant files and then mount virtiofs in guest with "-o dax=inode". ...
Yes this will be a real using case, where users could specify which files should be DAX enabled by setting FS_XFLAG_DAX attr **on host**. In this case, the FS_XFLAG_DAX attr could still be conveyed by FUSE_ATTR_DAX (introduced in this patch set) in FUSE_LOOKUP reply. Then extra option may be added to fuse daemon, e.g., '-o policy=server' for getting DAX attr according to fuse daemon's own policy, and '-o policy=flag' for querying persistent FS_XFLAG_DAX attr of the host inode.
And thus 'dax=server' mount option inside guest could be omitted and replaced by 'policy=server' option on the fuse daemon side. In this case, the fuse kernel module could be as simple as possible, while all other strategies could be implemented on the fuse daemon side.
Quoted 3 lines
... Guest will query state of
FS_XFLAG_DAXon inode
and enable DAX accordingly. (And server is not necessarily playing an active role in determining which files should use DAX).
It will be less efficient for guest to initiate another FUSE_IOCTL request to query if FS_XFLAG_DAX attr is on the inode. I think FUSE_ATTR_DAX flag in FUSE_LOOKUP reply shall be adequate for conveying DAX related attr.
However it is indeed an issue when it comes to the consistency of FS_XFLAG_DAX flag with ext4/xfs. There are following two semantics when using per-file DAX for ext4/xfs:
1. user sets FS_XFLAG_DAX attribute on inode, for which per-file DAX shall be enabled, and the attribute will be stored persistently on the inode;
2. user can query the FS_XFLAG_DAX attribute to see if DAX is enabled for specific file, when it's in 'dax=inode' mode.
For semantic 1), I'm quite doubted if there's necessity and possibility of this using case for fuse so far, given admin could already specify which files should be DAX enabled on the host side. If this semantic indeed shall be implemented later, then I'm afraid 'policy=server' option shall always be specified with fuse daemon, that is, fuse daemon shall always query the FS_XFLAG_DAX attr of the inode on the host.
For semantic 2), could we return a fake FS_XFLAG_DAX once the server shows that this file should be DAX enabled?
Quoted 11 lines
In summary, there seem to be two use cases.
A. virtiofsd/fuse-server wants do be able to come up with its own policy
to decide which inodes should use guest.B. guest client decides which inode use DAX based on
FS_XFLAG_DAXattr
on inode (and server does not play a role).To be able to different between these two cases, I was suggesting using "-o dax=inode" for B and "-o dax=server" for A.
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerRe: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 168 lines · JeffleXu
Quoted 75 lines · Vivek Goyal
Quoted 17 lines · Dave Chinner
Quoted 9 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
So will "-o dax=inode" be used for per file DAX where dax mode comes from server?
Quoted 6 lines
I think we discussed this. It will be better to leave "-o dax=inode" alone. It should be used when we are reading dax status from file attr (like ext4 and xfs).
And probably create separate option say "-o dax=server" where server specifies which inode should use dax.
Quoted 7 lines
That seems like a poor idea to me.
The server side already controls what the client side does by controlling the inode attributes that the client side sees. That is, if the server is going to specify whether the client side data access is going to use dax, then the server presents the client with an inode that has the DAX attribute flag set on it.
Hi Dave,
Currently in fuse/virtiofs, DAX is compltely controlled by client. Server has no say in it. If client is mounted with "-o dax", dax is enabled on all files otherwise dax is disabled on all files. One could think of implementing an option on server so that server could deny
mmap()requests that come from client, but so far nobody asked for such an option on server side.When you say "inode that has DAX attribute flag set on it", are you referring to "
S_DAX(ininode->i_flags)" or persistent attr "FS_XFLAG_DAX"?As of now
S_DAXon client side inode is set by fuse client whenever client mounted filesystem with "-o dax". And I think you are assuming that DAX attribute of inode is already coming from server. That's not the case. In fact that seems to be the proposal. Provide capability so that server can specify which inode should be using DAX and which inode should not be.Quoted 4 lines
In that case, turning off dax on the guest side should be communicated to the fuse server so the server turns off the DAX flag on the server side iff server side policy allows it.
Not sure what do you mean by server turns of DAX flag based on client turning off DAX. Server does not have to do anything. If client decides to not use DAX (in guest), it will not send
FUSE_SETUPMAPPINGrequests to server and that's it.Quoted 4 lines
When the guest
then purges it's local inode and reloads it from the server then it will get an inode with the DAX flag set according to server side policy.So right now we don't have a mechanism for server to specify DAX flag. And that's what this patch series is trying to do.
Quoted 5 lines
Hence if the server side is configured with "dax=always" or dax="never" semantics it means the client side inode flag state cannot control DAX mode. That means, regardless of the client side mount options, DAX is operating under always or never policy,
Hmm..., When you say "server side is configured with "dax=always", do you mean shared directory on host is mounted with "-o dax=always", or you mean some virtiofs server specific option which enables dax on all inodes from server side.
Quoted 4 lines
In general, DAX on host and DAX inside guest are completely independent. Host filesystem could very well be mounted with dax or without dax and that has no affect on guests's capability to be able to enable DAX or not.
Hi Dave, I think you are referring to "shared directory on host is mounted with "-o dax=always"" when you are saying "server side is configured with "dax=always". And just as Vivek said, there's no necessary relationship between the DAX mode in host and that in guest, technically.
Quoted 55 lines
Quoted 15 lines
enforced by the server side by direct control of the client inode DAX attribute flag. If dax=inode is in use on both sides, the the server honours the requests of the client to set/clear the inode flags and presents the inode flag according to the state the client side has requested.
This policy state probably should be communicated to
the fuse client from the server at mount time so policy conflicts can be be resolved at mount time (e.g. reject mount if policy conflict occurs, default to guest overrides server or vice versa, etc). This then means that that the client side mount policies will default to server side policy when they set "dax=inode" but also provide a local override for always or never local behaviour.Quoted 4 lines
Hence, AFAICT, there is no need for a 'dax=server' option - this seems to be exactly what 'dax=inode' behaviour means on the client side - it behaves according to how the server side propagates the DAX attribute to the client for each inode.
Ok. So "-o dax=inode" in fuse will have a different meaning as opposed to ext4/xfs. This will mean that server will pass DAX state of inode when inode is instantiated and client should honor that.
But what about
FS_XFLAG_DAXflag then. Say host file system does support this att and fuse/virtiofs allows querying and setting this attribute (I don't think it is allowed now). So will we not create a future conflict where in case of fuse/virtiofs "-o dax=inode" means something different and it does look atFS_XFLAG_DAXfile attr.Quoted 5 lines
Quoted 4 lines
Otherwise it will be very confusing. People familiar with "-o dax=inode" on ext4/xfs will expect file attr to work and that's not what we are implementing, IIUC.
Quoted 8 lines
The dax mount option behaviour is already confusing enough without adding yet another weird, poorly documented, easily misunderstood mode that behaves subtly different to existing modes.
Please try to make the virtiofs behaviour compatible with existing modes - it's not that hard to make the client dax=inode behaviour be controlled by the server side without any special client side mount modes.
Given I want to keep the option of similar behavior for "dax=inode" across ext4/xfs and virtiofs, I suggested "dax=server". Because I assumed that "dax=inode" means that dax is per inode property AND this per inode property is specified by persistent file attr
FS_XFLAG_DAX.But fuse/virtiofs will not be specifying dax property of inode using
FS_XFLAG_DAX(atleast as of now). And server will set DAX property using some bit in protocol.Quoted 8 lines
So these seem little different. If we use "dax=inode" for server specifying DAX property of inode, then in future if client can query/set
FS_XFLAG_DAXon inode, it will be a problem. There will be a conflict.Use case I was imagining was, say on host, user might set
FS_XFLAG_DAXattr on relevant files and then mount virtiofs in guest with "-o dax=inode". ...Yes this will be a real using case, where users could specify which files should be DAX enabled by setting
FS_XFLAG_DAXattr **on host**. In this case, theFS_XFLAG_DAXattr could still be conveyed byFUSE_ATTR_DAX(introduced in this patch set) inFUSE_LOOKUPreply. Then extra option may be added to fuse daemon, e.g., '-o policy=server' for getting DAX attr according to fuse daemon's own policy, and '-o policy=flag' for querying persistentFS_XFLAG_DAXattr of the host inode.And thus 'dax=server' mount option inside guest could be omitted and replaced by 'policy=server' option on the fuse daemon side. In this case, the fuse kernel module could be as simple as possible, while all other strategies could be implemented on the fuse daemon side.
Quoted 8 lines
Quoted 3 lines
... Guest will query state of
FS_XFLAG_DAXon inode
and enable DAX accordingly. (And server is not necessarily playing an active role in determining which files should use DAX).It will be less efficient for guest to initiate another
FUSE_IOCTLrequest to query ifFS_XFLAG_DAXattr is on the inode. I thinkFUSE_ATTR_DAXflag inFUSE_LOOKUPreply shall be adequate for conveying DAX related attr.
This is a good point. In case of fuse, for querying FS_XFLAG_DAX, there will be extra round trip for FUSE_IOCTL message. It will be better if server could have an option which enables checking FS_XFLAG_DAX and respond with FUSE_ATTR_DAX flag in FUSE_LOOKUP message.
Ok, so for the case of fuse, "-o dax=inode" can just mean that enablement of DAX is per inode. But which files should use DAX is determined by server. And client will not have a say in this.
And server could have dax options (as you mentioned) to determine whether it should query FS_XFLAG_DAX to determine dax status of file or it could implement its own policy (say size based) to determine which inodes should use DAX. We probably will have document this little deviation in behavior in Documentation/filesystem/dax.rst
I guess this sounds reasonable. There is little deviation from ext4/xfs but they are local filesystems to exact match might not make lot of sense in fuse context.
Quoted 9 lines
However it is indeed an issue when it comes to the consistency of
FS_XFLAG_DAXflag with ext4/xfs. There are following two semantics when using per-file DAX for ext4/xfs:1. user sets
FS_XFLAG_DAXattribute on inode, for which per-file DAX shall be enabled, and the attribute will be stored persistently on the inode;
So this one we can still support as long as fuse supports ioctl and query and setting FS_XFLAG_DAX, right?
Quoted 2 lines
2. user can query the
FS_XFLAG_DAXattribute to see if DAX is enabled for specific file, when it's in 'dax=inode' mode.
Documentation/filesystems/dax.rst says.
1. There exists an in-kernel file access mode flag S_DAX that corresponds to the statx flag STATX_ATTR_DAX. See the manpage for statx(2) for details about this access mode.
So my understanding is that if user wants to know if DAX is currently being used on a file, they should call statx() and check for STATX_ATTR_DAX flag instead? And this will be set based on S_DAX status of inode.
If this is correct, then it is not an issue. We should be able to return STATX_ATTR_DAX based on S_DAX and user should be able to figure out which files are using DAX. Just that it will not necessarily match FS_XFLAG_DAX because server might be configured to use its own custom policy to determine DAX status of a file/inode.
Quoted 11 lines
For semantic 1), I'm quite doubted if there's necessity and possibility of this using case for fuse so far, given admin could already specify which files should be DAX enabled on the host side. If this semantic indeed shall be implemented later, then I'm afraid 'policy=server' option shall always be specified with fuse daemon, that is, fuse daemon shall always query the
FS_XFLAG_DAXattr of the inode on the host.For semantic 2), could we return a fake
FS_XFLAG_DAXonce the server shows that this file should be DAX enabled?
I guess we don't have to return fake FS_XFLAG_DAX at all. If user space is calling statx(), we should just set STATX_ATTR_DAX if file/inode is using DAX.
Thanks
Vivek
Quoted 18 lines
Quoted 12 lines
In summary, there seem to be two use cases.
A. virtiofsd/fuse-server wants do be able to come up with its own policy
to decide which inodes should use guest.B. guest client decides which inode use DAX based on
FS_XFLAG_DAXattr
on inode (and server does not play a role).To be able to different between these two cases, I was suggesting using "-o dax=inode" for B and "-o dax=server" for A.
Signature · 2 lines
Thanks, Jeffle
Re: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 187 lines · Vivek Goyal
Quoted 168 lines · JeffleXu
Quoted 75 lines · Vivek Goyal
Quoted 17 lines · Dave Chinner
Quoted 9 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
So will "-o dax=inode" be used for per file DAX where dax mode comes from server?
Quoted 6 lines
I think we discussed this. It will be better to leave "-o dax=inode" alone. It should be used when we are reading dax status from file attr (like ext4 and xfs).
And probably create separate option say "-o dax=server" where server specifies which inode should use dax.
Quoted 7 lines
That seems like a poor idea to me.
The server side already controls what the client side does by controlling the inode attributes that the client side sees. That is, if the server is going to specify whether the client side data access is going to use dax, then the server presents the client with an inode that has the DAX attribute flag set on it.
Hi Dave,
Currently in fuse/virtiofs, DAX is compltely controlled by client. Server has no say in it. If client is mounted with "-o dax", dax is enabled on all files otherwise dax is disabled on all files. One could think of implementing an option on server so that server could deny
mmap()requests that come from client, but so far nobody asked for such an option on server side.When you say "inode that has DAX attribute flag set on it", are you referring to "
S_DAX(ininode->i_flags)" or persistent attr "FS_XFLAG_DAX"?As of now
S_DAXon client side inode is set by fuse client whenever client mounted filesystem with "-o dax". And I think you are assuming that DAX attribute of inode is already coming from server. That's not the case. In fact that seems to be the proposal. Provide capability so that server can specify which inode should be using DAX and which inode should not be.Quoted 4 lines
In that case, turning off dax on the guest side should be communicated to the fuse server so the server turns off the DAX flag on the server side iff server side policy allows it.
Not sure what do you mean by server turns of DAX flag based on client turning off DAX. Server does not have to do anything. If client decides to not use DAX (in guest), it will not send
FUSE_SETUPMAPPINGrequests to server and that's it.Quoted 4 lines
When the guest
then purges it's local inode and reloads it from the server then it will get an inode with the DAX flag set according to server side policy.So right now we don't have a mechanism for server to specify DAX flag. And that's what this patch series is trying to do.
Quoted 5 lines
Hence if the server side is configured with "dax=always" or dax="never" semantics it means the client side inode flag state cannot control DAX mode. That means, regardless of the client side mount options, DAX is operating under always or never policy,
Hmm..., When you say "server side is configured with "dax=always", do you mean shared directory on host is mounted with "-o dax=always", or you mean some virtiofs server specific option which enables dax on all inodes from server side.
Quoted 4 lines
In general, DAX on host and DAX inside guest are completely independent. Host filesystem could very well be mounted with dax or without dax and that has no affect on guests's capability to be able to enable DAX or not.
Hi Dave, I think you are referring to "shared directory on host is mounted with "-o dax=always"" when you are saying "server side is configured with "dax=always". And just as Vivek said, there's no necessary relationship between the DAX mode in host and that in guest, technically.
Quoted 55 lines
Quoted 15 lines
enforced by the server side by direct control of the client inode DAX attribute flag. If dax=inode is in use on both sides, the the server honours the requests of the client to set/clear the inode flags and presents the inode flag according to the state the client side has requested.
This policy state probably should be communicated to
the fuse client from the server at mount time so policy conflicts can be be resolved at mount time (e.g. reject mount if policy conflict occurs, default to guest overrides server or vice versa, etc). This then means that that the client side mount policies will default to server side policy when they set "dax=inode" but also provide a local override for always or never local behaviour.Quoted 4 lines
Hence, AFAICT, there is no need for a 'dax=server' option - this seems to be exactly what 'dax=inode' behaviour means on the client side - it behaves according to how the server side propagates the DAX attribute to the client for each inode.
Ok. So "-o dax=inode" in fuse will have a different meaning as opposed to ext4/xfs. This will mean that server will pass DAX state of inode when inode is instantiated and client should honor that.
But what about
FS_XFLAG_DAXflag then. Say host file system does support this att and fuse/virtiofs allows querying and setting this attribute (I don't think it is allowed now). So will we not create a future conflict where in case of fuse/virtiofs "-o dax=inode" means something different and it does look atFS_XFLAG_DAXfile attr.Quoted 5 lines
Quoted 4 lines
Otherwise it will be very confusing. People familiar with "-o dax=inode" on ext4/xfs will expect file attr to work and that's not what we are implementing, IIUC.
Quoted 8 lines
The dax mount option behaviour is already confusing enough without adding yet another weird, poorly documented, easily misunderstood mode that behaves subtly different to existing modes.
Please try to make the virtiofs behaviour compatible with existing modes - it's not that hard to make the client dax=inode behaviour be controlled by the server side without any special client side mount modes.
Given I want to keep the option of similar behavior for "dax=inode" across ext4/xfs and virtiofs, I suggested "dax=server". Because I assumed that "dax=inode" means that dax is per inode property AND this per inode property is specified by persistent file attr
FS_XFLAG_DAX.But fuse/virtiofs will not be specifying dax property of inode using
FS_XFLAG_DAX(atleast as of now). And server will set DAX property using some bit in protocol.Quoted 8 lines
So these seem little different. If we use "dax=inode" for server specifying DAX property of inode, then in future if client can query/set
FS_XFLAG_DAXon inode, it will be a problem. There will be a conflict.Use case I was imagining was, say on host, user might set
FS_XFLAG_DAXattr on relevant files and then mount virtiofs in guest with "-o dax=inode". ...Yes this will be a real using case, where users could specify which files should be DAX enabled by setting
FS_XFLAG_DAXattr **on host**. In this case, theFS_XFLAG_DAXattr could still be conveyed byFUSE_ATTR_DAX(introduced in this patch set) inFUSE_LOOKUPreply. Then extra option may be added to fuse daemon, e.g., '-o policy=server' for getting DAX attr according to fuse daemon's own policy, and '-o policy=flag' for querying persistentFS_XFLAG_DAXattr of the host inode.And thus 'dax=server' mount option inside guest could be omitted and replaced by 'policy=server' option on the fuse daemon side. In this case, the fuse kernel module could be as simple as possible, while all other strategies could be implemented on the fuse daemon side.
Quoted 8 lines
Quoted 3 lines
... Guest will query state of
FS_XFLAG_DAXon inode
and enable DAX accordingly. (And server is not necessarily playing an active role in determining which files should use DAX).It will be less efficient for guest to initiate another
FUSE_IOCTLrequest to query ifFS_XFLAG_DAXattr is on the inode. I thinkFUSE_ATTR_DAXflag inFUSE_LOOKUPreply shall be adequate for conveying DAX related attr.This is a good point. In case of fuse, for querying
FS_XFLAG_DAX, there will be extra round trip forFUSE_IOCTLmessage. It will be better if server could have an option which enables checkingFS_XFLAG_DAXand respond withFUSE_ATTR_DAXflag inFUSE_LOOKUPmessage.Ok, so for the case of fuse, "-o dax=inode" can just mean that enablement of DAX is per inode. But which files should use DAX is determined by server. And client will not have a say in this.
Quoted 5 lines
And server could have dax options (as you mentioned) to determine whether it should query
FS_XFLAG_DAXto determine dax status of file or it could implement its own policy (say size based) to determine which inodes should use DAX. We probably will have document this little deviation in behavior inDocumentation/filesystem/dax.rst
Agreed.
Quoted 10 lines
I guess this sounds reasonable. There is little deviation from ext4/xfs but they are local filesystems to exact match might not make lot of sense in fuse context.
Quoted 6 lines
However it is indeed an issue when it comes to the consistency of
FS_XFLAG_DAXflag with ext4/xfs. There are following two semantics when using per-file DAX for ext4/xfs:
Quoted 6 lines
Quoted 3 lines
1. user sets
FS_XFLAG_DAXattribute on inode, for which per-file DAX shall be enabled, and the attribute will be stored persistently on the inode;So this one we can still support as long as fuse supports ioctl and query and setting
FS_XFLAG_DAX, right?
Yes, if it is really needed later.
Quoted 15 lines
Quoted 3 lines
2. user can query the
FS_XFLAG_DAXattribute to see if DAX is enabled for specific file, when it's in 'dax=inode' mode.
Documentation/filesystems/dax.rstsays.1. There exists an in-kernel file access mode flag
S_DAXthat corresponds to the statx flagSTATX_ATTR_DAX. See the manpage for statx(2) for details about this access mode.So my understanding is that if user wants to know if DAX is currently being used on a file, they should call
statx()and check forSTATX_ATTR_DAXflag instead? And this will be set based onS_DAXstatus of inode.
Quoted 5 lines
If this is correct, then it is not an issue. We should be able to return
STATX_ATTR_DAXbased onS_DAXand user should be able to figure out which files are using DAX. Just that it will not necessarily matchFS_XFLAG_DAXbecause server might be configured to use its own custom policy to determine DAX status of a file/inode.
Good catch! You are right. Then semantic 2) can be matched without any modification.
Quoted 10 lines
Quoted 10 lines
For semantic 1), I'm quite doubted if there's necessity and possibility of this using case for fuse so far, given admin could already specify which files should be DAX enabled on the host side. If this semantic indeed shall be implemented later, then I'm afraid 'policy=server' option shall always be specified with fuse daemon, that is, fuse daemon shall always query the
FS_XFLAG_DAXattr of the inode on the host.
Quoted 7 lines
Quoted 2 lines
For semantic 2), could we return a fake
FS_XFLAG_DAXonce the server shows that this file should be DAX enabled?I guess we don't have to return fake
FS_XFLAG_DAXat all. If user space is callingstatx(), we should just setSTATX_ATTR_DAXif file/inode is using DAX.
Yes.
Thanks! I will send a new version later based on these discussions.
Signature · 2 lines
Thanks, Jeffle
Re: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
In reply to Vivek GoyalQuoted 26 lines · Vivek Goyal
Quoted 17 lines · Dave Chinner
Quoted 9 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
So will "-o dax=inode" be used for per file DAX where dax mode comes from server?
Quoted 6 lines
I think we discussed this. It will be better to leave "-o dax=inode" alone. It should be used when we are reading dax status from file attr (like ext4 and xfs).
And probably create separate option say "-o dax=server" where server specifies which inode should use dax.
Quoted 7 lines
That seems like a poor idea to me.
The server side already controls what the client side does by controlling the inode attributes that the client side sees. That is, if the server is going to specify whether the client side data access is going to use dax, then the server presents the client with an inode that has the DAX attribute flag set on it.
Quoted 8 lines
Hi Dave,
Currently in fuse/virtiofs, DAX is compltely controlled by client. Server has no say in it. If client is mounted with "-o dax", dax is enabled on all files otherwise dax is disabled on all files. One could think of implementing an option on server so that server could deny
mmap()requests that come from client, but so far nobody asked for such an option on server side.
Can you please refer to the new options "always|never|inode" in disucssions, because the "-o dax" option is deprecated and we really need to center discussions around the new options, not the deprecated one.
Quoted 3 lines
When you say "inode that has DAX attribute flag set on it", are you referring to "
S_DAX(ininode->i_flags)" or persistent attr "FS_XFLAG_DAX"?
Neither. The S_DAX flags is the VFS struct inode state that tells the kernel what to do with that inode. The FS_XFLAG_DAX is the userspace API status/control flag that represents the current state of the persistent per-inode DAX control attribute.
What I'm talking about the persistent attribute that is present on stable storage. e.g. in the XFS on-disk inode the flag is XFS_DIFLAG2_DAX which is never seen outside low level XFS code. In the case of FUSE, this is the persistent server-side DAX attribute state. The client gets this information from the server via an inode attribute defined in the FUSE protocol. The client translates that network protocol attribute to S_DAX status on FS inode instantiation, and to/from FS_XFLAG_DAX for user control under dax=inode.
In the case that the user changes FS_XFLAG_DAX, the FUSE client needs to communicate that attribute change to the server, where the server then changes the persistent state of the on-disk inode so that the next time the client requests that inode, it gets the state it previously set. Unless, of course, there are server side policy overrides (never/always).
IOWs, dax=inode on the client side is essentially "follow server side policy" because the server maintains the persistent DAX flag state in the server side inode...
Quoted 2 lines
As of now
S_DAXon client side inode is set by fuse client whenever client mounted filesystem with "-o dax". And I think you are assuming
Of course. Similarly, if the client uses dax=never, the S_DAX is never set on the VFS inode. But if dax=inode is set, where does the DAX attribute state that determines S_DAX comes from .....?
Quoted 4 lines
that DAX attribute of inode is already coming from server. That's not the case. In fact that seems to be the proposal. Provide capability so that server can specify which inode should be using DAX and which inode should not be.
Yup, that's exactly what I'm talking about - this is "dax=inode" behaviour, because the client follows what the server tells it in the inode attributes that arrive over the wire.
Quoted 8 lines
Quoted 3 lines
In that case, turning off dax on the guest side should be communicated to the fuse server so the server turns off the DAX flag on the server side iff server side policy allows it.
Not sure what do you mean by server turns of DAX flag based on client turning off DAX. Server does not have to do anything. If client decides to not use DAX (in guest), it will not send
FUSE_SETUPMAPPINGrequests to server and that's it.
Where does the client get it's per-inode DAX policy from if dax=inode is, like other DAX filesystems, the default behaviour?
Where is the persistent storage of that per-inode attribute kept?
Quoted 14 lines
Quoted 4 lines
When the guest
then purges it's local inode and reloads it from the server then it will get an inode with the DAX flag set according to server side policy.So right now we don't have a mechanism for server to specify DAX flag. And that's what this patch series is trying to do.
Quoted 5 lines
Hence if the server side is configured with "dax=always" or dax="never" semantics it means the client side inode flag state cannot control DAX mode. That means, regardless of the client side mount options, DAX is operating under always or never policy,
Quoted 4 lines
Hmm..., When you say "server side is configured with "dax=always", do you mean shared directory on host is mounted with "-o dax=always", or you mean some virtiofs server specific option which enables dax on all inodes from server side.
I don't care. That's a server side implementation detail. You can keep it in a private xattr for all I care.
Quoted 4 lines
In general, DAX on host and DAX inside guest are completely independent. Host filesystem could very well be mounted with dax or without dax and that has no affect on guests's capability to be able to enable DAX or not.
If you have both the host and guest accessing the same files with different modes, then you have a data coherency problem that is guaranteed to corrupt data.
Quoted 7 lines
Quoted 3 lines
seems to be exactly what 'dax=inode' behaviour means on the client side - it behaves according to how the server side propagates the DAX attribute to the client for each inode.
Ok. So "-o dax=inode" in fuse will have a different meaning as opposed to ext4/xfs. This will mean that server will pass DAX state of inode when inode is instantiated and client should honor that.
No, that's exactly what dax=inode means: DAX behaviour is per-inode state that users must probe via statx() to determine if dax is active or not.
Quoted 3 lines
But what about
FS_XFLAG_DAXflag then. Say host file system does support this att and fuse/virtiofs allows querying and setting this attribute (I don't think it is allowed now).
FS_XFLAG_DAX is the ioctl-based control API for the client side. It's not a persistent flag.
Quoted 3 lines
So will we not create a future conflict where in case of fuse/virtiofs "-o dax=inode" means something different and it does look at
FS_XFLAG_DAXfile attr.
Like many, I suspect you might be mis-understanding the DAX API.
There are 4 parts to it:
- persistent filesystem inode state flag
- ioctls to manipulate persistent inode state flag (FS_XFLAG_DAX)
- mount options to override persistent inode state flag
- VFS inode state indicating DAX is active (S_DAX, STATX_ATTR_DAX)
You seem to be conflating the user API FS_XFLAG_DAX flag with whatever the filesystem uses to physically storage that state. They are not the same thing. FS_XFLAG_DAX also has no correlation with the mount options - we can change the on-disk flag state regardless of the mount option in use. We can even change the on-disk state flag on *kernels that don't support DAX*. Changing the on-disk attribute *does not change S_DAX*.
That's because the on-disk persistent state is a property of the filesystem's on-disk format, not a property of the kernel that is running on that machine. The persistent state flag is managed as a completely independent filesystem inode attribute, but it only affects behaviour when the dax=inode mode has been selected.
Control of the behaviour by this persistent inode flag is what dax=inode means. It does not define how that attribute flag is managed, it just defines the policy that an the inode's behaviour will be determined by it's dax attribute state.
OTOH, S_DAX is used by the kernel to enable DAX access. It is the _mechanism_, not the policy. We control S_DAX by mount option and/or the filesystems persistent inode state flag, and behaviour is determined by these policies at VFS inode instantiation time. Change the policy, turf the inode out of cache and re-instantiate it, and S_DAX for that inode is recalculated from the new policy.
For FUSE, the server provides the persistent storage and nothing else. The ioctls to manipulate dax state are client side ioctls, as are the mount options and the S_DAX vfs inode state. Hence for server side management of the per-inode S_DAX state, the FUSE protocol needs to be able to pass the per-inode persistent DAX attribute state to the client and the client needs to honor that attribute from the server.
How the FUSE server stores this persistent attribute is up to the server implementation. If the server wants FUSE access to be independent of host access, it can't use the persistent inode flags in the host filesystem - it will need to use it's own xattrs. If the server wants host access to be coherent with the guest, then it will need to implement that in a different way.
But the FUSE client doesn't care about any of this server side policy mumbo-jumbo - it just needs to do what the server sends to it in the DAX inode attribute. And that's exactly what dax=inode means...
Quoted 10 lines
In summary, there seem to be two use cases.
A. virtiofsd/fuse-server wants do be able to come up with its own policy to decide which inodes should use guest.
B. guest client decides which inode use DAX based on
FS_XFLAG_DAXattr on inode (and server does not play a role).To be able to different between these two cases, I was suggesting using "-o dax=inode" for B and "-o dax=server" for A.
dax=inode covrees both these cases. All the server side needs to do is set the inode attribute according to policy, and all the client needs to do is obey the server side per-inode attribute. IOWs, client side using "dax=inode" means the server side controls the DAX behaviour via the FUSE DAX attribute.
If the server wants the client to always use DAX, then it always sets the FUSE inode attribute. If the server says "never use DAX", then it never sets the FUSE inode attribute. If the server doesn't want the client to control policy, then it just rejects attempts to change the per-inode persistent DAX flag via client side ioctl(FS_XFLAG_DAX) calls. Hence we have use case A completely covered.
For case B, which is true dax=inode behaviour at the client, then the server side honours the client side requests to change the persistent FUSE DAX inode attribute via client side FS_XFLAG_DAX ioctls.
See? At the client side, there is absolutely no difference between server side policy control and true dax=inode support. The client is completely unaware of server side policies and that's how the client side should be implemented. The applications have to be prepared for policy to change dynamically with either dax=server or dax=inode, so there's no difference to applications running in the guest either.
Hence I just don't see the justification for a special DAX mode from an architectural POV. It's no more work to implement per-inode DAX properly form the start and we don't end up with a special, subtly different mode.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 36 lines · Dave Chinner
Quoted 26 lines · Vivek Goyal
Quoted 17 lines · Dave Chinner
Quoted 9 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
So will "-o dax=inode" be used for per file DAX where dax mode comes from server?
Quoted 6 lines
I think we discussed this. It will be better to leave "-o dax=inode" alone. It should be used when we are reading dax status from file attr (like ext4 and xfs).
And probably create separate option say "-o dax=server" where server specifies which inode should use dax.
Quoted 7 lines
That seems like a poor idea to me.
The server side already controls what the client side does by controlling the inode attributes that the client side sees. That is, if the server is going to specify whether the client side data access is going to use dax, then the server presents the client with an inode that has the DAX attribute flag set on it.
Quoted 8 lines
Hi Dave,
Currently in fuse/virtiofs, DAX is compltely controlled by client. Server has no say in it. If client is mounted with "-o dax", dax is enabled on all files otherwise dax is disabled on all files. One could think of implementing an option on server so that server could deny
mmap()requests that come from client, but so far nobody asked for such an option on server side.
Quoted 4 lines
Can you please refer to the new options "always|never|inode" in disucssions, because the "-o dax" option is deprecated and we really need to center discussions around the new options, not the deprecated one.
Consistent with ext4/xfs, "-o dax" is equal to "-o dax=always", and DAX is always enabled/disabled with "-o dax=always|never" regardless the per inode dax flag.
With "-o dax=inode", the client gets the per inode dax flag from FUSE protocol (during aops->lookup()) fed by server, deciding if DAX shall be enabled or not for this specific file.
Quoted 10 lines
Quoted 4 lines
When you say "inode that has DAX attribute flag set on it", are you referring to "
S_DAX(ininode->i_flags)" or persistent attr "FS_XFLAG_DAX"?Neither. The
S_DAXflags is the VFS struct inode state that tells the kernel what to do with that inode. TheFS_XFLAG_DAXis the userspace API status/control flag that represents the current state of the persistent per-inode DAX control attribute.
Quoted 9 lines
What I'm talking about the persistent attribute that is present on stable storage. e.g. in the XFS on-disk inode the flag is
XFS_DIFLAG2_DAXwhich is never seen outside low level XFS code. In the case of FUSE, this is the persistent server-side DAX attribute state. The client gets this information from the server via an inode attribute defined in the FUSE protocol. The client translates that network protocol attribute toS_DAXstatus on FS inode instantiation, and to/fromFS_XFLAG_DAXfor user control under dax=inode.
Got it.
Quoted 7 lines
In the case that the user changes
FS_XFLAG_DAX, the FUSE client needs to communicate that attribute change to the server, where the server then changes the persistent state of the on-disk inode so that the next time the client requests that inode, it gets the state it previously set. Unless, of course, there are server side policy overrides (never/always).
One thing I'm concerned with is that, is the following behavior consistent with the semantics of per-file DAX in ext4/xfs?
Client changes FS_XFLAG_DAX, this change is communicated to server and then server also returns **success**. Then client finds that this file is not DAX enabled, since server doesn't honor the previously set state.
IOWs, shall server always honor the persistent per-inode attribute of host file (if the change of FS_XFLAG_DAX inside guest is stored as persistent per-inode attribute on host file)?
Quoted 30 lines
IOWs, dax=inode on the client side is essentially "follow server side policy" because the server maintains the persistent DAX flag state in the server side inode...
Quoted 2 lines
As of now
S_DAXon client side inode is set by fuse client whenever client mounted filesystem with "-o dax". And I think you are assumingOf course. Similarly, if the client uses dax=never, the
S_DAXis never set on the VFS inode. But if dax=inode is set, where does the DAX attribute state that determinesS_DAXcomes from .....?Quoted 4 lines
that DAX attribute of inode is already coming from server. That's not the case. In fact that seems to be the proposal. Provide capability so that server can specify which inode should be using DAX and which inode should not be.
Yup, that's exactly what I'm talking about - this is "dax=inode" behaviour, because the client follows what the server tells it in the inode attributes that arrive over the wire.
Quoted 8 lines
Quoted 3 lines
In that case, turning off dax on the guest side should be communicated to the fuse server so the server turns off the DAX flag on the server side iff server side policy allows it.
Not sure what do you mean by server turns of DAX flag based on client turning off DAX. Server does not have to do anything. If client decides to not use DAX (in guest), it will not send
FUSE_SETUPMAPPINGrequests to server and that's it.
Quoted 4 lines
Where does the client get it's per-inode DAX policy from if dax=inode is, like other DAX filesystems, the default behaviour?
Where is the persistent storage of that per-inode attribute kept?
In the latest patch set, it is not supported yet to change FS_XFLAG_DAX (and thus setting/clearing persistent per-inode attribute) inside guest, since this scenario is not urgently needed as the real using case. Currently the per-inode dax attribute is completely fed from server thourgh FUSE protocol, e.g. server could set/clear the per-inode dax attribute depending on the file size.
The previous path set indeed had ever supported changing FS_XFLAG_DAX and accordingly setting/clearing persistent per-inode attribute inside guest. For "passthrough" type virtiofsd, the persistent per-inode attribute is stored as XFS_DIFLAG2_DAX/EXT4_DAX_FL on host file directly, since this is what "passthrough" means.
Quoted 156 lines
Quoted 15 lines
Quoted 5 lines
When the guest
then purges it's local inode and reloads it from the server then it will get an inode with the DAX flag set according to server side policy.So right now we don't have a mechanism for server to specify DAX flag. And that's what this patch series is trying to do.
Quoted 5 lines
Hence if the server side is configured with "dax=always" or dax="never" semantics it means the client side inode flag state cannot control DAX mode. That means, regardless of the client side mount options, DAX is operating under always or never policy,
Quoted 4 lines
Hmm..., When you say "server side is configured with "dax=always", do you mean shared directory on host is mounted with "-o dax=always", or you mean some virtiofs server specific option which enables dax on all inodes from server side.
I don't care. That's a server side implementation detail. You can keep it in a private xattr for all I care.
Quoted 4 lines
In general, DAX on host and DAX inside guest are completely independent. Host filesystem could very well be mounted with dax or without dax and that has no affect on guests's capability to be able to enable DAX or not.
If you have both the host and guest accessing the same files with different modes, then you have a data coherency problem that is guaranteed to corrupt data.
Quoted 7 lines
Quoted 3 lines
seems to be exactly what 'dax=inode' behaviour means on the client side - it behaves according to how the server side propagates the DAX attribute to the client for each inode.
Ok. So "-o dax=inode" in fuse will have a different meaning as opposed to ext4/xfs. This will mean that server will pass DAX state of inode when inode is instantiated and client should honor that.
No, that's exactly what dax=inode means: DAX behaviour is per-inode state that users must probe via
statx()to determine if dax is active or not.Quoted 3 lines
But what about
FS_XFLAG_DAXflag then. Say host file system does support this att and fuse/virtiofs allows querying and setting this attribute (I don't think it is allowed now).
FS_XFLAG_DAXis the ioctl-based control API for the client side. It's not a persistent flag.Quoted 3 lines
So will we not create a future conflict where in case of fuse/virtiofs "-o dax=inode" means something different and it does look at
FS_XFLAG_DAXfile attr.Like many, I suspect you might be mis-understanding the DAX API.
There are 4 parts to it:
- persistent filesystem inode state flag
- ioctls to manipulate persistent inode state flag (FS_XFLAG_DAX)
- mount options to override persistent inode state flag
- VFS inode state indicating DAX is active (S_DAX,STATX_ATTR_DAX)You seem to be conflating the user API
FS_XFLAG_DAXflag with whatever the filesystem uses to physically storage that state. They are not the same thing.FS_XFLAG_DAXalso has no correlation with the mount options - we can change the on-disk flag state regardless of the mount option in use. We can even change the on-disk state flag on *kernels that don't support DAX*. Changing the on-disk attribute *does not change S_DAX*.That's because the on-disk persistent state is a property of the filesystem's on-disk format, not a property of the kernel that is running on that machine. The persistent state flag is managed as a completely independent filesystem inode attribute, but it only affects behaviour when the dax=inode mode has been selected.
Control of the behaviour by this persistent inode flag is what dax=inode means. It does not define how that attribute flag is managed, it just defines the policy that an the inode's behaviour will be determined by it's dax attribute state.
OTOH,
S_DAXis used by the kernel to enable DAX access. It is the _mechanism_, not the policy. We controlS_DAXby mount option and/or the filesystems persistent inode state flag, and behaviour is determined by these policies at VFS inode instantiation time. Change the policy, turf the inode out of cache and re-instantiate it, andS_DAXfor that inode is recalculated from the new policy.For FUSE, the server provides the persistent storage and nothing else. The ioctls to manipulate dax state are client side ioctls, as are the mount options and the
S_DAXvfs inode state. Hence for server side management of the per-inodeS_DAXstate, the FUSE protocol needs to be able to pass the per-inode persistent DAX attribute state to the client and the client needs to honor that attribute from the server.How the FUSE server stores this persistent attribute is up to the server implementation. If the server wants FUSE access to be independent of host access, it can't use the persistent inode flags in the host filesystem - it will need to use it's own xattrs. If the server wants host access to be coherent with the guest, then it will need to implement that in a different way.
But the FUSE client doesn't care about any of this server side policy mumbo-jumbo - it just needs to do what the server sends to it in the DAX inode attribute. And that's exactly what dax=inode means...
Quoted 10 lines
In summary, there seem to be two use cases.
A. virtiofsd/fuse-server wants do be able to come up with its own policy to decide which inodes should use guest.
B. guest client decides which inode use DAX based on
FS_XFLAG_DAXattr on inode (and server does not play a role).To be able to different between these two cases, I was suggesting using "-o dax=inode" for B and "-o dax=server" for A.
dax=inode covrees both these cases. All the server side needs to do is set the inode attribute according to policy, and all the client needs to do is obey the server side per-inode attribute. IOWs, client side using "dax=inode" means the server side controls the DAX behaviour via the FUSE DAX attribute.
If the server wants the client to always use DAX, then it always sets the FUSE inode attribute. If the server says "never use DAX", then it never sets the FUSE inode attribute. If the server doesn't want the client to control policy, then it just rejects attempts to change the per-inode persistent DAX flag via client side
ioctl(FS_XFLAG_DAX)calls. Hence we have use case A completely covered.For case B, which is true dax=inode behaviour at the client, then the server side honours the client side requests to change the persistent FUSE DAX inode attribute via client side
FS_XFLAG_DAXioctls.See? At the client side, there is absolutely no difference between server side policy control and true dax=inode support. The client is completely unaware of server side policies and that's how the client side should be implemented. The applications have to be prepared for policy to change dynamically with either dax=server or dax=inode, so there's no difference to applications running in the guest either.
Hence I just don't see the justification for a special DAX mode from an architectural POV. It's no more work to implement per-inode DAX properly form the start and we don't end up with a special, subtly different mode.
Cheers,
Dave.
Signature · 2 lines
Thanks, Jeffle
Re: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 11 lines · JeffleXu
Quoted 9 lines · Dave Chinner
Quoted 2 lines · Vivek Goyal
Quoted 1 line · Dave Chinner
On Thu, Sep 23, 2021 at 03:02:41PM -0400, Vivek Goyal wrote:
In the case that the user changes
FS_XFLAG_DAX, the FUSE client needs to communicate that attribute change to the server, where the server then changes the persistent state of the on-disk inode so that the next time the client requests that inode, it gets the state it previously set. Unless, of course, there are server side policy overrides (never/always).
Quoted 6 lines
One thing I'm concerned with is that, is the following behavior consistent with the semantics of per-file DAX in ext4/xfs?
Client changes
FS_XFLAG_DAX, this change is communicated to server and then server also returns **success**. Then client finds that this file is not DAX enabled, since server doesn't honor the previously set state.
FS_XFLAG_DAX is advisory in nature - it does not have to be honored at inode instantiation.
Quoted 3 lines
IOWs, shall server always honor the persistent per-inode attribute of host file (if the change of
FS_XFLAG_DAXinside guest is stored as persistent per-inode attribute on host file)?
If the user set the flag, then queries it, the server should be returning the state that the user set, regardless of whether it is being honored at inode instantiation time.
Remember, FS_XFLAG_DAX does not imply S_DAX and vice versa.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Not sure what do you mean by server turns of DAX flag based on client turning off DAX. Server does not have to do anything. If client decides to not use DAX (in guest), it will not send
FUSE_SETUPMAPPINGrequests to server and that's it.
Quoted 8 lines
Quoted 4 lines
Where does the client get it's per-inode DAX policy from if dax=inode is, like other DAX filesystems, the default behaviour?
Where is the persistent storage of that per-inode attribute kept?
In the latest patch set, it is not supported yet to change
FS_XFLAG_DAX(and thus setting/clearing persistent per-inode attribute) inside guest, since this scenario is not urgently needed as the real using case.
AFAICT the FS_IOC_FS{GS}ETXATTR ioctl is already supported by the fuse client and it sends the ioctl to the server. Hence the client should already support persistent FS_XFLAG_DAX manipulations regardless of where/how the attribute is stored by the server. Did you actually add code to the client in this patchset to stop FS_XFLAG_DAX from being propagated to the server?
Quoted 3 lines
Currently the per-inode dax attribute is completely fed from server thourgh FUSE protocol, e.g. server could set/clear the per-inode dax attribute depending on the file size.
Yup, that's a policy dax=inode on the client side would allow. Indeed, this same policy could also be implemented as a client side policy, allowing user control instead of admin control of such conditional DAX behaviour... :)
Quoted 5 lines
The previous path set indeed had ever supported changing
FS_XFLAG_DAXand accordingly setting/clearing persistent per-inode attribute inside guest. For "passthrough" type virtiofsd, the persistent per-inode attribute is stored as XFS_DIFLAG2_DAX/EXT4_DAX_FL on host file directly, since this is what "passthrough" means.
Right, but that's server side storage implementation details, not a protocol or client side detail. What I can't find in the current client is where this per-inode flag is actually used in the FUSE dax inode init path - it just checks whether the connection has DAX state set up. Hence I don't see how FS_XFLAG_DAX control from the client currently has any influence on the client side DAX usage.
Seems somewhat crazy to me explicitly want to remove that client side control of per-inode behaviour whilst adding the missing client side bits that allow for the per-inode policy control from server side. Can we please just start with the common, compatible dax=inode behaviour on the client side, then layer the server side policy control options over the top of that?
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 15 lines · Dave Chinner
Quoted 14 lines · JeffleXu
Quoted 9 lines · Dave Chinner
Quoted 2 lines · Vivek Goyal
Quoted 1 line · Dave Chinner
On Thu, Sep 23, 2021 at 03:02:41PM -0400, Vivek Goyal wrote:
In the case that the user changes
FS_XFLAG_DAX, the FUSE client needs to communicate that attribute change to the server, where the server then changes the persistent state of the on-disk inode so that the next time the client requests that inode, it gets the state it previously set. Unless, of course, there are server side policy overrides (never/always).One thing I'm concerned with is that, is the following behavior consistent with the semantics of per-file DAX in ext4/xfs?
Quoted 6 lines
Quoted 3 lines
Client changes
FS_XFLAG_DAX, this change is communicated to server and then server also returns **success**. Then client finds that this file is not DAX enabled, since server doesn't honor the previously set state.
FS_XFLAG_DAXis advisory in nature - it does not have to be honored at inode instantiation.
Fine.
Quoted 10 lines
Quoted 4 lines
IOWs, shall server always honor the persistent per-inode attribute of host file (if the change of
FS_XFLAG_DAXinside guest is stored as persistent per-inode attribute on host file)?If the user set the flag, then queries it, the server should be returning the state that the user set, regardless of whether it is being honored at inode instantiation time.
Remember,
FS_XFLAG_DAXdoes not implyS_DAXand vice versa
Got it.
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Not sure what do you mean by server turns of DAX flag based on client turning off DAX. Server does not have to do anything. If client decides to not use DAX (in guest), it will not send
FUSE_SETUPMAPPINGrequests to server and that's it.Quoted 8 lines
Quoted 4 lines
Where does the client get it's per-inode DAX policy from if dax=inode is, like other DAX filesystems, the default behaviour?
Where is the persistent storage of that per-inode attribute kept?
In the latest patch set, it is not supported yet to change
FS_XFLAG_DAX(and thus setting/clearing persistent per-inode attribute) inside guest, since this scenario is not urgently needed as the real using case.
Quoted 6 lines
AFAICT the FS_IOC_FS{GS}ETXATTR ioctl is already supported by the fuse client and it sends the ioctl to the server. Hence the client should already support persistent
FS_XFLAG_DAXmanipulations regardless of where/how the attribute is stored by the server. Did you actually add code to the client in this patchset to stopFS_XFLAG_DAXfrom being propagated to the server?
Yes fuse client supports FS_IOC_FS{GS}ETXATTR ioctl already, but AFAIK "passthrough" type virtiofsd doesn't support FUSE_IOCTL yet. My previous patch had ever added support for FUSE_IOCTL to virtiofsd.
Quoted 16 lines
Quoted 4 lines
Currently the per-inode dax attribute is completely fed from server thourgh FUSE protocol, e.g. server could set/clear the per-inode dax attribute depending on the file size.
Yup, that's a policy dax=inode on the client side would allow. Indeed, this same policy could also be implemented as a client side policy, allowing user control instead of admin control of such conditional DAX behaviour... :)
Quoted 5 lines
The previous path set indeed had ever supported changing
FS_XFLAG_DAXand accordingly setting/clearing persistent per-inode attribute inside guest. For "passthrough" type virtiofsd, the persistent per-inode attribute is stored as XFS_DIFLAG2_DAX/EXT4_DAX_FL on host file directly, since this is what "passthrough" means.
Quoted 6 lines
Right, but that's server side storage implementation details, not a protocol or client side detail. What I can't find in the current client is where this per-inode flag is actually used in the FUSE dax inode init path - it just checks whether the connection has DAX state set up. Hence I don't see how
FS_XFLAG_DAXcontrol from the client currently has any influence on the client side DAX usage.
Fuse client fetches the per-inode DAX attribute from fuse_entry_out.attr.flags of FUSE_LOOKUP reply. It's implemented in patch 4 of this patch set.
The background info is that, fuse client will send a FUSE_LOOKUP request to server during inode instantiation, while FS_XFLAG_DAX flag is not included in the FUSE_LOOKUP reply, and thus fuse client need to send another FUSE_IOCTL if it wants to query the persistent inode flags. To remove this extra fuse request during inode instantiation, this flag is merged into FUSE_LOOKUP reply (fuse_entry_out.attr.flags) as FUSE_ATTR_DAX. Then if FUSE_ATTR_DAX flag is set in fuse_entry_out.attr.flags, then fuse client knows that this file shall be DAX enabled.
IOWs, under this mechanism it relies on fuse server to check persistent inode flags on host, and then set FUSE_ATTR_DAX flag accordingly.
Quoted 7 lines
Seems somewhat crazy to me explicitly want to remove that client side control of per-inode behaviour whilst adding the missing client side bits that allow for the per-inode policy control from server side. Can we please just start with the common, compatible dax=inode behaviour on the client side, then layer the server side policy control options over the top of that?
Hi Vivek,
It seems that we shall also support setting/clearing FS_XFLAG_DAX inside guest? If that's the case, then how to design virtiofsd behavior? I mean, is it mandatory or optional for virtiofsd to query FS_XFLAG_DAX of host files when guest is mounted with "dax=inode"? If it's optional, then the performance may be better since it doesn't need to do one extra FS_IOC_FSGETXATTR ioctl when handling FUSE_LOOKUP, but admin needs to specify "-o policy=flag" to virtiofsd explicitly if it's really needed.
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerRe: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 116 lines · JeffleXu
Quoted 15 lines · Dave Chinner
Quoted 14 lines · JeffleXu
Quoted 9 lines · Dave Chinner
Quoted 2 lines · Vivek Goyal
Quoted 1 line · Dave Chinner
On Thu, Sep 23, 2021 at 03:02:41PM -0400, Vivek Goyal wrote:
In the case that the user changes
FS_XFLAG_DAX, the FUSE client needs to communicate that attribute change to the server, where the server then changes the persistent state of the on-disk inode so that the next time the client requests that inode, it gets the state it previously set. Unless, of course, there are server side policy overrides (never/always).One thing I'm concerned with is that, is the following behavior consistent with the semantics of per-file DAX in ext4/xfs?
Quoted 6 lines
Quoted 3 lines
Client changes
FS_XFLAG_DAX, this change is communicated to server and then server also returns **success**. Then client finds that this file is not DAX enabled, since server doesn't honor the previously set state.
FS_XFLAG_DAXis advisory in nature - it does not have to be honored at inode instantiation.Fine.
Quoted 10 lines
Quoted 4 lines
IOWs, shall server always honor the persistent per-inode attribute of host file (if the change of
FS_XFLAG_DAXinside guest is stored as persistent per-inode attribute on host file)?If the user set the flag, then queries it, the server should be returning the state that the user set, regardless of whether it is being honored at inode instantiation time.
Remember,
FS_XFLAG_DAXdoes not implyS_DAXand vice versaGot it.
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Not sure what do you mean by server turns of DAX flag based on client turning off DAX. Server does not have to do anything. If client decides to not use DAX (in guest), it will not send
FUSE_SETUPMAPPINGrequests to server and that's it.Quoted 8 lines
Quoted 4 lines
Where does the client get it's per-inode DAX policy from if dax=inode is, like other DAX filesystems, the default behaviour?
Where is the persistent storage of that per-inode attribute kept?
In the latest patch set, it is not supported yet to change
FS_XFLAG_DAX(and thus setting/clearing persistent per-inode attribute) inside guest, since this scenario is not urgently needed as the real using case.Quoted 6 lines
AFAICT the FS_IOC_FS{GS}ETXATTR ioctl is already supported by the fuse client and it sends the ioctl to the server. Hence the client should already support persistent
FS_XFLAG_DAXmanipulations regardless of where/how the attribute is stored by the server. Did you actually add code to the client in this patchset to stopFS_XFLAG_DAXfrom being propagated to the server?Yes fuse client supports FS_IOC_FS{GS}ETXATTR ioctl already, but AFAIK "passthrough" type virtiofsd doesn't support
FUSE_IOCTLyet. My previous patch had ever added support forFUSE_IOCTLto virtiofsd.Quoted 16 lines
Quoted 4 lines
Currently the per-inode dax attribute is completely fed from server thourgh FUSE protocol, e.g. server could set/clear the per-inode dax attribute depending on the file size.
Yup, that's a policy dax=inode on the client side would allow. Indeed, this same policy could also be implemented as a client side policy, allowing user control instead of admin control of such conditional DAX behaviour... :)
Quoted 5 lines
The previous path set indeed had ever supported changing
FS_XFLAG_DAXand accordingly setting/clearing persistent per-inode attribute inside guest. For "passthrough" type virtiofsd, the persistent per-inode attribute is stored as XFS_DIFLAG2_DAX/EXT4_DAX_FL on host file directly, since this is what "passthrough" means.Quoted 6 lines
Right, but that's server side storage implementation details, not a protocol or client side detail. What I can't find in the current client is where this per-inode flag is actually used in the FUSE dax inode init path - it just checks whether the connection has DAX state set up. Hence I don't see how
FS_XFLAG_DAXcontrol from the client currently has any influence on the client side DAX usage.Fuse client fetches the per-inode DAX attribute from fuse_entry_out.attr.flags of
FUSE_LOOKUPreply. It's implemented in patch 4 of this patch set.The background info is that, fuse client will send a
FUSE_LOOKUPrequest to server during inode instantiation, whileFS_XFLAG_DAXflag is not included in theFUSE_LOOKUPreply, and thus fuse client need to send anotherFUSE_IOCTLif it wants to query the persistent inode flags. To remove this extra fuse request during inode instantiation, this flag is merged intoFUSE_LOOKUPreply (fuse_entry_out.attr.flags) asFUSE_ATTR_DAX. Then ifFUSE_ATTR_DAXflag is set in fuse_entry_out.attr.flags, then fuse client knows that this file shall be DAX enabled.IOWs, under this mechanism it relies on fuse server to check persistent inode flags on host, and then set
FUSE_ATTR_DAXflag accordingly.Quoted 7 lines
Seems somewhat crazy to me explicitly want to remove that client side control of per-inode behaviour whilst adding the missing client side bits that allow for the per-inode policy control from server side. Can we please just start with the common, compatible dax=inode behaviour on the client side, then layer the server side policy control options over the top of that?
Hi Vivek,
Quoted 7 lines
It seems that we shall also support setting/clearing
FS_XFLAG_DAXinside guest? If that's the case, then how to design virtiofsd behavior? I mean, is it mandatory or optional for virtiofsd to queryFS_XFLAG_DAXof host files when guest is mounted with "dax=inode"? If it's optional, then the performance may be better since it doesn't need to do one extraFS_IOC_FSGETXATTRioctl when handlingFUSE_LOOKUP, but admin needs to specify "-o policy=flag" to virtiofsd explicitly if it's really needed.
Hi Jeffle,
How about first doing a patch series to just enable ioctl in virtiofsd. I know David Gilbert and others had security concenrs w.r.t. These are coming from untrusted guest and they had concerns that we should only allow selective operations as needed (opted-in by admin). So may be a daemon option which specifies which operations to allow.
Once that's done, we probably will have to do a patch series, to make sure FS_XFLAG_DAX inherit behavior is working properly. Especially that behavior about inheriting FS_XFLAG_DAX flag when a new file is created and parent dir has FS_XFLAG_DAX set. May be we can just rely on host filesystem doing it? Limitation will be that it will only work if host fs is ext4/xfs.
w.r.t virtiofsd, I think we can provide an option say "-o dax_policy=<option>", which controls what policy is in effect. So if a daemon specific policy is in effect, we can skip checking FS_XFLAG_DAX state. In fact, checking FS_XFLAG_DAX state also should probably be a policy option and not enabled by default.
Say, "-o dax_policy=FS_XFLAG_DAX" will enable cheking FS_XFLAG_DAX on inode during FUSE_LOOKUP time. Otherwise daemon can fallback to its own policy and set DAX flag in lookup reply accordingly.
Vivek
Vivek Goyal
maintainerRe: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
In reply to Dave ChinnerQuoted 178 lines · Dave Chinner
Quoted 26 lines · Vivek Goyal
Quoted 17 lines · Dave Chinner
Quoted 9 lines · Vivek Goyal
Quoted 4 lines · Jeffle Xu
We add 'always', 'never', and 'inode' (default). '-o dax' continues to operate the same which is equivalent to 'always'. To be consistemt with ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax=' option is specified, the default behaviour is equal to 'inode'.
So will "-o dax=inode" be used for per file DAX where dax mode comes from server?
Quoted 6 lines
I think we discussed this. It will be better to leave "-o dax=inode" alone. It should be used when we are reading dax status from file attr (like ext4 and xfs).
And probably create separate option say "-o dax=server" where server specifies which inode should use dax.
Quoted 7 lines
That seems like a poor idea to me.
The server side already controls what the client side does by controlling the inode attributes that the client side sees. That is, if the server is going to specify whether the client side data access is going to use dax, then the server presents the client with an inode that has the DAX attribute flag set on it.
Quoted 8 lines
Hi Dave,
Currently in fuse/virtiofs, DAX is compltely controlled by client. Server has no say in it. If client is mounted with "-o dax", dax is enabled on all files otherwise dax is disabled on all files. One could think of implementing an option on server so that server could deny
mmap()requests that come from client, but so far nobody asked for such an option on server side.Can you please refer to the new options "always|never|inode" in disucssions, because the "-o dax" option is deprecated and we really need to center discussions around the new options, not the deprecated one.
Quoted 3 lines
When you say "inode that has DAX attribute flag set on it", are you referring to "
S_DAX(ininode->i_flags)" or persistent attr "FS_XFLAG_DAX"?Neither. The
S_DAXflags is the VFS struct inode state that tells the kernel what to do with that inode. TheFS_XFLAG_DAXis the userspace API status/control flag that represents the current state of the persistent per-inode DAX control attribute.What I'm talking about the persistent attribute that is present on stable storage. e.g. in the XFS on-disk inode the flag is
XFS_DIFLAG2_DAXwhich is never seen outside low level XFS code. In the case of FUSE, this is the persistent server-side DAX attribute state. The client gets this information from the server via an inode attribute defined in the FUSE protocol. The client translates that network protocol attribute toS_DAXstatus on FS inode instantiation, and to/fromFS_XFLAG_DAXfor user control under dax=inode.In the case that the user changes
FS_XFLAG_DAX, the FUSE client needs to communicate that attribute change to the server, where the server then changes the persistent state of the on-disk inode so that the next time the client requests that inode, it gets the state it previously set. Unless, of course, there are server side policy overrides (never/always).IOWs, dax=inode on the client side is essentially "follow server side policy" because the server maintains the persistent DAX flag state in the server side inode...
Quoted 2 lines
As of now
S_DAXon client side inode is set by fuse client whenever client mounted filesystem with "-o dax". And I think you are assumingOf course. Similarly, if the client uses dax=never, the
S_DAXis never set on the VFS inode. But if dax=inode is set, where does the DAX attribute state that determinesS_DAXcomes from .....?Quoted 4 lines
that DAX attribute of inode is already coming from server. That's not the case. In fact that seems to be the proposal. Provide capability so that server can specify which inode should be using DAX and which inode should not be.
Yup, that's exactly what I'm talking about - this is "dax=inode" behaviour, because the client follows what the server tells it in the inode attributes that arrive over the wire.
Quoted 8 lines
Quoted 3 lines
In that case, turning off dax on the guest side should be communicated to the fuse server so the server turns off the DAX flag on the server side iff server side policy allows it.
Not sure what do you mean by server turns of DAX flag based on client turning off DAX. Server does not have to do anything. If client decides to not use DAX (in guest), it will not send
FUSE_SETUPMAPPINGrequests to server and that's it.Where does the client get it's per-inode DAX policy from if dax=inode is, like other DAX filesystems, the default behaviour?
Where is the persistent storage of that per-inode attribute kept?
Quoted 14 lines
Quoted 4 lines
When the guest
then purges it's local inode and reloads it from the server then it will get an inode with the DAX flag set according to server side policy.So right now we don't have a mechanism for server to specify DAX flag. And that's what this patch series is trying to do.
Quoted 5 lines
Hence if the server side is configured with "dax=always" or dax="never" semantics it means the client side inode flag state cannot control DAX mode. That means, regardless of the client side mount options, DAX is operating under always or never policy,
Quoted 4 lines
Hmm..., When you say "server side is configured with "dax=always", do you mean shared directory on host is mounted with "-o dax=always", or you mean some virtiofs server specific option which enables dax on all inodes from server side.
I don't care. That's a server side implementation detail. You can keep it in a private xattr for all I care.
Quoted 4 lines
In general, DAX on host and DAX inside guest are completely independent. Host filesystem could very well be mounted with dax or without dax and that has no affect on guests's capability to be able to enable DAX or not.
If you have both the host and guest accessing the same files with different modes, then you have a data coherency problem that is guaranteed to corrupt data.
Quoted 7 lines
Quoted 3 lines
seems to be exactly what 'dax=inode' behaviour means on the client side - it behaves according to how the server side propagates the DAX attribute to the client for each inode.
Ok. So "-o dax=inode" in fuse will have a different meaning as opposed to ext4/xfs. This will mean that server will pass DAX state of inode when inode is instantiated and client should honor that.
No, that's exactly what dax=inode means: DAX behaviour is per-inode state that users must probe via
statx()to determine if dax is active or not.Quoted 3 lines
But what about
FS_XFLAG_DAXflag then. Say host file system does support this att and fuse/virtiofs allows querying and setting this attribute (I don't think it is allowed now).
FS_XFLAG_DAXis the ioctl-based control API for the client side. It's not a persistent flag.Quoted 3 lines
So will we not create a future conflict where in case of fuse/virtiofs "-o dax=inode" means something different and it does look at
FS_XFLAG_DAXfile attr.Like many, I suspect you might be mis-understanding the DAX API.
There are 4 parts to it:
- persistent filesystem inode state flag
- ioctls to manipulate persistent inode state flag (FS_XFLAG_DAX)
- mount options to override persistent inode state flag
- VFS inode state indicating DAX is active (S_DAX,STATX_ATTR_DAX)You seem to be conflating the user API
FS_XFLAG_DAXflag with whatever the filesystem uses to physically storage that state. They are not the same thing.FS_XFLAG_DAXalso has no correlation with the mount options - we can change the on-disk flag state regardless of the mount option in use. We can even change the on-disk state flag on *kernels that don't support DAX*. Changing the on-disk attribute *does not change S_DAX*.That's because the on-disk persistent state is a property of the filesystem's on-disk format, not a property of the kernel that is running on that machine. The persistent state flag is managed as a completely independent filesystem inode attribute, but it only affects behaviour when the dax=inode mode has been selected.
Quoted 4 lines
Control of the behaviour by this persistent inode flag is what dax=inode means. It does not define how that attribute flag is managed, it just defines the policy that an the inode's behaviour will be determined by it's dax attribute state.
So here is my confusion. Say client mounts filesystem with "dax=inode" and file "foo.txt" has FS_XFLAG_DAX set, do I have
Quoted 36 lines
OTOH,
S_DAXis used by the kernel to enable DAX access. It is the _mechanism_, not the policy. We controlS_DAXby mount option and/or the filesystems persistent inode state flag, and behaviour is determined by these policies at VFS inode instantiation time. Change the policy, turf the inode out of cache and re-instantiate it, andS_DAXfor that inode is recalculated from the new policy.For FUSE, the server provides the persistent storage and nothing else. The ioctls to manipulate dax state are client side ioctls, as are the mount options and the
S_DAXvfs inode state. Hence for server side management of the per-inodeS_DAXstate, the FUSE protocol needs to be able to pass the per-inode persistent DAX attribute state to the client and the client needs to honor that attribute from the server.How the FUSE server stores this persistent attribute is up to the server implementation. If the server wants FUSE access to be independent of host access, it can't use the persistent inode flags in the host filesystem - it will need to use it's own xattrs. If the server wants host access to be coherent with the guest, then it will need to implement that in a different way.
But the FUSE client doesn't care about any of this server side policy mumbo-jumbo - it just needs to do what the server sends to it in the DAX inode attribute. And that's exactly what dax=inode means...
Quoted 8 lines
In summary, there seem to be two use cases.
A. virtiofsd/fuse-server wants do be able to come up with its own policy to decide which inodes should use guest.
B. guest client decides which inode use DAX based on
FS_XFLAG_DAXattr on inode (and server does not play a role).
Quoted 8 lines
Quoted 2 lines
To be able to different between these two cases, I was suggesting using "-o dax=inode" for B and "-o dax=server" for A.
dax=inode covrees both these cases. All the server side needs to do is set the inode attribute according to policy, and all the client needs to do is obey the server side per-inode attribute. IOWs, client side using "dax=inode" means the server side controls the DAX behaviour via the FUSE DAX attribute.
Hi Dave,
Can a filesystem mounted with "-o dax=inode" enable DAX on a file even if FS_XFLAG_DAX attr is not set on the file. I think that's something new which will happen in case of fuse and currently does not happen with ext4/xfs.
That's the use case A which wants to enable DAX attribute of a file based on its own policy (irrespective of state of FS_XFLAG_DAX on file).
IIUC, you are ok with this. Just want to be sure because this is a subtle change from ext4/xfs behavior which will only enable DAX on a file if FS_XFLAG_DAX is set (with dax=inode). IOW, on ext4/xfs if FS_XFLAG_DAX is not set on file then DAX will not be enabled (dax=inode).
I don't want applications to be making assumption that if FS_XFFLAG_DAX is not set on a file, then DAX will not be enabled on that file because that's what exactly fuse/virtiofs can do.
Quoted 8 lines
If the server wants the client to always use DAX, then it always sets the FUSE inode attribute. If the server says "never use DAX", then it never sets the FUSE inode attribute. If the server doesn't want the client to control policy, then it just rejects attempts to change the per-inode persistent DAX flag via client side
ioctl(FS_XFLAG_DAX)calls. Hence we have use case A completely covered.
Ok, I think the behavior of ioctl(FS_XFLAG_DAX) calls is confusing me most in this context. IIUC, you are suggesting that server will define the policy whether it is supporting ioctl(FS_XFLAG_DAX) or not. This is will part of feature negotiation of fuse protocol.
If server decides to support ioctl(FS_XFLAG_DAX), then it should work similar to ext4/xfs and also follow inheritance rules.
If server decides to not support (or not enable) ioctl(FS_XFLAG_DAX), then server need to reject any attempt from client to set FS_XFLAG_DAX. And if client queries the current state of FS_XFLAG_DAX, then we need to return it is not set (even if it set on underlying filesystem). That way client will think FS_XFLAG_DAX is not set and will not expect DAX to be necessarily enabled.
Quoted 13 lines
For case B, which is true dax=inode behaviour at the client, then the server side honours the client side requests to change the persistent FUSE DAX inode attribute via client side
FS_XFLAG_DAXioctls.See? At the client side, there is absolutely no difference between server side policy control and true dax=inode support. The client is completely unaware of server side policies and that's how the client side should be implemented. The applications have to be prepared for policy to change dynamically with either dax=server or dax=inode, so there's no difference to applications running in the guest either.
Quoted 4 lines
Hence I just don't see the justification for a special DAX mode from an architectural POV. It's no more work to implement per-inode DAX properly form the start and we don't end up with a special, subtly different mode.
Ok, So I guess initially we could just implement "-o dax=inode" and *not support FS_XFLAG_DAX" API at all. Fuse server will decide which inodes should enable DAX and communicate that to client in a message response as part of protocol. Any attempt by client to set attr FS_XFLAG_DAX should fail (Say -ENOTSUP). And any query of FS_XFLAG_DAX attr should return attr is not set.
One could support "FS_XFLAG_DAX" API down the line. And in that case fuser server should allow setting and querying FS_XFLAG_DAX and follow inheritance rules. Also dax flag of inode should be set according to state of FS_XFLAG_DAX attr (and not some internal decision of server). IOW, server will have to follow state of FS_XFLAG_DAX attr in deciding DAX state of inode if it decided to support FS_XFLAG_DAX attr API.
Thanks
Vivek
Re: [PATCH v5 2/5] fuse: make DAX mount option a tri-state
Quoted 19 lines · Vivek Goyal
Quoted 9 lines · Dave Chinner
Quoted 8 lines · Vivek Goyal
In summary, there seem to be two use cases.
A. virtiofsd/fuse-server wants do be able to come up with its own policy to decide which inodes should use guest.
B. guest client decides which inode use DAX based on
FS_XFLAG_DAXattr on inode (and server does not play a role).Quoted 8 lines
Quoted 2 lines
To be able to different between these two cases, I was suggesting using "-o dax=inode" for B and "-o dax=server" for A.
dax=inode covrees both these cases. All the server side needs to do is set the inode attribute according to policy, and all the client needs to do is obey the server side per-inode attribute. IOWs, client side using "dax=inode" means the server side controls the DAX behaviour via the FUSE DAX attribute.
Quoted 6 lines
Hi Dave,
Can a filesystem mounted with "-o dax=inode" enable DAX on a file even if
FS_XFLAG_DAXattr is not set on the file. I think that's something new which will happen in case of fuse and currently does not happen with ext4/xfs.
It can happen - the on-disk flag is defined as advisory in Documentation/filesystems/dax.rst:
2. There exists a persistent flag FS_XFLAG_DAX that can be applied to regular files and directories. This advisory flag can be set or cleared at any time, but doing so does not immediately affect the S_DAX state.
So, we can have things like the inode get initialised, then the on disk flag is cleared. Now you have an inode that is accessed by DAX in memory, and it will continue to be accessed using DAX until to it removed from memory. IOWs, you can then check the on disk flag and see that it is clear, yet check statx(STATX_ATTR_DAX) and see that DAX is enabled.
Another counter example is that an inode has been reflink copied and then the on-disk flag is set on the inode. Some time later it is instantiated in memory, the filesystem sees that it has DAX and shared extents and turns off DAX because it's not supported with shared extents right now. So you can check the on-disk flag and see that DAX -should- be enabled, but when you check statx(STATX_ATTR_DAX) you will always set that DAX is disabled.
Another example: on disk bit is set, file contains inline data, or compressed data, or encrypted data and so cannot provide direct access to the storage. In which case, the filesysetm will ignore the on disk bit and never use DAX on that inode.
IOWs, dax=inode means that the inode contains the *desired policy for that inode*. It is a hint, not an enforced policy. The filesystem and or kernel infrastructure can decide to ignore the hint, and that is one of the reasons why checking statx(STATX_ATTR_DAX) is required if the application needs DAX to operate correctly.
Quoted 3 lines
That's the use case A which wants to enable DAX attribute of a file based on its own policy (irrespective of state of
FS_XFLAG_DAXon file).
Perfectly fine - dax=inode simply says "DAX policy is defined on per-inode granularity". It does not guarantee any specific behaviour will be followed because the per-inode policy flag is advisory.
Quoted 5 lines
IIUC, you are ok with this. Just want to be sure because this is a subtle change from ext4/xfs behavior which will only enable DAX on a file if
FS_XFLAG_DAXis set (with dax=inode). IOW, on ext4/xfs ifFS_XFLAG_DAXis not set on file then DAX will not be enabled (dax=inode).
Again, the flag is -advisory- so there is nothing stopping us from having other policies that combine with or override the on-disk flag.
Fundamentally, the options are:
- "always" meaning *everything* *always* uses DAX.
- "never" meaning *nothing* *ever* uses DAX.
- "inode" meaning "derive behaviour from the inode advisory flag".
"dax=inode" does not rule out behaviour being derived from other policy mechanisms - it just means that in the absence of any other policy, decide behaviour dynamically on a per-inode granularity. We defined an API to set/clear that advisory flag on a per-inode basis, and how it gets used is only limited by your imagination.
For example, the "inherit DAX from parent directory" policy is based on setting the DAX flag set on the directory. We cannot use DAX to access directory data directly, so this flag on a directory becomes a *policy propagation mechanism* rather than an enabling hint.
Hence the user might not even be aware that the admin of the system (or even a package installer) set up DAX propagation policies. Hence all their data directories are defaulting to using DAX even though they never set a DAX flag or dax=always mount option themselves.
IOWs, having the FUSE server dynamically and/or silently propagate, control or influence client side DAX behaviour under client side dax=inode behaviour falls well within the "admin controlled policy propoagation" guise that dax=inode is supposed to allow admins to implement.
Quoted 3 lines
I don't want applications to be making assumption that if
FS_XFFLAG_DAXis not set on a file, then DAX will not be enabled on that file because that's what exactly fuse/virtiofs can do.
Applications *must* check statx(STATX_ATTR_DAX) after they have opened the file to determine if DAX is in use. FS_XFLAG_DAX is an advisory hint, not an indication of current behaviour, and it's irrelevant if dax=never or dax=always is in use. In those cases, statx() is pretty much the only way to sanely check the DAX state of an open file.
Quoted 16 lines
Quoted 7 lines
If the server wants the client to always use DAX, then it always sets the FUSE inode attribute. If the server says "never use DAX", then it never sets the FUSE inode attribute. If the server doesn't want the client to control policy, then it just rejects attempts to change the per-inode persistent DAX flag via client side
ioctl(FS_XFLAG_DAX)calls. Hence we have use case A completely covered.Ok, I think the behavior of
ioctl(FS_XFLAG_DAX)calls is confusing me most in this context. IIUC, you are suggesting that server will define the policy whether it is supportingioctl(FS_XFLAG_DAX)or not. This is will part of feature negotiation of fuse protocol.If server decides to support
ioctl(FS_XFLAG_DAX), then it should work similar to ext4/xfs and also follow inheritance rules.
Quoted 6 lines
If server decides to not support (or not enable)
ioctl(FS_XFLAG_DAX), then server need to reject any attempt from client to setFS_XFLAG_DAX. And if client queries the current state ofFS_XFLAG_DAX, then we need to return it is not set (even if it set on underlying filesystem). That way client will thinkFS_XFLAG_DAXis not set and will not expect DAX to be necessarily enabled.
No, I'm not suggesting that's what you do. How the server controls the DAX policy is largely irrelevant to the discussion.
Start by looking at how the client implements FS_XFLAG_DAX. It's already supported, because the fuse client already has the FSGETXATTR/FSSETXATTR protocol mechanisms for doing this (fuse_fileattr_get(), fuse_fileattr_set()). Hence the client is already passing FS_XFLAG_DAX to the server for persistent storage, and it should already be getting it back from the server.
The basis of XFS/ext4 dax=inode behaviour is already implemented in the fuse client. All you need to do is hook up fuse_dax_inode_init() to look at that flag to determine behaviour instead of always applying the connection DAX state as the determination. i.e.:
if (!fc->dax)
return;
if ("dax=never")
return;
if ("dax=inode" && !fuse_inode_has_dax(inode))
return;
inode->i_flags |= S_DAX;
inode->i_data.a_ops = &fuse_dax_file_aops;And there's not much more to implement on the client side for dax=inode behaviour.
So, if you want server side policy control, the FUSE protocol needs to ensure that fuse_inode_has_dax(inode) follows whatever the server side requires the policy to be. This may be combined with the user FS_XFLAG_DAX attribute, or it may be a separate inode flag that overrides FS_XFLAG_DAX. But that's FUSE implementation detail, and not something I'm concerned about.
The fact is that server side override doesn't change the client side logic or behaviour. The FS_XFLAG_DAX behaviour follows what the user sets, dax=inode follows that in the absence of other policy overrides that the user may or may not know about but are applied on a per-inode granularity...
Quoted 12 lines
Quoted 12 lines
For case B, which is true dax=inode behaviour at the client, then the server side honours the client side requests to change the persistent FUSE DAX inode attribute via client side
FS_XFLAG_DAXioctls.See? At the client side, there is absolutely no difference between server side policy control and true dax=inode support. The client is completely unaware of server side policies and that's how the client side should be implemented. The applications have to be prepared for policy to change dynamically with either dax=server or dax=inode, so there's no difference to applications running in the guest either.
Quoted 7 lines
Quoted 4 lines
Hence I just don't see the justification for a special DAX mode from an architectural POV. It's no more work to implement per-inode DAX properly form the start and we don't end up with a special, subtly different mode.
Ok, So I guess initially we could just implement "-o dax=inode" and *not support
FS_XFLAG_DAX" API at all.
No, please don't do that, and please stop wasting my time by making me have to repeatedly explain to you why doing stuff like this is harmful to our user base. Implement compatible, common dax=inode behaviour *first*, then layer your server side policy variants on top of that.
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH v5 3/5 fuse: support per-file DAX
In reply to Jeffle XuExpand the fuse protocol to support per-file DAX.
FUSE_ATTR_DAX flag is added indicating if DAX shall be enabled for corresponding file when replying FUSE_LOOKUP request.
--- include/uapi/linux/fuse.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
include/uapi/linux/fuse.h +6 -1
@@ -184,6 +184,9 @@* * 7.34 * - add FUSE_SYNCFS +* +* 7.35 +* - add FUSE_ATTR_DAX */ #ifndef _LINUX_FUSE_H
@@ -219,7 +222,7 @@#define FUSE_KERNEL_VERSION 7/** Minor version number of this interface */ -#define FUSE_KERNEL_MINOR_VERSION 34 +#define FUSE_KERNEL_MINOR_VERSION 35/** The node ID of the root inode */ #define FUSE_ROOT_ID 1
@@ -449,8 +452,10 @@ struct fuse_file_lock {* fuse_attr flags * * FUSE_ATTR_SUBMOUNT: Object is a submount root +* FUSE_ATTR_DAX: Enable DAX for this file in per-file DAX mode */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)/** * Open flags
Signature · 1 line
2.27.0
PATCH v5 4/5 fuse: enable per-file DAX
In reply to Jeffle XuDAX may be limited in some specific situation. When the number of usable DAX windows is under watermark, the recalim routine will be triggered to reclaim some DAX windows. It may have a negative impact on the performance, since some processes may need to wait for DAX windows to be recalimed and reused then. To mitigate the performance degradation, the overall DAX window need to be expanded larger.
However, simply expanding the DAX window may not be a good deal in some scenario. To maintain one DAX window chunk (i.e., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest, which is greater than the memory footprint if it uses guest page cache when DAX disabled. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Per-file DAX feature is introduced to address this issue, by offering a finer grained control for dax to users, trying to achieve a balance between performance and memory overhead.
The FUSE_ATTR_DAX flag in FUSE_LOOKUP reply is used to indicate whether DAX should be enabled or not for corresponding file. Currently the state whether DAX is enabled or not for the file is initialized only when inode is instantiated.
--- fs/fuse/dax.c | 12 ++++++++---- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 4 ++-- fs/fuse/inode.c | 2 +- 4 files changed, 13 insertions(+), 9 deletions(-)
fs/fuse/dax.c +8 -4
@@ -1333,7 +1333,7 @@ static const struct address_space_operations fuse_dax_file_aops = { .invalidatepage = noop_invalidatepage, }; -static bool fuse_should_enable_dax (struct inode *inode) +static bool fuse_should_enable_dax (struct inode *inode,unsigned int flags) {struct fuse_conn *fc =get_fuse_conn (inode);unsigned int dax_mode = fc->dax_mode;
@@ -1342,12 +1342,16 @@ static bool fuse_should_enable_dax(struct inode *inode)if (dax_mode ==FUSE_DAX_NEVER )return false; -return true; +if (dax_mode ==FUSE_DAX_ALWAYS ) +return true; + +/* dax_mode == FUSE_DAX_INODE */ +return flags &FUSE_ATTR_DAX ; } -void fuse_dax_inode_init (struct inode *inode) +void fuse_dax_inode_init (struct inode *inode,unsigned int flags) { -if (!fuse_should_enable_dax (inode)) +if (!fuse_should_enable_dax (inode, flags))return ; inode->i_flags |=S_DAX ;
fs/fuse/file.c +2 -2
@@ -3163,7 +3163,7 @@ static const struct address_space_operations fuse_file_aops = { .write_end = fuse_write_end, }; -void fuse_init_file_inode (struct inode *inode) +void fuse_init_file_inode (struct inode *inode,unsigned int flags) {struct fuse_inode *fi =get_fuse_inode (inode);
@@ -3177,5 +3177,5 @@ void fuse_init_file_inode(struct inode *inode) fi->writepages =RB_ROOT ;if (IS_ENABLED (CONFIG_FUSE_DAX )) -fuse_dax_inode_init (inode); +fuse_dax_inode_init (inode, flags); }
fs/fuse/fuse_i.h +2 -2
@@ -1016,7 +1016,7 @@ int fuse_notify_poll_wakeup(struct fuse_conn *fc,/** * Initialize file operations on a regular file */ -void fuse_init_file_inode (struct inode *inode); +void fuse_init_file_inode (struct inode *inode,unsigned int flags);/** * Initialize inode operations on regular files and special files
@@ -1268,7 +1268,7 @@ int fuse_dax_conn_alloc(struct fuse_conn *fc, enum fuse_dax_mode mode, struct dax_device *dax_dev);void fuse_dax_conn_free (struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi); -void fuse_dax_inode_init (struct inode *inode); +void fuse_dax_inode_init (struct inode *inode,unsigned int flags);void fuse_dax_inode_cleanup (struct inode *inode);bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment);void fuse_dax_cancel_work (struct fuse_conn *fc);
fs/fuse/inode.c +1 -1
@@ -280,7 +280,7 @@ static void fuse_init_inode(struct inode *inode, struct fuse_attr *attr) inode->i_ctime.tv_nsec = attr->ctimensec;if (S_ISREG (inode->i_mode)) {fuse_init_common (inode); -fuse_init_file_inode (inode); +fuse_init_file_inode (inode, attr->flags); }else if (S_ISDIR (inode->i_mode))fuse_init_dir (inode);else if (S_ISLNK (inode->i_mode))
Signature · 1 line
2.27.0
PATCH v5 5/5 fuse: mark inode DONT_CACHE when per-file DAX hint changes
In reply to Jeffle XuWhen the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state.
Hence mark the inode and corresponding dentries as DONE_CACHE once the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.
In summary, when the per-file DAX hint changes for an *opened* file, the DAX state of the file won't be updated until this file is closed and reopened later.
--- fs/fuse/dax.c | 9 +++++++++ fs/fuse/fuse_i.h | 1 + fs/fuse/inode.c | 3 +++ 3 files changed, 13 insertions(+)
fs/fuse/dax.c +9 -0
@@ -1358,6 +1358,15 @@ void fuse_dax_inode_init(struct inode *inode, unsigned int flags) inode->i_data.a_ops = &fuse_dax_file_aops; } +void fuse_dax_dontcache (struct inode *inode,unsigned int flags) +{ +struct fuse_conn *fc =get_fuse_conn (inode); + +if (fc->dax_mode ==FUSE_DAX_INODE && + (!!IS_DAX (inode) != !!(flags &FUSE_ATTR_DAX ))) +d_mark_dontcache (inode); +} +bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment) {if (fc->dax && (map_alignment >FUSE_DAX_SHIFT )) {
fs/fuse/fuse_i.h +1 -0
@@ -1270,6 +1270,7 @@ void fuse_dax_conn_free(struct fuse_conn *fc);bool fuse_dax_inode_alloc (struct super_block *sb,struct fuse_inode *fi);void fuse_dax_inode_init (struct inode *inode,unsigned int flags);void fuse_dax_inode_cleanup (struct inode *inode); +void fuse_dax_dontcache (struct inode *inode,unsigned int flags);bool fuse_dax_check_alignment (struct fuse_conn *fc,unsigned int map_alignment);void fuse_dax_cancel_work (struct fuse_conn *fc);
fs/fuse/inode.c +3 -0
@@ -268,6 +268,9 @@ void fuse_change_attributes(struct inode *inode, struct fuse_attr *attr,if (inval)invalidate_inode_pages2 (inode->i_mapping); } + +if (IS_ENABLED (CONFIG_FUSE_DAX )) +fuse_dax_dontcache (inode, attr->flags); }static void fuse_init_inode (struct inode *inode,struct fuse_attr *attr)
Signature · 1 line
2.27.0
v5 cover virtiofsd: support per-file DAX
In reply to Jeffle Xuchanges since v4:
- decide whether DAX shall be enabled or not solely depending on file size (DAX is disabled for files smaller than 32KB)
- negotiation during FUSE_INIT is droped
- drop support for .ioctl() for passthrough
changes since v2/v3:
Patch 4 in v2 is incomplete by mistake and it will fail to be compiled. I had ever sent a seperate patch 4 of v3. Now I send the whole complete set in v4. Except for this, there's no other diferrence.
Jeffle Xu (2):
virtiofsd: add FUSE_ATTR_DAX to fuse protocol
virtiofsd: support per-file DAX
include/standard-headers/linux/fuse.h | 1 + tools/virtiofsd/passthrough_ll.c | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+)
Signature · 1 line
2.27.0
PATCH v5 1/2 virtiofsd: add FUSE_ATTR_DAX to fuse protocol
... in prep for the following support for per-file DAX feature.
--- include/standard-headers/linux/fuse.h | 1 + 1 file changed, 1 insertion(+)
include/standard-headers/linux/fuse.h +1 -0
@@ -440,6 +440,7 @@ struct fuse_file_lock {* FUSE_ATTR_SUBMOUNT: Object is a submount root */ #define FUSE_ATTR_SUBMOUNT (1 << 0) +#define FUSE_ATTR_DAX (1 << 1)/** * Open flags
Signature · 1 line
2.27.0
PATCH v5 2/2 virtiofsd: support per-file DAX
In reply to Jeffle XuWhen DAX window is fully utilized and needs to be exapnded to avoid the performance fluctuation triggered by DAX window recaliming, it may not be wise to allocate DAX window for files with size smaller than some specific point, considering from the perspective of reducing memory overhead.
To maintain one DAX window chunk (e.g., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Thus only flag the file with FUSE_ATTR_DAX when the file size is greater than 32 KB. The guest will enable DAX only for those files flagged with FUSE_ATTR_DAX, when virtiofs is mounted with '-o dax=inode'.
To be noted that both FUSE_LOOKUP and FUSE_READDIRPLUS are affected, and will convey FUSE_ATTR_DAX flag to the guest.
--- tools/virtiofsd/passthrough_ll.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
tools/virtiofsd/passthrough_ll.c +19 -0
@@ -53,12 +53,26 @@#include <sys/syscall.h> #include <sys/wait.h> #include <sys/xattr.h> +#include <sys/user.h> #include <syslog.h> #include "qemu/cutils.h" #include "passthrough_helpers.h" #include "passthrough_seccomp.h" +/* +* One page descriptor (64 bytes in size) needs to be maintained for every page +* in the DAX window chunk, i.e., there is certain guest memory overhead when +* DAX is enabled. Thus disable DAX for those files smaller than this certain +* memory overhead if virtiofs is mounted in per-file DAX mode, in which case +* the guest page cache will consume less memory when DAX is disabled. +*/ +#define FUSE_DAX_SHIFT 21 +#define PAGE_DESC_SHIFT 6 +#define FUSE_PERFILE_DAX_SHIFT \ + (FUSE_DAX_SHIFT - PAGE_SHIFT + PAGE_DESC_SHIFT) +#define FUSE_PERFILE_DAX_POINT (1 << FUSE_PERFILE_DAX_SHIFT) +/* Keep track of inode posix locks for each owner. */ struct lo_inode_plock {uint64_t lock_owner;
@@ -1023,6 +1037,11 @@ static int lo_do_lookup(fuse_req_t req, fuse_ino_t parent, const char *name, e->attr_flags |=FUSE_ATTR_SUBMOUNT ; } +if (S_ISREG (e->attr.st_mode) && + (e->attr.st_size >FUSE_PERFILE_DAX_POINT )) { + e->attr_flags |=FUSE_ATTR_DAX ; + } + inode =lo_find (lo, &e->attr, mnt_id);if (inode) {close (newfd);
Signature · 1 line
2.27.0
Vivek Goyal
maintainerQuoted 37 lines · Jeffle Xu
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
Any comment is welcome.
[1] commit
9cb20f94afcd("fs/ext4: Make DAX mount option a tri-state")
[2] commit02beb2686ff9("fs/xfs: Make DAX mount option a tri-state")[Purpose]
DAX may be limited in some specific situation. When the number of usable DAX windows is under watermark, the recalim routine will be triggered to reclaim some DAX windows. It may have a negative impact on the performance, since some processes may need to wait for DAX windows to be recalimed and reused then. To mitigate the performance degradation, the overall DAX window need to be expanded larger.However, simply expanding the DAX window may not be a good deal in some scenario. To maintain one DAX window chunk (i.e., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest, which is greater than the memory footprint if it uses guest page cache when DAX disabled. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Per-file DAX feature is introduced to address this issue, by offering a finer grained control for dax to users, trying to achieve a balance between performance and memory overhead.
[Note]
When the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state, since dynamic switching needs to switcha_opsatomiclly. Ira Weiny had ever implemented a so calledi_aops_semlock [3] but eventually gave up since the complexity of the implementation [4][5][6][7].
Quoted 5 lines
Hence mark the inode and corresponding dentries as
DONE_CACHEonce the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.
If we don't cache inode (if no fd is open), will it not have negative performance impact. When we cache inodes, we also have all the dax mappings cached as well. So if a process opens the same file again, it gets all the mappings already in place and it does not have to call FUSE_SETUPMAPPING again.
Vivek
Quoted 53 lines
In summary, when the per-file DAX hint changes for an *opened* file, the DAX state of the file won't be updated until this file is closed and reopened later. This is also how ext4/xfs per-file DAX works.
[3] https://lore.kernel.org/lkml/[email protected]/
[4] https://patchwork.kernel.org/project/xfs/cover/[email protected]/
[5] https://lore.kernel.org/lkml/[email protected]/
[6] https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/
[7] https://lore.kernel.org/lkml/20200403182904.GP80283@magnolia/chanegs since v4:
- drop support for setting/clearingFS_DAXinside guest
- and thus drop the negotiation phase duringFUSE_INITchanges since v3:
- bug fix (patch 6): s/"IS_DAX(inode) != newdax"/"!!IS_DAX(inode) != newdax"
- duringFUSE_INIT, advertise capability for per-file DAX only when mounted as "-o dax=inode" (patch 4)changes since v2:
- modifyfuse_show_options()accordingly to make it compatible with new tri-state mount option (patch 2)
- extract FUSE protocol changes into one seperate patch (patch 3)
- FUSE server/client need to negotiate if they support per-file DAX (patch 4)
- extractDONT_CACHElogic into patch 6/7v4: https://lore.kernel.org/linux-fsdevel/[email protected]/
v3: https://www.spinics.net/lists/linux-fsdevel/msg200852.html
v2: https://www.spinics.net/lists/linux-fsdevel/msg199584.html
v1: https://www.spinics.net/lists/linux-virtualization/msg51008.htmlJeffle Xu (5):
fuse: addfuse_should_enable_dax()helper
fuse: make DAX mount option a tri-state
fuse: support per-file DAX
fuse: enable per-file DAX
fuse: mark inodeDONT_CACHEwhen per-file DAX hint changesfs/fuse/dax.c | 32 +++++++++++++++++++++++++++++--- fs/fuse/file.c | 4 ++-- fs/fuse/fuse_i.h | 19 +++++++++++++++---- fs/fuse/inode.c | 15 +++++++++++---- fs/fuse/virtio_fs.c | 16 ++++++++++++++-- include/uapi/linux/fuse.h | 7 ++++++- 6 files changed, 77 insertions(+), 16 deletions(-)Signature · 1 line
2.27.0
Sorry for the late reply, as your previous reply was moved to junk box by the algorithm...
Quoted 44 lines · Vivek Goyal
Quoted 37 lines · Jeffle Xu
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
Any comment is welcome.
[1] commit
9cb20f94afcd("fs/ext4: Make DAX mount option a tri-state")
[2] commit02beb2686ff9("fs/xfs: Make DAX mount option a tri-state")[Purpose]
DAX may be limited in some specific situation. When the number of usable DAX windows is under watermark, the recalim routine will be triggered to reclaim some DAX windows. It may have a negative impact on the performance, since some processes may need to wait for DAX windows to be recalimed and reused then. To mitigate the performance degradation, the overall DAX window need to be expanded larger.However, simply expanding the DAX window may not be a good deal in some scenario. To maintain one DAX window chunk (i.e., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest, which is greater than the memory footprint if it uses guest page cache when DAX disabled. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Per-file DAX feature is introduced to address this issue, by offering a finer grained control for dax to users, trying to achieve a balance between performance and memory overhead.
[Note]
When the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state, since dynamic switching needs to switcha_opsatomiclly. Ira Weiny had ever implemented a so calledi_aops_semlock [3] but eventually gave up since the complexity of the implementation [4][5][6][7].Quoted 5 lines
Hence mark the inode and corresponding dentries as
DONE_CACHEonce the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.
Quoted 6 lines
If we don't cache inode (if no fd is open), will it not have negative performance impact. When we cache inodes, we also have all the dax mappings cached as well. So if a process opens the same file again, it gets all the mappings already in place and it does not have to call
FUSE_SETUPMAPPINGagain.
What does 'all the dax mappings cached' mean when 'we cache inodes'?
If the per-file DAX hint indeed changes for a large sized file, with quite many page caches or DAX mapping already in the address space, then marking it DONT_CACHE means evicting the inode as soon as possible, which means flushing the page caches or removing all DAX mappings. When the inode is reopened next time, page cache is re-instantiated or FUSE_SETUPMAPPING is called again. Then the negative performance impact indeed exist in this case.
But this performance impact only exist when the per-file DAX hint changes halfway, that is, the hint suddenly changes after the virtiofs has already mounted in the guest.
Signature · 2 lines
Thanks, Jeffle
Vivek Goyal
maintainerQuoted 66 lines · JeffleXu
Sorry for the late reply, as your previous reply was moved to junk box by the algorithm...
Quoted 44 lines · Vivek Goyal
Quoted 37 lines · Jeffle Xu
This patchset adds support of per-file DAX for virtiofs, which is inspired by Ira Weiny's work on ext4[1] and xfs[2].
Any comment is welcome.
[1] commit
9cb20f94afcd("fs/ext4: Make DAX mount option a tri-state")
[2] commit02beb2686ff9("fs/xfs: Make DAX mount option a tri-state")[Purpose]
DAX may be limited in some specific situation. When the number of usable DAX windows is under watermark, the recalim routine will be triggered to reclaim some DAX windows. It may have a negative impact on the performance, since some processes may need to wait for DAX windows to be recalimed and reused then. To mitigate the performance degradation, the overall DAX window need to be expanded larger.However, simply expanding the DAX window may not be a good deal in some scenario. To maintain one DAX window chunk (i.e., 2MB in size), 32KB (512 * 64 bytes) memory footprint will be consumed for page descriptors inside guest, which is greater than the memory footprint if it uses guest page cache when DAX disabled. Thus it'd better disable DAX for those files smaller than 32KB, to reduce the demand for DAX window and thus avoid the unworthy memory overhead.
Per-file DAX feature is introduced to address this issue, by offering a finer grained control for dax to users, trying to achieve a balance between performance and memory overhead.
[Note]
When the per-file DAX hint changes while the file is still *opened*, it is quite complicated and maybe fragile to dynamically change the DAX state, since dynamic switching needs to switcha_opsatomiclly. Ira Weiny had ever implemented a so calledi_aops_semlock [3] but eventually gave up since the complexity of the implementation [4][5][6][7].Quoted 5 lines
Hence mark the inode and corresponding dentries as
DONE_CACHEonce the per-file DAX hint changes, so that the inode instance will be evicted and freed as soon as possible once the file is closed and the last reference to the inode is put. And then when the file gets reopened next time, the new instantiated inode will reflect the new DAX state.Quoted 6 lines
If we don't cache inode (if no fd is open), will it not have negative performance impact. When we cache inodes, we also have all the dax mappings cached as well. So if a process opens the same file again, it gets all the mappings already in place and it does not have to call
FUSE_SETUPMAPPINGagain.What does 'all the dax mappings cached' mean when 'we cache inodes'?
If the per-file DAX hint indeed changes for a large sized file, with quite many page caches or DAX mapping already in the address space, then marking it
DONT_CACHEmeans evicting the inode as soon as possible, which means flushing the page caches or removing all DAX mappings. When the inode is reopened next time, page cache is re-instantiated orFUSE_SETUPMAPPINGis called again. Then the negative performance impact indeed exist in this case.
Quoted 3 lines
But this performance impact only exist when the per-file DAX hint changes halfway, that is, the hint suddenly changes after the virtiofs has already mounted in the guest.
Ok, got it. I think I saw that in the code. I had assumed that an inode will always be marked don't cache. That's not the case. It will be marked don't cache only if inode property changes (from dax to non-dax or vice-a-versa). That seems fine.
Vivek