Skip to contentSkip to conversation

viriofsd: do not fall back to buffer io when cache=auto

RFC PATCH In review piaojun 3 participants 18 messages Aug 14, 2019 – Aug 23, 2019

lore
Share & tools
lore

piaojun

author
#0 lore raw

RFC PATCH 1/1 viriofsd: do not fall back to buffer io when cache=auto

Direct io flags will be tured off even if cache=auto, which seems a little bit strange. It's better to keep the open flags set by user. If I missed the discussion about this issue, please let me know, thanks.

---
 contrib/virtiofsd/passthrough_ll.c | 4 ++++
 1 file changed, 4 insertions(+)
contrib/virtiofsd/passthrough_ll.c +4 -0
index 321bbb2..a080f0d 100644
@@ -1687,6 +1687,8 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 
 out:
 	lo_inode_put(lo, &parent_inode);
@@ -1899,6 +1901,8 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 	fuse_reply_open(req, fi);
 }
 
author
#1 lore raw

Ping, any comments on this patch?

Quoted 31 lines · piaojun

Direct io flags will be tured off even if cache=auto, which seems a little bit strange. It's better to keep the open flags set by user. If I missed the discussion about this issue, please let me know, thanks.

---
 contrib/virtiofsd/passthrough_ll.c | 4 ++++
 1 file changed, 4 insertions(+)
contrib/virtiofsd/passthrough_ll.c +4 -0
index 321bbb2..a080f0d 100644
@@ -1687,6 +1687,8 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 
 out:
 	lo_inode_put(lo, &parent_inode);
@@ -1899,6 +1901,8 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 	fuse_reply_open(req, fi);
 }
 
#2 lore raw
In reply to piaojun
Quoted 23 lines · piaojun

Direct io flags will be tured off even if cache=auto, which seems a little bit strange. It's better to keep the open flags set by user. If I missed the discussion about this issue, please let me know, thanks.

---
 contrib/virtiofsd/passthrough_ll.c | 4 ++++
 1 file changed, 4 insertions(+)
contrib/virtiofsd/passthrough_ll.c +2 -0
index 321bbb2..a080f0d 100644
@@ -1687,6 +1687,8 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 
 out:
 	lo_inode_put(lo, &parent_inode);

contrib/virtiofsd/passthrough_ll.c static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi) · line 1906

Quoted 6 lines
contrib/virtiofsd/passthrough_ll.c +2 -0
@@ -1899,6 +1901,8 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 	fuse_reply_open(req, fi);

IMHO, this is necessary, it seems that in the case cache=auto/always, virtiofsd already follows what guest requests, i.e. virtiofsd only does direct I/O when guest opens file with O_DIRECT.

If I understand the code correctly, fi->direct_io = 1 is used to force guest to use direct I/O even if guest opens file without O_DIRECT.

Thanks,
Eryu

Quoted 8 lines
contrib/virtiofsd/passthrough_ll.c +0 -0
 }
 
Signature · 4 lines
_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs
author
#3 lore raw
Quoted 35 lines · Eryu Guan
Quoted 23 lines · piaojun

Direct io flags will be tured off even if cache=auto, which seems a little bit strange. It's better to keep the open flags set by user. If I missed the discussion about this issue, please let me know, thanks.

---
 contrib/virtiofsd/passthrough_ll.c | 4 ++++
 1 file changed, 4 insertions(+)
contrib/virtiofsd/passthrough_ll.c +2 -0
index 321bbb2..a080f0d 100644
@@ -1687,6 +1687,8 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 
 out:
 	lo_inode_put(lo, &parent_inode);

contrib/virtiofsd/passthrough_ll.c static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi) · line 1906

Quoted 6 lines
contrib/virtiofsd/passthrough_ll.c +2 -0
@@ -1899,6 +1901,8 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 	fuse_reply_open(req, fi);

IMHO, this is necessary, it seems that in the case cache=auto/always, virtiofsd already follows what guest requests, i.e. virtiofsd only does direct I/O when guest opens file with O_DIRECT.

Quoted 1 line

From my testing, virtiofsd won't do direct io even if guest opens file

with O_DIRECT when cache=auto/alway. It's weird to convert direct io to buffer io at host side.

Quoted 3 lines

If I understand the code correctly, fi->direct_io = 1 is used to force guest to use direct I/O even if guest opens file without O_DIRECT.

But in fact, direct I/O will be forced to buffer I/O even if with O_DIRECT. I will look into this problem by further testing, and very glad to share my test result.

Jun

#4 lore raw
Quoted 24 lines · piaojun
Quoted 21 lines · Eryu Guan
Quoted 20 lines · piaojun

Direct io flags will be tured off even if cache=auto, which seems a little bit strange. It's better to keep the open flags set by user. If I missed the discussion about this issue, please let me know, thanks.

---
 contrib/virtiofsd/passthrough_ll.c | 4 ++++
 1 file changed, 4 insertions(+)
contrib/virtiofsd/passthrough_ll.c +2 -0
index 321bbb2..a080f0d 100644
@@ -1687,6 +1687,8 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 
Quoted 11 lines
Quoted 11 lines

contrib/virtiofsd/passthrough_ll.c static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi) · line 1906

Quoted 9 lines
contrib/virtiofsd/passthrough_ll.c +2 -0
 out:
 	lo_inode_put(lo, &parent_inode);
@@ -1899,6 +1901,8 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 	fuse_reply_open(req, fi);

IMHO, this is necessary, it seems that in the case cache=auto/always,

Huge typo here, I meant "not necessary" here..

Quoted 6 lines
Quoted 2 lines

virtiofsd already follows what guest requests, i.e. virtiofsd only does direct I/O when guest opens file with O_DIRECT.

From my testing, virtiofsd won't do direct io even if guest opens file with O_DIRECT when cache=auto/alway. It's weird to convert direct io to buffer io at host side.

That doesn't match what's in my mind, I think FUSE_OPEN will pass guest open flags to virtiofsd and virtiofsd saves the fd (which is opened with O_DIRECT) in fd map, then subsequent FUSE_WRITE does direct write to that fd.

Quoted 8 lines
Quoted 4 lines

If I understand the code correctly, fi->direct_io = 1 is used to force guest to use direct I/O even if guest opens file without O_DIRECT.

But in fact, direct I/O will be forced to buffer I/O even if with O_DIRECT. I will look into this problem by further testing, and very glad to share my test result.

Thanks!

Eryu

author
#5 lore raw
Quoted 42 lines · Eryu Guan
Quoted 24 lines · piaojun
Quoted 21 lines · Eryu Guan
Quoted 20 lines · piaojun

Direct io flags will be tured off even if cache=auto, which seems a little bit strange. It's better to keep the open flags set by user. If I missed the discussion about this issue, please let me know, thanks.

---
 contrib/virtiofsd/passthrough_ll.c | 4 ++++
 1 file changed, 4 insertions(+)
contrib/virtiofsd/passthrough_ll.c +2 -0
index 321bbb2..a080f0d 100644
@@ -1687,6 +1687,8 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 
Quoted 11 lines
Quoted 11 lines

contrib/virtiofsd/passthrough_ll.c static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi) · line 1906

Quoted 9 lines
contrib/virtiofsd/passthrough_ll.c +2 -0
 out:
 	lo_inode_put(lo, &parent_inode);
@@ -1899,6 +1901,8 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 	fuse_reply_open(req, fi);

IMHO, this is necessary, it seems that in the case cache=auto/always,

Huge typo here, I meant "not necessary" here..

Quoted 3 lines
Quoted 2 lines

virtiofsd already follows what guest requests, i.e. virtiofsd only does direct I/O when guest opens file with O_DIRECT.

Quoted 8 lines
Quoted 3 lines

From my testing, virtiofsd won't do direct io even if guest opens file with O_DIRECT when cache=auto/alway. It's weird to convert direct io to buffer io at host side.

That doesn't match what's in my mind, I think FUSE_OPEN will pass guest open flags to virtiofsd and virtiofsd saves the fd (which is opened with O_DIRECT) in fd map, then subsequent FUSE_WRITE does direct write to that fd.

Yes, I agree with this. And my comment was a little confuing, so I have to resend the patch later to make it more clear.

Jun

author
#6 lore raw

RFC PATCH 1/1 virtiofsd: do not fall back to buffer io when cache=auto/always

In reply to piaojun

When O_DIRECT flags is set by Guest, virtiofsd will open file with O_DIRECT, but unset 'fi->direct_io' which makes Guest go buffer io path. That causes inconsistency between Guest and Host.

The cache option in virtiofsd should not affect the file io mode, so set 'fi->direct_io' according to 'fi->flags'.

---
 contrib/virtiofsd/passthrough_ll.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
contrib/virtiofsd/passthrough_ll.c +7 -2
index ca11764..a9c98d0 100644
@@ -1774,8 +1774,10 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->fh = fh;
 		err = lo_do_lookup(req, parent, name, &e);
 	}
-	if (lo->cache == CACHE_NONE)
+	if (fi->flags & O_DIRECT)
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
 
@@ -1982,8 +1984,11 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 	}
 
 	fi->fh = fh;
-	if (lo->cache == CACHE_NONE)
+
+	if (fi->flags & O_DIRECT)
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
 	fuse_reply_open(req, fi);
#7 lore raw

Re: [Virtio-fs] [PATCH RESEND][RFC] virtiofsd: do not fall back to buffer io when cache=auto/always

Quoted 17 lines · piaojun

When O_DIRECT flags is set by Guest, virtiofsd will open file with O_DIRECT, but unset 'fi->direct_io' which makes Guest go buffer io path. That causes inconsistency between Guest and Host.

The cache option in virtiofsd should not affect the file io mode, so set 'fi->direct_io' according to 'fi->flags'.

---
 contrib/virtiofsd/passthrough_ll.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
contrib/virtiofsd/passthrough_ll.c +0 -0
index ca11764..a9c98d0 100644

contrib/virtiofsd/passthrough_ll.c static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name, · line 1777

Quoted 5 lines
contrib/virtiofsd/passthrough_ll.c +1 -1
@@ -1774,8 +1774,10 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->fh = fh;
 		err = lo_do_lookup(req, parent, name, &e);
 	}
-	if (lo->cache == CACHE_NONE)
+	if (fi->flags & O_DIRECT)

Looks like this changes cache=none behavior. Previously, we set direct_io when cache=none and guest will go to dio path even when the file is opened for buffer io.

But with this change, guest could do buffer io in cache=none case.

contrib/virtiofsd/passthrough_ll.c static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name, · line 1780

Quoted 3 lines
contrib/virtiofsd/passthrough_ll.c +2 -0
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;

And this only causes guest to invalidate page cache at open(2) time, but still go buffer io path on cache=none.

contrib/virtiofsd/passthrough_ll.c static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name, · line 1782

Quoted 2 lines
contrib/virtiofsd/passthrough_ll.c +0 -0
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;

I'm thinking about something like (totally untested):

contrib/virtiofsd/passthrough_ll.c +2 -2
@@ -1774,7 +1774,7 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
                fi->fh = fh;
                err = lo_do_lookup(req, parent, name, &e);
        }
-       if (lo->cache == CACHE_NONE)
+       if (lo->cache == CACHE_NONE || fi->flags & O_DIRECT)
                fi->direct_io = 1;
        else if (lo->cache == CACHE_ALWAYS)
                fi->keep_cache = 1;
@@ -1982,7 +1982,7 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
        }
 
        fi->fh = fh;
-       if (lo->cache == CACHE_NONE)
+       if (lo->cache == CACHE_NONE || fi->flags & O_DIRECT)
                fi->direct_io = 1;
        else if (lo->cache == CACHE_ALWAYS)
                fi->keep_cache = 1;

contrib/virtiofsd/passthrough_ll.c static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi) · line 1993

Quoted 13 lines
contrib/virtiofsd/passthrough_ll.c +4 -1
@@ -1982,8 +1984,11 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 	}
 
 	fi->fh = fh;
-	if (lo->cache == CACHE_NONE)
+
+	if (fi->flags & O_DIRECT)
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;

And this code snippet duplicates with lo_create(), maybe we could factor it into a new helper function.

Thanks,
Eryu

author
#8 lore raw

Re: [Virtio-fs] [PATCH RESEND][RFC] virtiofsd: do not fall back to buffer io when cache=auto/always

Quoted 28 lines · Eryu Guan
Quoted 17 lines · piaojun

When O_DIRECT flags is set by Guest, virtiofsd will open file with O_DIRECT, but unset 'fi->direct_io' which makes Guest go buffer io path. That causes inconsistency between Guest and Host.

The cache option in virtiofsd should not affect the file io mode, so set 'fi->direct_io' according to 'fi->flags'.

---
 contrib/virtiofsd/passthrough_ll.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
contrib/virtiofsd/passthrough_ll.c +0 -0
index ca11764..a9c98d0 100644

contrib/virtiofsd/passthrough_ll.c static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name, · line 1777

Quoted 5 lines
contrib/virtiofsd/passthrough_ll.c +1 -1
@@ -1774,8 +1774,10 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->fh = fh;
 		err = lo_do_lookup(req, parent, name, &e);
 	}
-	if (lo->cache == CACHE_NONE)
+	if (fi->flags & O_DIRECT)

Looks like this changes cache=none behavior. Previously, we set direct_io when cache=none and guest will go to dio path even when the file is opened for buffer io.

Quoted 2 lines

But with this change, guest could do buffer io in cache=none case.

IMO, the cache option is only used for determining whether using the last inode's cache when do open(). The user could choose direct or buffer io no matter if the cache exists.

Moreover, the cache option only works for the first time buffer io, and has little effect on subsequent ones.

Quoted 6 lines

contrib/virtiofsd/passthrough_ll.c static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name, · line 1780

Quoted 3 lines
contrib/virtiofsd/passthrough_ll.c +2 -0
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;

And this only causes guest to invalidate page cache at open(2) time, but still go buffer io path on cache=none.

As my comments above, io path has nothing to do with inode's current page cache.

Quoted 18 lines

contrib/virtiofsd/passthrough_ll.c static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name, · line 1782

Quoted 3 lines
contrib/virtiofsd/passthrough_ll.c +0 -0
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;

I'm thinking about something like (totally untested):

contrib/virtiofsd/passthrough_ll.c +1 -1
@@ -1774,7 +1774,7 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
                fi->fh = fh;
                err = lo_do_lookup(req, parent, name, &e);
        }
-       if (lo->cache == CACHE_NONE)
+       if (lo->cache == CACHE_NONE || fi->flags & O_DIRECT)
                fi->direct_io = 1;
        else if (lo->cache == CACHE_ALWAYS)
                fi->keep_cache = 1;

contrib/virtiofsd/passthrough_ll.c static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi) · line 1988

Quoted 8 lines
contrib/virtiofsd/passthrough_ll.c +1 -1
@@ -1982,7 +1982,7 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
        }
 
        fi->fh = fh;
-       if (lo->cache == CACHE_NONE)
+       if (lo->cache == CACHE_NONE || fi->flags & O_DIRECT)
                fi->direct_io = 1;
        else if (lo->cache == CACHE_ALWAYS)
                fi->keep_cache = 1;

This will mix cache option with io flags which seems weird.

Quoted 5 lines
Quoted 5 lines
contrib/virtiofsd/passthrough_ll.c +0 -0
@@ -1982,8 +1984,11 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 	}
 
Quoted 12 lines

contrib/virtiofsd/passthrough_ll.c static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name, · line 1780

Quoted 9 lines
contrib/virtiofsd/passthrough_ll.c +4 -1
 	fi->fh = fh;
-	if (lo->cache == CACHE_NONE)
+
+	if (fi->flags & O_DIRECT)
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;

And this code snippet duplicates with lo_create(), maybe we could factor it into a new helper function.

It makes sense and I will consider adding a helper.

Jun

#17 lore raw
In reply to piaojun
Quoted 3 lines · piaojun

Direct io flags will be tured off even if cache=auto, which seems a little bit strange. It's better to keep the open flags set by user. If I missed the discussion about this issue, please let me know, thanks.

fi->direct_io and fi->keep_cache are controlling hints for guest kernel. So for cache=none, we want to enforce direct I/O in guest hence we set fi->direct_io (FOPEN_DIRECT_IO) in reply to guest.

For cache=always, we want to cache data and metadata hence we don't set fi->direct_io. Also, we don't want to invalidate current cache contents of file in guest hence we set fi->keep_cache (FOPEN_KEEP_CACHE)

For cache=auto, want guest kernel to cache data and metadata hence we don't set fi->direct_io. We want guest kernel to invalidate data on next open hence we don't set fi->keep_cache.

If guest is doing direct IO on file and fi->flags has O_DIRECT set, that should not do anything to fi->direct_io. That's a hint for guest kernel enforced by virtiofsd.

I am not sure what problem you are trying to solve with this patch.

Thanks
Vivek

Quoted 34 lines
---
 contrib/virtiofsd/passthrough_ll.c | 4 ++++
 1 file changed, 4 insertions(+)
contrib/virtiofsd/passthrough_ll.c +4 -0
index 321bbb2..a080f0d 100644
@@ -1687,6 +1687,8 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 
 out:
 	lo_inode_put(lo, &parent_inode);
@@ -1899,6 +1901,8 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 		fi->direct_io = 1;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
+	else
+		fi->direct_io = !!(fi->flags & O_DIRECT);
 	fuse_reply_open(req, fi);
 }
 
Signature · 4 lines
_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs
#9 lore raw

Re: [Virtio-fs] [PATCH RESEND][RFC] virtiofsd: do not fall back to buffer io when cache=auto/always

In reply to piaojun
Quoted 2 lines · piaojun

When O_DIRECT flags is set by Guest, virtiofsd will open file with O_DIRECT, but unset 'fi->direct_io' which makes Guest go buffer io path.

Where is virtiofsd unset fi->direct_io flag? All I can see is that it sets this flag if cache=none?

Thanks
Vivek

Quoted 45 lines

That causes inconsistency between Guest and Host.

The cache option in virtiofsd should not affect the file io mode, so set 'fi->direct_io' according to 'fi->flags'.

---
 contrib/virtiofsd/passthrough_ll.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
contrib/virtiofsd/passthrough_ll.c +7 -2
index ca11764..a9c98d0 100644
@@ -1774,8 +1774,10 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->fh = fh;
 		err = lo_do_lookup(req, parent, name, &e);
 	}
-	if (lo->cache == CACHE_NONE)
+	if (fi->flags & O_DIRECT)
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
 
@@ -1982,8 +1984,11 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 	}
 
 	fi->fh = fh;
-	if (lo->cache == CACHE_NONE)
+
+	if (fi->flags & O_DIRECT)
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
 	fuse_reply_open(req, fi);
Signature · 4 lines
_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs
#10 lore raw

Re: [Virtio-fs] [PATCH RESEND][RFC] virtiofsd: do not fall back to buffer io when cache=auto/always

Quoted 7 lines · Vivek Goyal
Quoted 2 lines · piaojun

When O_DIRECT flags is set by Guest, virtiofsd will open file with O_DIRECT, but unset 'fi->direct_io' which makes Guest go buffer io path.

Where is virtiofsd unset fi->direct_io flag? All I can see is that it sets this flag if cache=none?

Ok, I looked at it little bit more closely and I don't think it is a problem. With cache=always/auto, we don't set fi->direct_io. That means server is not enforcing a direct I/O policy on client.

Now if client decides to direct I/O by opening file with O_DIRECT, it will still take direct I/O path and call fuse_direct_IO(). Look at fuse_file_aops->fuse_direct_IO().

Thanks
Vivek

Quoted 48 lines

Thanks
Vivek

Quoted 45 lines

That causes inconsistency between Guest and Host.

The cache option in virtiofsd should not affect the file io mode, so set 'fi->direct_io' according to 'fi->flags'.

---
 contrib/virtiofsd/passthrough_ll.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
contrib/virtiofsd/passthrough_ll.c +7 -2
index ca11764..a9c98d0 100644
@@ -1774,8 +1774,10 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->fh = fh;
 		err = lo_do_lookup(req, parent, name, &e);
 	}
-	if (lo->cache == CACHE_NONE)
+	if (fi->flags & O_DIRECT)
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
 
@@ -1982,8 +1984,11 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 	}
 
 	fi->fh = fh;
-	if (lo->cache == CACHE_NONE)
+
+	if (fi->flags & O_DIRECT)
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
 	fuse_reply_open(req, fi);
Signature · 4 lines
_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs
author
#11 lore raw

Re: [Virtio-fs] [PATCH RESEND][RFC] virtiofsd: do not fall back to buffer io when cache=auto/always

Quoted 9 lines · Vivek Goyal
Quoted 7 lines · Vivek Goyal
Quoted 2 lines · piaojun

When O_DIRECT flags is set by Guest, virtiofsd will open file with O_DIRECT, but unset 'fi->direct_io' which makes Guest go buffer io path.

Where is virtiofsd unset fi->direct_io flag? All I can see is that it sets this flag if cache=none?

Quoted 7 lines

Ok, I looked at it little bit more closely and I don't think it is a problem. With cache=always/auto, we don't set fi->direct_io. That means server is not enforcing a direct I/O policy on client.

Now if client decides to direct I/O by opening file with O_DIRECT, it will still take direct I/O path and call fuse_direct_IO(). Look at fuse_file_aops->fuse_direct_IO().

The problem is that when Guest open with O_DIRECT, the Host won't set open_flags with FOPEN_DIRECT_IO if CACHE_AUTO is set. It makes Guest goes into fuse_cache_write_iter() rather than fuse_direct_write_iter(). In other words, the Guest's I/O mode will be changed by virtiofsd.

You cleared my first doubt that 'fi->direct_io' and 'fi->keep_cache' are used to control Guest I/O mode together, and there may be different I/O modes between Guest and Host if I undersatnd it correctly. I admit this offers a more flexible config to user, but also introduces more complex option. Maybe we could just use *keep_cache* to handle all these options.

My second doubt is that *keep_cache* is used to control whether reusing the cache of last open(). IMO, it has nothing to do with the following I/O path.

Jun

Quoted 54 lines

Thanks
Vivek

Quoted 44 lines

Thanks
Vivek

Quoted 41 lines

That causes inconsistency between Guest and Host.

The cache option in virtiofsd should not affect the file io mode, so set 'fi->direct_io' according to 'fi->flags'.

---
 contrib/virtiofsd/passthrough_ll.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
contrib/virtiofsd/passthrough_ll.c +7 -2
index ca11764..a9c98d0 100644
@@ -1774,8 +1774,10 @@ static void lo_create(fuse_req_t req, fuse_ino_t parent, const char *name,
 		fi->fh = fh;
 		err = lo_do_lookup(req, parent, name, &e);
 	}
-	if (lo->cache == CACHE_NONE)
+	if (fi->flags & O_DIRECT)
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
 
@@ -1982,8 +1984,11 @@ static void lo_open(fuse_req_t req, fuse_ino_t ino, struct fuse_file_info *fi)
 	}
 
 	fi->fh = fh;
-	if (lo->cache == CACHE_NONE)
+
+	if (fi->flags & O_DIRECT)
 		fi->direct_io = 1;
+	if (lo->cache == CACHE_NONE)
+		fi->keep_cache = 0;
 	else if (lo->cache == CACHE_ALWAYS)
 		fi->keep_cache = 1;
 	fuse_reply_open(req, fi);
Quoted 4 lines
Quoted 4 lines

_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs

.

#12 lore raw

Re: [Virtio-fs] [PATCH RESEND][RFC] virtiofsd: do not fall back to buffer io when cache=auto/always

Quoted 16 lines · piaojun
Quoted 13 lines · Vivek Goyal
Quoted 7 lines · Vivek Goyal
Quoted 2 lines · piaojun

When O_DIRECT flags is set by Guest, virtiofsd will open file with O_DIRECT, but unset 'fi->direct_io' which makes Guest go buffer io path.

Where is virtiofsd unset fi->direct_io flag? All I can see is that it sets this flag if cache=none?

Ok, I looked at it little bit more closely and I don't think it is a problem. With cache=always/auto, we don't set fi->direct_io. That means server is not enforcing a direct I/O policy on client.

Quoted 8 lines
Quoted 3 lines

Now if client decides to direct I/O by opening file with O_DIRECT, it will still take direct I/O path and call fuse_direct_IO(). Look at fuse_file_aops->fuse_direct_IO().

The problem is that when Guest open with O_DIRECT, the Host won't set open_flags with FOPEN_DIRECT_IO if CACHE_AUTO is set. It makes Guest goes into fuse_cache_write_iter() rather than fuse_direct_write_iter(). In other words, the Guest's I/O mode will be changed by virtiofsd.

I think there are two direct I/O paths. One is where server requests that direct I/O be done on the file and in that case FOPEN_DIRECT_IO is set.

Other case is where client initiates direct I/O and in that case FOPEN_DIRECT_IO is not set and we take regular path which is capable of doing both direct and buffered I/O. For example, for reads we do following.

fuse_file_read_iter()
fuse_cache_read_iter()

      generic_file_read_iter()
	mapping->a_ops->direct_IO() <--- (If iocb->ki_flags & IOCB_DIRECT)
	  fuse_direct_IO()

Similary for write path.

fuse_file_write_iter()
fuse_cache_write_iter()

     generic_file_direct_write() <----- (If iocb->ki_flags & IOCB_DIRECT)
        mapping->a_ops->direct_IO()
	   fuse_direct_IO()

Can you give it a try. I don't think we have the problem what you are describing.

Quoted 11 lines

You cleared my first doubt that 'fi->direct_io' and 'fi->keep_cache' are used to control Guest I/O mode together, and there may be different I/O modes between Guest and Host if I undersatnd it correctly. I admit this offers a more flexible config to user, but also introduces more complex option. Maybe we could just use *keep_cache* to handle all these options.

My second doubt is that *keep_cache* is used to control whether reusing the cache of last open(). IMO, it has nothing to do with the following I/O path.

FOPEN_KEEP_CACHE is already part of fuse protocol. cache=always is just making use of that option so that cache contents are not discarded upon next open.

Thanks
Vivek

author
#13 lore raw

Re: [Virtio-fs] [PATCH RESEND][RFC] virtiofsd: do not fall back to buffer io when cache=auto/always

Quoted 43 lines · Vivek Goyal
Quoted 16 lines · piaojun
Quoted 13 lines · Vivek Goyal
Quoted 7 lines · Vivek Goyal
Quoted 2 lines · piaojun

When O_DIRECT flags is set by Guest, virtiofsd will open file with O_DIRECT, but unset 'fi->direct_io' which makes Guest go buffer io path.

Where is virtiofsd unset fi->direct_io flag? All I can see is that it sets this flag if cache=none?

Ok, I looked at it little bit more closely and I don't think it is a problem. With cache=always/auto, we don't set fi->direct_io. That means server is not enforcing a direct I/O policy on client.

Quoted 8 lines
Quoted 3 lines

Now if client decides to direct I/O by opening file with O_DIRECT, it will still take direct I/O path and call fuse_direct_IO(). Look at fuse_file_aops->fuse_direct_IO().

The problem is that when Guest open with O_DIRECT, the Host won't set open_flags with FOPEN_DIRECT_IO if CACHE_AUTO is set. It makes Guest goes into fuse_cache_write_iter() rather than fuse_direct_write_iter(). In other words, the Guest's I/O mode will be changed by virtiofsd.

I think there are two direct I/O paths. One is where server requests that direct I/O be done on the file and in that case FOPEN_DIRECT_IO is set.

Other case is where client initiates direct I/O and in that case FOPEN_DIRECT_IO is not set and we take regular path which is capable of doing both direct and buffered I/O. For example, for reads we do following.

fuse_file_read_iter()
fuse_cache_read_iter()

      generic_file_read_iter()
	mapping->a_ops->direct_IO() <--- (If iocb->ki_flags & IOCB_DIRECT)
	  fuse_direct_IO()

Similary for write path.

Quoted 8 lines

fuse_file_write_iter()
fuse_cache_write_iter()

     generic_file_direct_write() <----- (If iocb->ki_flags & IOCB_DIRECT)
        mapping->a_ops->direct_IO()
	   fuse_direct_IO()

Can you give it a try. I don't think we have the problem what you are describing.

Thanks for your detailed explanation, and perhaps I missed the regular path you mentioned above, so I will have a try to confirm this later.

Quoted 9 lines
Quoted 9 lines

You cleared my first doubt that 'fi->direct_io' and 'fi->keep_cache' are used to control Guest I/O mode together, and there may be different I/O modes between Guest and Host if I undersatnd it correctly. I admit this offers a more flexible config to user, but also introduces more complex option. Maybe we could just use *keep_cache* to handle all these options.

Quoted 7 lines
Quoted 3 lines

My second doubt is that *keep_cache* is used to control whether reusing the cache of last open(). IMO, it has nothing to do with the following I/O path.

FOPEN_KEEP_CACHE is already part of fuse protocol. cache=always is just making use of that option so that cache contents are not discarded upon next open.

It still confused me that probably we needn't set fi->direct_io if CACHE_NONE is set to indicate Guest go direct I/O path. As Guest could also go direct I/O through regular path you have mentioned above.

Jun

#14 lore raw

Re: [Virtio-fs] [PATCH RESEND][RFC] virtiofsd: do not fall back to buffer io when cache=auto/always

On Wed, Aug 21, 2019 at 10:38:39PM +0800, piaojun wrote:

[..]

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

You cleared my first doubt that 'fi->direct_io' and 'fi->keep_cache' are used to control Guest I/O mode together, and there may be different I/O modes between Guest and Host if I undersatnd it correctly. I admit this offers a more flexible config to user, but also introduces more complex option. Maybe we could just use *keep_cache* to handle all these options.

My second doubt is that *keep_cache* is used to control whether reusing the cache of last open(). IMO, it has nothing to do with the following I/O path.

Quoted 7 lines
Quoted 3 lines

FOPEN_KEEP_CACHE is already part of fuse protocol. cache=always is just making use of that option so that cache contents are not discarded upon next open.

It still confused me that probably we needn't set fi->direct_io if CACHE_NONE is set to indicate Guest go direct I/O path. As Guest could also go direct I/O through regular path you have mentioned above.

fi->direct_io is needed so that server can *enforce* direct I/O on guest. If guest applications are not doing direct I/O, without fi->direct_io, you can't force client to do direct I/O.

Vivek

author
#15 lore raw

Re: [Virtio-fs] [PATCH RESEND][RFC] virtiofsd: do not fall back to buffer io when cache=auto/always

Quoted 19 lines · Vivek Goyal

On Wed, Aug 21, 2019 at 10:38:39PM +0800, piaojun wrote:

[..]

Quoted 16 lines
Quoted 15 lines
Quoted 11 lines

You cleared my first doubt that 'fi->direct_io' and 'fi->keep_cache' are used to control Guest I/O mode together, and there may be different I/O modes between Guest and Host if I undersatnd it correctly. I admit this offers a more flexible config to user, but also introduces more complex option. Maybe we could just use *keep_cache* to handle all these options.

My second doubt is that *keep_cache* is used to control whether reusing the cache of last open(). IMO, it has nothing to do with the following I/O path.

FOPEN_KEEP_CACHE is already part of fuse protocol. cache=always is just making use of that option so that cache contents are not discarded upon next open.

Quoted 7 lines
Quoted 3 lines

It still confused me that probably we needn't set fi->direct_io if CACHE_NONE is set to indicate Guest go direct I/O path. As Guest could also go direct I/O through regular path you have mentioned above.

fi->direct_io is needed so that server can *enforce* direct I/O on guest. If guest applications are not doing direct I/O, without fi->direct_io, you can't force client to do direct I/O.

I get yout point, it seems nothing wrong with it, but it indeed made me puzzled once. Maybe we'd better wait for some response from real user in the future.

Thanks,
Jun

Quoted 3 lines

Vivek

author
#16 lore raw

Re: [Virtio-fs] [PATCH RESEND][RFC] virtiofsd: do not fall back to buffer io when cache=auto/always

In reply to Vivek Goyal
Quoted 43 lines · Vivek Goyal
Quoted 16 lines · piaojun
Quoted 13 lines · Vivek Goyal
Quoted 7 lines · Vivek Goyal
Quoted 2 lines · piaojun

When O_DIRECT flags is set by Guest, virtiofsd will open file with O_DIRECT, but unset 'fi->direct_io' which makes Guest go buffer io path.

Where is virtiofsd unset fi->direct_io flag? All I can see is that it sets this flag if cache=none?

Ok, I looked at it little bit more closely and I don't think it is a problem. With cache=always/auto, we don't set fi->direct_io. That means server is not enforcing a direct I/O policy on client.

Quoted 8 lines
Quoted 3 lines

Now if client decides to direct I/O by opening file with O_DIRECT, it will still take direct I/O path and call fuse_direct_IO(). Look at fuse_file_aops->fuse_direct_IO().

The problem is that when Guest open with O_DIRECT, the Host won't set open_flags with FOPEN_DIRECT_IO if CACHE_AUTO is set. It makes Guest goes into fuse_cache_write_iter() rather than fuse_direct_write_iter(). In other words, the Guest's I/O mode will be changed by virtiofsd.

I think there are two direct I/O paths. One is where server requests that direct I/O be done on the file and in that case FOPEN_DIRECT_IO is set.

Other case is where client initiates direct I/O and in that case FOPEN_DIRECT_IO is not set and we take regular path which is capable of doing both direct and buffered I/O. For example, for reads we do following.

fuse_file_read_iter()
fuse_cache_read_iter()

      generic_file_read_iter()
	mapping->a_ops->direct_IO() <--- (If iocb->ki_flags & IOCB_DIRECT)
	  fuse_direct_IO()

Similary for write path.

Quoted 9 lines

fuse_file_write_iter()
fuse_cache_write_iter()

     generic_file_direct_write() <----- (If iocb->ki_flags & IOCB_DIRECT)
        mapping->a_ops->direct_IO()
	   fuse_direct_IO()

Can you give it a try. I don't think we have the problem what you are describing.

I make a try as you said, and it indeed goes direct path in guest even if cache=always. So my doubt has been cleared.

Thanks,
Jun