virtiofsd fixes
cover virtiofsd fixes
Hi,
This set contains a few fixes against virtiofsd.
Patch 1 fixes dio read hang.
Patch 2 adds ns support for file timestamp in stat(2).
Patch 3 fixes a problem found by generic/413 which shows vhost-user backend doesn't include all memory mappings in use.
Patch 4 uses fallocate(2) instead to support more modes.
thanks,
liubo
Eryu Guan (1):
virtiofsd: send reply correctly on read failure
Jiufei Xue (1):
virtiofsd: support nanosecond resolution for file timestamp
Xiaoguang Wang (2):
virtiofsd: use file-backend memory region for virtiofsd's cache area
virtiofsd: use fallocate(2) instead posix_fallocate(3)
configure | 16 +++++++++++ contrib/virtiofsd/fuse_lowlevel.c | 4 +-- contrib/virtiofsd/fuse_misc.h | 1 + contrib/virtiofsd/fuse_virtio.c | 12 ++++++-- contrib/virtiofsd/passthrough_ll.c | 11 ++----- hw/virtio/vhost-user-fs.c | 59 +++++++++++++++++++++++++++++++------- 6 files changed, 80 insertions(+), 23 deletions(-)
Signature · 1 line
1.8.3.1
PATCH 1/4 virtiofsd: send reply correctly on read failure
From: Eryu Guan <[email protected]>
Currently when a lo_read() operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.
This is easily triggered by a direct read with non-aligned length.
Fix it by detecting preadv(2) error in virtio_send_data_iov(), and teaching fuse_reply_data() to reply error on error case.
--- contrib/virtiofsd/fuse_lowlevel.c | 4 ++-- contrib/virtiofsd/fuse_virtio.c | 12 +++++++++--- 2 files changed, 11 insertions(+), 5 deletions(-)
contrib/virtiofsd/fuse_lowlevel.c +2 -2
@@ -524,11 +524,11 @@ int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, out.error =0 ; res =fuse_send_data_iov (req->se, req->ch, iov,1 , bufv, flags); -if (res <=0 ) { +if (res >=0 ) {fuse_free_req (req);return res; }else { -return fuse_reply_err (req, res); +return fuse_reply_err (req, -res); } }
contrib/virtiofsd/fuse_virtio.c +9 -3
@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, ret =preadv (buf->buf[0 ].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0 ].pos);if (se->debug) -fprintf (stderr,"%s: preadv_res=%d len=%zd\n" , - __func__, ret, len); +fprintf (stderr,"%s: preadv_res=%d(%s) len=%zd\n" , + __func__, ret,strerror (errno), len); +if (ret ==-1 ) { + ret = -errno; +free (in_sg_cpy); + goto err; + }if (ret < len && ret) {if (se->debug)fprintf (stderr,"%s: ret < len\n" , __func__);
@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,vu_queue_notify (&se->virtio_dev->dev, q); err: - ch->qi->reply_sent = true; +if (!ret) + ch->qi->reply_sent = true;return ret; }
Signature · 1 line
1.8.3.1
Re: [Virtio-fs] [PATCH 1/4] virtiofsd: send reply correctly on read failure
Quoted 10 lines · * Liu Bo ([email protected]) wrote:
From: Eryu Guan <[email protected]>
Currently when a
lo_read()operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.This is easily triggered by a direct read with non-aligned length.
Fix it by detecting preadv(2) error in
virtio_send_data_iov(), and teachingfuse_reply_data()to reply error on error case.
Thank you for spotting this.
Quoted 12 lines
Reviewed-by Liu Bo <[email protected]>Signed-off-by Eryu Guan <[email protected]>--- contrib/virtiofsd/fuse_lowlevel.c | 4 ++-- contrib/virtiofsd/fuse_virtio.c | 12 +++++++++--- 2 files changed, 11 insertions(+), 5 deletions(-)contrib/virtiofsd/fuse_lowlevel.c +0
-0
contrib/virtiofsd/fuse_lowlevel.c int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, · line 527
Quoted 5 lines
contrib/virtiofsd/fuse_lowlevel.c +1
-1@@ -524,11 +524,11 @@ int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, out.error =0 ; res =fuse_send_data_iov (req->se, req->ch, iov,1 , bufv, flags); -if (res <=0 ) { +if (res >=0 ) {
I need to go and ask the upstream fuse list about this; it's not clear to me what fuse_send_data_iov is supposed to return; let me clarify that and get back to you.
Dave
Quoted 45 lines
contrib/virtiofsd/fuse_lowlevel.c +1
-1fuse_free_req (req);return res; }else { -return fuse_reply_err (req, res); +return fuse_reply_err (req, -res); } }contrib/virtiofsd/fuse_virtio.c +9
-3@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, ret =preadv (buf->buf[0 ].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0 ].pos);if (se->debug) -fprintf (stderr,"%s: preadv_res=%d len=%zd\n" , - __func__, ret, len); +fprintf (stderr,"%s: preadv_res=%d(%s) len=%zd\n" , + __func__, ret,strerror (errno), len); +if (ret ==-1 ) { + ret = -errno; +free (in_sg_cpy); + goto err; + }if (ret < len && ret) {if (se->debug)fprintf (stderr,"%s: ret < len\n" , __func__);@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,vu_queue_notify (&se->virtio_dev->dev, q); err: - ch->qi->reply_sent = true; +if (!ret) + ch->qi->reply_sent = true;return ret; }Signature · 6 lines
1.8.3.1 _______________________________________________ Virtio-fs mailing list [email protected] https://www.redhat.com/mailman/listinfo/virtio-fs
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Virtio-fs] [PATCH 1/4] virtiofsd: send reply correctly on read failure
Quoted 28 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 10 lines · * Liu Bo ([email protected]) wrote:
From: Eryu Guan <[email protected]>
Currently when a
lo_read()operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.This is easily triggered by a direct read with non-aligned length.
Fix it by detecting preadv(2) error in
virtio_send_data_iov(), and teachingfuse_reply_data()to reply error on error case.Thank you for spotting this.
Quoted 14 lines
Reviewed-by Liu Bo <[email protected]>Signed-off-by Eryu Guan <[email protected]>--- contrib/virtiofsd/fuse_lowlevel.c | 4 ++-- contrib/virtiofsd/fuse_virtio.c | 12 +++++++++--- 2 files changed, 11 insertions(+), 5 deletions(-)contrib/virtiofsd/fuse_lowlevel.c +0
-0@@ -524,11 +524,11 @@ int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, out.error =0 ;
Quoted 7 lines
contrib/virtiofsd/fuse_lowlevel.c int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, · line 527
Quoted 3 lines
contrib/virtiofsd/fuse_lowlevel.c +1
-1res =fuse_send_data_iov (req->se, req->ch, iov,1 , bufv, flags); -if (res <=0 ) { +if (res >=0 ) {I need to go and ask the upstream fuse list about this; it's not clear to me what
fuse_send_data_iovis supposed to return; let me clarify that and get back to you.
Miklos replied to me on the upstream fuse list:
https://sourceforge.net/p/fuse/mailman/fuse-devel/thread/CAOssrKdVViWewmh3nrfKkq6eaWNJ629AR7w90KG3XT2hV9_D1w%40mail.gmail.com/#msg36642807
so this comparison is actually already correct because a negative value means that something is wrong with the fuse connection (or in our case virtio) so we can't send an error reply anyway.
Quoted 17 lines
Dave
Quoted 15 lines
contrib/virtiofsd/fuse_lowlevel.c +1
-1fuse_free_req (req);return res; }else { -return fuse_reply_err (req, res); +return fuse_reply_err (req, -res); } }contrib/virtiofsd/fuse_virtio.c +0
-0@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, ret =preadv (buf->buf[0 ].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0 ].pos);
Quoted 7 lines
contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 339
Quoted 7 lines
contrib/virtiofsd/fuse_virtio.c +4
-2if (se->debug) -fprintf (stderr,"%s: preadv_res=%d len=%zd\n" , - __func__, ret, len); +fprintf (stderr,"%s: preadv_res=%d(%s) len=%zd\n" , + __func__, ret,strerror (errno), len); +if (ret ==-1 ) { + ret = -errno;
So this will need to be ret = errno to give a postive error
response to mean that the error was on the file side on the fuse side. (There's also another case below that where I set ret = EIO where it should be EIO I think)
Quoted 9 lines
Quoted 9 lines
contrib/virtiofsd/fuse_virtio.c +3
-0+free (in_sg_cpy); + goto err; + }if (ret < len && ret) {if (se->debug)fprintf (stderr,"%s: ret < len\n" , __func__);@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,vu_queue_notify (&se->virtio_dev->dev, q);
Quoted 4 lines
contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 388
Quoted 4 lines
contrib/virtiofsd/fuse_virtio.c +2
-1err: - ch->qi->reply_sent = true; +if (!ret) + ch->qi->reply_sent = true;
Yes, I think that's OK.
Dave
Quoted 10 lines
Quoted 10 lines
contrib/virtiofsd/fuse_virtio.c +0
-0return ret; }Signature · 6 lines
1.8.3.1 _______________________________________________ Virtio-fs mailing list [email protected] https://www.redhat.com/mailman/listinfo/virtio-fs
Quoted 7 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Virtio-fs] [PATCH 1/4] virtiofsd: send reply correctly on read failure
Quoted 37 lines · Dr. David Alan Gilbert
Quoted 28 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 10 lines · * Liu Bo ([email protected]) wrote:
From: Eryu Guan <[email protected]>
Currently when a
lo_read()operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.This is easily triggered by a direct read with non-aligned length.
Fix it by detecting preadv(2) error in
virtio_send_data_iov(), and teachingfuse_reply_data()to reply error on error case.Thank you for spotting this.
Quoted 14 lines
Reviewed-by Liu Bo <[email protected]>Signed-off-by Eryu Guan <[email protected]>--- contrib/virtiofsd/fuse_lowlevel.c | 4 ++-- contrib/virtiofsd/fuse_virtio.c | 12 +++++++++--- 2 files changed, 11 insertions(+), 5 deletions(-)contrib/virtiofsd/fuse_lowlevel.c +0
-0@@ -524,11 +524,11 @@ int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, out.error =0 ;Quoted 7 lines
contrib/virtiofsd/fuse_lowlevel.c int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, · line 527
Quoted 3 lines
contrib/virtiofsd/fuse_lowlevel.c +1
-1res =fuse_send_data_iov (req->se, req->ch, iov,1 , bufv, flags); -if (res <=0 ) { +if (res >=0 ) {I need to go and ask the upstream fuse list about this; it's not clear to me what
fuse_send_data_iovis supposed to return; let me clarify that and get back to you.
Quoted 7 lines
Miklos replied to me on the upstream fuse list:
https://sourceforge.net/p/fuse/mailman/fuse-devel/thread/CAOssrKdVViWewmh3nrfKkq6eaWNJ629AR7w90KG3XT2hV9_D1w%40mail.gmail.com/#msg36642807so this comparison is actually already correct because a negative value means that something is wrong with the fuse connection (or in our case virtio) so we can't send an error reply anyway.
I see, that makes sense.
Quoted 17 lines
Quoted 17 lines
Dave
Quoted 15 lines
contrib/virtiofsd/fuse_lowlevel.c +1
-1fuse_free_req (req);return res; }else { -return fuse_reply_err (req, res); +return fuse_reply_err (req, -res); } }contrib/virtiofsd/fuse_virtio.c +0
-0@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, ret =preadv (buf->buf[0 ].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0 ].pos);
Quoted 13 lines
Quoted 7 lines
contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 339
Quoted 7 lines
contrib/virtiofsd/fuse_virtio.c +4
-2if (se->debug) -fprintf (stderr,"%s: preadv_res=%d len=%zd\n" , - __func__, ret, len); +fprintf (stderr,"%s: preadv_res=%d(%s) len=%zd\n" , + __func__, ret,strerror (errno), len); +if (ret ==-1 ) { + ret = -errno;So this will need to be ret = errno to give a postive error
response to mean that the error was on the file side on the fuse side. (There's also another case below that where I set ret = EIO where it should be EIO I think)
OK, do you want me to fix both in the same patch?
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
contrib/virtiofsd/fuse_virtio.c +3
-0+free (in_sg_cpy); + goto err; + }if (ret < len && ret) {if (se->debug)fprintf (stderr,"%s: ret < len\n" , __func__);@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,vu_queue_notify (&se->virtio_dev->dev, q);
Quoted 6 lines
Quoted 4 lines
contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 388
Quoted 4 lines
contrib/virtiofsd/fuse_virtio.c +2
-1err: - ch->qi->reply_sent = true; +if (!ret) + ch->qi->reply_sent = true;Yes, I think that's OK.
Correct, this is the one causing fuse kernel side's hang.
thanks,
-liubo
Quoted 22 lines
Dave
Quoted 10 lines
Quoted 10 lines
contrib/virtiofsd/fuse_virtio.c +0
-0return ret; }Signature · 6 lines
1.8.3.1 _______________________________________________ Virtio-fs mailing list [email protected] https://www.redhat.com/mailman/listinfo/virtio-fsQuoted 7 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Virtio-fs] [PATCH 1/4] virtiofsd: send reply correctly on read failure
Quoted 79 lines · * Liu Bo ([email protected]) wrote:
Quoted 37 lines · Dr. David Alan Gilbert
Quoted 28 lines · * Dr. David Alan Gilbert ([email protected]) wrote:
Quoted 10 lines · * Liu Bo ([email protected]) wrote:
From: Eryu Guan <[email protected]>
Currently when a
lo_read()operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.This is easily triggered by a direct read with non-aligned length.
Fix it by detecting preadv(2) error in
virtio_send_data_iov(), and teachingfuse_reply_data()to reply error on error case.Thank you for spotting this.
Quoted 14 lines
Reviewed-by Liu Bo <[email protected]>Signed-off-by Eryu Guan <[email protected]>--- contrib/virtiofsd/fuse_lowlevel.c | 4 ++-- contrib/virtiofsd/fuse_virtio.c | 12 +++++++++--- 2 files changed, 11 insertions(+), 5 deletions(-)contrib/virtiofsd/fuse_lowlevel.c +0
-0@@ -524,11 +524,11 @@ int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, out.error =0 ;Quoted 7 lines
contrib/virtiofsd/fuse_lowlevel.c int fuse_reply_data(fuse_req_t req, struct fuse_bufvec *bufv, · line 527
Quoted 3 lines
contrib/virtiofsd/fuse_lowlevel.c +1
-1res =fuse_send_data_iov (req->se, req->ch, iov,1 , bufv, flags); -if (res <=0 ) { +if (res >=0 ) {I need to go and ask the upstream fuse list about this; it's not clear to me what
fuse_send_data_iovis supposed to return; let me clarify that and get back to you.Quoted 7 lines
Miklos replied to me on the upstream fuse list:
https://sourceforge.net/p/fuse/mailman/fuse-devel/thread/CAOssrKdVViWewmh3nrfKkq6eaWNJ629AR7w90KG3XT2hV9_D1w%40mail.gmail.com/#msg36642807so this comparison is actually already correct because a negative value means that something is wrong with the fuse connection (or in our case virtio) so we can't send an error reply anyway.
I see, that makes sense.
Quoted 17 lines
Quoted 17 lines
Dave
Quoted 15 lines
contrib/virtiofsd/fuse_lowlevel.c +1
-1fuse_free_req (req);return res; }else { -return fuse_reply_err (req, res); +return fuse_reply_err (req, -res); } }contrib/virtiofsd/fuse_virtio.c +0
-0@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, ret =preadv (buf->buf[0 ].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0 ].pos);Quoted 13 lines
Quoted 7 lines
contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 339
Quoted 7 lines
contrib/virtiofsd/fuse_virtio.c +4
-2if (se->debug) -fprintf (stderr,"%s: preadv_res=%d len=%zd\n" , - __func__, ret, len); +fprintf (stderr,"%s: preadv_res=%d(%s) len=%zd\n" , + __func__, ret,strerror (errno), len); +if (ret ==-1 ) { + ret = -errno;So this will need to be ret = errno to give a postive error
response to mean that the error was on the file side on the fuse side. (There's also another case below that where I set ret = EIO where it should be EIO I think)
Quoted 1 line
OK, do you want me to fix both in the same patch?
Yes please.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
contrib/virtiofsd/fuse_virtio.c +3
-0+free (in_sg_cpy); + goto err; + }if (ret < len && ret) {if (se->debug)fprintf (stderr,"%s: ret < len\n" , __func__);@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,vu_queue_notify (&se->virtio_dev->dev, q);
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
contrib/virtiofsd/fuse_virtio.c int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, · line 388
Quoted 4 lines
contrib/virtiofsd/fuse_virtio.c +2
-1err: - ch->qi->reply_sent = true; +if (!ret) + ch->qi->reply_sent = true;Yes, I think that's OK.
Correct, this is the one causing fuse kernel side's hang.
Thanks!
Dave (Note, I'm out for a week)
Quoted 25 lines
thanks,
-liuboQuoted 22 lines
Dave
Quoted 10 lines
Quoted 10 lines
contrib/virtiofsd/fuse_virtio.c +0
-0return ret; }Signature · 6 lines
1.8.3.1 _______________________________________________ Virtio-fs mailing list [email protected] https://www.redhat.com/mailman/listinfo/virtio-fsQuoted 7 lines
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK_______________________________________________
Virtio-fs mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/virtio-fs--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH v2 1/1 virtiofsd: send reply correctly on read failure
In reply to Liu BoCurrently when a lo_read() operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.
This is easily triggered by a direct read with non-aligned length.
Fix it by detecting preadv(2) error in virtio_send_data_iov(), and teaching fuse_reply_data() to reply error on error case.
---
v2:
- return postive error number on read error
- also return postive EIO in short read case
- drop "-1" -> "-EOPNOTSUPP" update, which should be in a sperate patch
contrib/virtiofsd/fuse_virtio.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)
contrib/virtiofsd/fuse_virtio.c +10 -4
@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, ret =preadv (buf->buf[0 ].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0 ].pos);if (se->debug) -fprintf (stderr,"%s: preadv_res=%d len=%zd\n" , - __func__, ret, len); +fprintf (stderr,"%s: preadv_res=%d(%s) len=%zd\n" , + __func__, ret,strerror (errno), len); +if (ret ==-1 ) { + ret = errno; +free (in_sg_cpy); + goto err; + }if (ret < len && ret) {if (se->debug)fprintf (stderr,"%s: ret < len\n" , __func__);
@@ -356,7 +361,7 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,if (ret != len) {if (se->debug)fprintf (stderr,"%s: ret!=len\n" , __func__); - ret = -EIO ; + ret =EIO ;free (in_sg_cpy); goto err; }
@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,vu_queue_notify (&se->virtio_dev->dev, q); err: - ch->qi->reply_sent = true; +if (ret ==0 ) + ch->qi->reply_sent = true;return ret; }
Signature · 1 line
2.14.4.44.g2045bb6
Re: [Virtio-fs] [PATCH v2] virtiofsd: send reply correctly on read failure
Quoted 11 lines · * Eryu Guan ([email protected]) wrote:
Currently when a
lo_read()operation fails, we don't send the failure back to fuse client, and read(2) operation from guest kernel would hang on waiting for the reply.This is easily triggered by a direct read with non-aligned length.
Fix it by detecting preadv(2) error in
virtio_send_data_iov(), and teachingfuse_reply_data()to reply error on error case.Reviewed-by Liu Bo <[email protected]>Signed-off-by Eryu Guan <[email protected]>
Thank you, I've added that to my dev tree.
Quoted 55 lines
---
v2:
- return postive error number on read error
- also return postive EIO in short read case
- drop "-1" -> "-EOPNOTSUPP" update, which should be in a sperate patchcontrib/virtiofsd/fuse_virtio.c | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-)contrib/virtiofsd/fuse_virtio.c +10
-4@@ -333,8 +333,13 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch, ret =preadv (buf->buf[0 ].fd, in_sg_ptr, in_sg_cpy_count, buf->buf[0 ].pos);if (se->debug) -fprintf (stderr,"%s: preadv_res=%d len=%zd\n" , - __func__, ret, len); +fprintf (stderr,"%s: preadv_res=%d(%s) len=%zd\n" , + __func__, ret,strerror (errno), len); +if (ret ==-1 ) { + ret = errno; +free (in_sg_cpy); + goto err; + }if (ret < len && ret) {if (se->debug)fprintf (stderr,"%s: ret < len\n" , __func__);@@ -356,7 +361,7 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,if (ret != len) {if (se->debug)fprintf (stderr,"%s: ret!=len\n" , __func__); - ret = -EIO ; + ret =EIO ;free (in_sg_cpy); goto err; }@@ -379,7 +384,8 @@ int virtio_send_data_iov(struct fuse_session *se, struct fuse_chan *ch,vu_queue_notify (&se->virtio_dev->dev, q); err: - ch->qi->reply_sent = true; +if (ret ==0 ) + ch->qi->reply_sent = true;return ret; }Signature · 6 lines
2.14.4.44.g2045bb6 _______________________________________________ Virtio-fs mailing list [email protected] https://www.redhat.com/mailman/listinfo/virtio-fs
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH 2/4 virtiofsd: support nanosecond resolution for file timestamp
In reply to Liu BoFrom: Jiufei Xue <[email protected]>
Define HAVE_STRUCT_STAT_ST_ATIM to 1 if st_atim' is member of struct stat' which means support nanosecond resolution for the file timestamp fields.
--- configure | 16 ++++++++++++++++ contrib/virtiofsd/fuse_misc.h | 1 + 2 files changed, 17 insertions(+)
configure +16 -0
@@ -5147,6 +5147,19 @@ if compile_prog "" "" ; then strchrnul=yes fi +######################################### +# check if we have st_atim + +st_atim=no +cat > $TMPC << EOF +#include <sys/stat.h> +#include <stddef.h> +int main(void) { return offsetof(struct stat, st_atim); } +EOF +if compile_prog "" "" ; then + st_atim=yes +fi + ########################################## # check if trace backend exists
@@ -6757,6 +6770,9 @@ fi if test "$strchrnul" = "yes" ; then echo "HAVE_STRCHRNUL=y" >> $config_host_mak fi +if test "$st_atim" = "yes" ; then + echo "HAVE_STRUCT_STAT_ST_ATIM=y" >> $config_host_mak +fi if test "$byteswap_h" = "yes" ; then echo "CONFIG_BYTESWAP_H=y" >> $config_host_mak fi
contrib/virtiofsd/fuse_misc.h +1 -0
@@ -7,6 +7,7 @@*/ #include <pthread.h> +#include "config-host.h" /* Versioned symbols cannot be used in some cases because it
Signature · 1 line
1.8.3.1
Re: [Virtio-fs] [PATCH 2/4] virtiofsd: support nanosecond resolution for file timestamp
Quoted 8 lines · * Liu Bo ([email protected]) wrote:
From: Jiufei Xue <[email protected]>
Define
HAVE_STRUCT_STAT_ST_ATIMto 1 ifst_atim' is member ofstruct stat' which means support nanosecond resolution for the file timestamp fields.Signed-off-by Jiufei Xue <[email protected]>Reviewed-by Joseph Qi <[email protected]>
Thanks, applied to my tree.
Quoted 58 lines
--- configure | 16 ++++++++++++++++ contrib/virtiofsd/fuse_misc.h | 1 + 2 files changed, 17 insertions(+)configure +16
-0@@ -5147,6 +5147,19 @@ if compile_prog "" "" ; then strchrnul=yes fi +######################################### +# check if we have st_atim + +st_atim=no +cat > $TMPC << EOF +#include <sys/stat.h> +#include <stddef.h> +int main(void) { return offsetof(struct stat, st_atim); } +EOF +if compile_prog "" "" ; then + st_atim=yes +fi + ########################################## # check if trace backend exists@@ -6757,6 +6770,9 @@ fi if test "$strchrnul" = "yes" ; then echo "HAVE_STRCHRNUL=y" >> $config_host_mak fi +if test "$st_atim" = "yes" ; then + echo "HAVE_STRUCT_STAT_ST_ATIM=y" >> $config_host_mak +fi if test "$byteswap_h" = "yes" ; then echo "CONFIG_BYTESWAP_H=y" >> $config_host_mak ficontrib/virtiofsd/fuse_misc.h +1
-0@@ -7,6 +7,7 @@*/ #include <pthread.h> +#include "config-host.h" /* Versioned symbols cannot be used in some cases because it Signature · 6 lines
1.8.3.1 _______________________________________________ Virtio-fs mailing list [email protected] https://www.redhat.com/mailman/listinfo/virtio-fs
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH 3/4 virtiofsd: use file-backend memory region for virtiofsd's cache area
In reply to Liu BoFrom: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3, write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:
[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9
And virtiofsd crashed because vu_gpa_to_va() can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, so vu_gpa_to_va() will fail.
To fix this issue, we create this vhost cache area with a file backend memory area.
--- hw/virtio/vhost-user-fs.c | 59 +++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 49 insertions(+), 10 deletions(-)
hw/virtio/vhost-user-fs.c +49 -10
@@ -300,6 +300,35 @@ static bool vuf_guest_notifier_pending(VirtIODevice *vdev, int idx)return vhost_virtqueue_pending (&fs->vhost_dev, idx); } +static int vuf_get_tmpfile (size_t size) +{ +const char *tmpdir =g_get_tmp_dir (); +gchar *fname; +int fd; + + fname =g_strdup_printf ("%s/virtio-fs-cache-XXXXXX" , tmpdir); + fd =mkstemp (fname); +if (fd <0 ) { +fprintf (stderr,"%s: mkstemp(%s) failed\n" , __func__, fname); +g_free (fname); +return -1 ; + } + +/* +* Note: this temp file would be deleted until file is closed or +* process exits. +*/ +unlink (fname); +if (ftruncate (fd, size) ==-1 ) { +fprintf (stderr,"%s: ftruncate(%s) failed\n" , __func__, fname); +close (fd); + fd =-1 ; + } + +g_free (fname); +return fd; +} +static void vuf_device_realize (DeviceState *dev,Error **errp) {VirtIODevice *vdev =VIRTIO_DEVICE (dev);
@@ -349,16 +378,26 @@ static void vuf_device_realize(DeviceState *dev, Error **errp) "no smaller than the page size");return ; } -/* We need a region with some host memory, 'ram' is the easiest */ -memory_region_init_ram_nomigrate (&fs->cache,OBJECT (vdev), -"virtio-fs-cache" , - fs->conf.cache_size,NULL ); -/* -* But we don't actually want anyone reading/writing the raw -* area with no cache data. -*/ -mprotect (memory_region_get_ram_ptr (&fs->cache), fs->conf.cache_size, -PROT_NONE ); + +if (fs->conf.cache_size) { +int fd; + + fd =vuf_get_tmpfile (fs->conf.cache_size); +if (fd <0 ) { +error_setg (errp,"failed to initialize virtio-fs-cache" ); +return ; + } +memory_region_init_ram_from_fd (&fs->cache,OBJECT (vdev), +"virtio-fs-cache" , + fs->conf.cache_size, true, fd,NULL ); + +/* +* But we don't actually want anyone reading/writing the raw +* area with no cache data. +*/ +mprotect (memory_region_get_ram_ptr (&fs->cache), + fs->conf.cache_size,PROT_NONE ); + }if (!vhost_user_init (&fs->vhost_user, &fs->conf.chardev, errp)) {return ;
Signature · 1 line
1.8.3.1
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
Quoted 37 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3,write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9And virtiofsd crashed because
vu_gpa_to_va()can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, sovu_gpa_to_va()will fail.
Quoted 2 lines
To fix this issue, we create this vhost cache area with a file backend memory area.
Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?
It worries me a little exposing the area back to the daemon; the guest can write the BAR and change the mapping, I doubt anything would notice that (but also I doubt it happens much).
Dave
Quoted 90 lines
Reviewed-by Liu Bo <[email protected]>Signed-off-by Xiaoguang Wang <[email protected]>--- hw/virtio/vhost-user-fs.c | 59 +++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 49 insertions(+), 10 deletions(-)hw/virtio/vhost-user-fs.c +49
-10@@ -300,6 +300,35 @@ static bool vuf_guest_notifier_pending(VirtIODevice *vdev, int idx)return vhost_virtqueue_pending (&fs->vhost_dev, idx); } +static int vuf_get_tmpfile (size_t size) +{ +const char *tmpdir =g_get_tmp_dir (); +gchar *fname; +int fd; + + fname =g_strdup_printf ("%s/virtio-fs-cache-XXXXXX" , tmpdir); + fd =mkstemp (fname); +if (fd <0 ) { +fprintf (stderr,"%s: mkstemp(%s) failed\n" , __func__, fname); +g_free (fname); +return -1 ; + } + +/* +* Note: this temp file would be deleted until file is closed or +* process exits. +*/ +unlink (fname); +if (ftruncate (fd, size) ==-1 ) { +fprintf (stderr,"%s: ftruncate(%s) failed\n" , __func__, fname); +close (fd); + fd =-1 ; + } + +g_free (fname); +return fd; +} +static void vuf_device_realize (DeviceState *dev,Error **errp) {VirtIODevice *vdev =VIRTIO_DEVICE (dev);@@ -349,16 +378,26 @@ static void vuf_device_realize(DeviceState *dev, Error **errp) "no smaller than the page size");return ; } -/* We need a region with some host memory, 'ram' is the easiest */ -memory_region_init_ram_nomigrate (&fs->cache,OBJECT (vdev), -"virtio-fs-cache" , - fs->conf.cache_size,NULL ); -/* -* But we don't actually want anyone reading/writing the raw -* area with no cache data. -*/ -mprotect (memory_region_get_ram_ptr (&fs->cache), fs->conf.cache_size, -PROT_NONE ); + +if (fs->conf.cache_size) { +int fd; + + fd =vuf_get_tmpfile (fs->conf.cache_size); +if (fd <0 ) { +error_setg (errp,"failed to initialize virtio-fs-cache" ); +return ; + } +memory_region_init_ram_from_fd (&fs->cache,OBJECT (vdev), +"virtio-fs-cache" , + fs->conf.cache_size, true, fd,NULL ); + +/* +* But we don't actually want anyone reading/writing the raw +* area with no cache data. +*/ +mprotect (memory_region_get_ram_ptr (&fs->cache), + fs->conf.cache_size,PROT_NONE ); + }if (!vhost_user_init (&fs->vhost_user, &fs->conf.chardev, errp)) {return ;Signature · 6 lines
1.8.3.1 _______________________________________________ Virtio-fs mailing list [email protected] https://www.redhat.com/mailman/listinfo/virtio-fs
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
Quoted 38 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3,write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9And virtiofsd crashed because
vu_gpa_to_va()can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, sovu_gpa_to_va()will fail.
Quoted 7 lines
Quoted 2 lines
To fix this issue, we create this vhost cache area with a file backend memory area.
Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Quoted 4 lines
It worries me a little exposing the area back to the daemon; the guest can write the BAR and change the mapping, I doubt anything would notice that (but also I doubt it happens much).
If two virtiofsd processes are involved then it's even harder since they do not have up-to-date access the other's DAX window.
Stefan
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
Quoted 47 lines · Stefan Hajnoczi
Quoted 38 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3,write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9And virtiofsd crashed because
vu_gpa_to_va()can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, sovu_gpa_to_va()will fail.Quoted 7 lines
Quoted 2 lines
To fix this issue, we create this vhost cache area with a file backend memory area.
Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?
Quoted 5 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Can you please elaborate how the buffer is outside guest RAM? Is it also via direct IO?
Quoted 8 lines
Quoted 4 lines
It worries me a little exposing the area back to the daemon; the guest can write the BAR and change the mapping, I doubt anything would notice that (but also I doubt it happens much).
If two virtiofsd processes are involved then it's even harder since they do not have up-to-date access the other's DAX window.
In case of direct IO, kvm is able to make sure that guest's dax mapping is sync'd with the underlying host mmap region, isn't it?
thanks,
-liubo
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3,write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9And virtiofsd crashed because
vu_gpa_to_va()can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, sovu_gpa_to_va()will fail.Quoted 2 lines
To fix this issue, we create this vhost cache area with a file backend memory area.
Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?
Quoted 8 lines
Quoted 5 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Can you please elaborate how the buffer is outside guest RAM? Is it also via direct IO?
The DAX window is a PCI BAR on the virtio-fs PCI device. It is not guest RAM.
vhost-user only shares guest RAM with the vhost-user device backend process (virtiofsd). Therefore virtiofsd does not have access to the contents of the DAX window.
This only happens when the virtio-fs file system is mounted with the "dax" option.
Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
It worries me a little exposing the area back to the daemon; the guest can write the BAR and change the mapping, I doubt anything would notice that (but also I doubt it happens much).
If two virtiofsd processes are involved then it's even harder since they do not have up-to-date access the other's DAX window.
In case of direct IO, kvm is able to make sure that guest's dax mapping is sync'd with the underlying host mmap region, isn't it?
See my explanation above. In theory a virtiofsd could keep track of the mapping region by performing the same mmap(2) system calls as the QEMU process based on the FUSE_SETUPMAPPING requests (currently only QEMU does the mmaps). What I'm saying here is that it gets even harder when multiple virtiofsd processes are involved because they do not have access to each other's DAX windows.
Stefan
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3,write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9And virtiofsd crashed because
vu_gpa_to_va()can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, sovu_gpa_to_va()will fail.Quoted 2 lines
To fix this issue, we create this vhost cache area with a file backend memory area.
Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Stefan,
Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?
This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.
Thanks
Vivek
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
Quoted 57 lines · Vivek Goyal
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3,write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9And virtiofsd crashed because
vu_gpa_to_va()can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, sovu_gpa_to_va()will fail.Quoted 2 lines
To fix this issue, we create this vhost cache area with a file backend memory area.
Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Stefan,
Quoted 7 lines
Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?
This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.
Yes, interesting idea!
When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.
Implementing this is a little tricky because the libvhost-user code probably fails before fuse_lowlevel.c is able to parse the FUSE request header. It will require reworking libvhost-user and fuse_virtio.c code, I think.
Stefan
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
Quoted 68 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 57 lines · Vivek Goyal
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3,write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9And virtiofsd crashed because
vu_gpa_to_va()can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, sovu_gpa_to_va()will fail.Quoted 2 lines
To fix this issue, we create this vhost cache area with a file backend memory area.
Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Stefan,
Quoted 7 lines
Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?
This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.
Yes, interesting idea!
Quoted 8 lines
When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.
Implementing this is a little tricky because the libvhost-user code probably fails before
fuse_lowlevel.cis able to parse the FUSE request header. It will require reworking libvhost-user andfuse_virtio.ccode, I think.
Yes, this doesn't look too bad; I need to tweak
vu_queue_pop->vu_queue_map_desc->virtqueue_map_desc
to give back a list of unmappable parts of the iovec (assuming that the first few elements of the iovec are mappable then the rest aren't and not allowing weird mixes).
One thing that worries me a bit is that if we do a read() or write() in the qemu code, it might block on the mmap'd backing file.
Dave
Quoted 1 line
Stefan
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
Quoted 78 lines · Dr. David Alan Gilbert
Quoted 68 lines · * Stefan Hajnoczi ([email protected]) wrote:
Quoted 57 lines · Vivek Goyal
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3,write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9And virtiofsd crashed because
vu_gpa_to_va()can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, sovu_gpa_to_va()will fail.Quoted 2 lines
To fix this issue, we create this vhost cache area with a file backend memory area.
Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Stefan,
Quoted 7 lines
Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?
This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.
Yes, interesting idea!
Quoted 8 lines
When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.
Implementing this is a little tricky because the libvhost-user code probably fails before
fuse_lowlevel.cis able to parse the FUSE request header. It will require reworking libvhost-user andfuse_virtio.ccode, I think.
Quoted 8 lines
Yes, this doesn't look too bad; I need to tweak
vu_queue_pop->vu_queue_map_desc->virtqueue_map_desc
to give back a list of unmappable parts of the iovec (assuming that the first few elements of the iovec are mappable then the rest aren't and not allowing weird mixes).One thing that worries me a bit is that if we do a
read()orwrite()in the qemu code, it might block on the mmap'd backing file.
Even a memcpy (no read()/write()) could block if the page is not resident :(. We have to expect that this operation could block.
Stefan
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
In reply to Stefan HajnocziQuoted 63 lines · Stefan Hajnoczi
Quoted 62 lines · Vivek Goyal
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3,write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9And virtiofsd crashed because
vu_gpa_to_va()can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, sovu_gpa_to_va()will fail.Quoted 2 lines
To fix this issue, we create this vhost cache area with a file backend memory area.
Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Stefan,
Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?
Quoted 9 lines
Quoted 2 lines
This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.
Yes, interesting idea!
When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.
It is probably not easy to do.
Imagine the following case,
// foo1 is on a dax virtiofs, foo2 is on a nondax virtiofs
p = mmap(foo1, ...);write(foo2, p, ...);
virtiofsd where foo2 is using needs to interpret gpa from virtiofs where foo1 exists along with fd being foo1, but a write fuse_req doesn't have foo1's fd.
And are you suggesting that qemu goes to read the data on gpa and returns via vhost-user message? or let this virtiofsd (foo2) do mmap on foo1 directly?
thanks,
-liubo
Quoted 6 lines
Implementing this is a little tricky because the libvhost-user code probably fails before
fuse_lowlevel.cis able to parse the FUSE request header. It will require reworking libvhost-user andfuse_virtio.ccode, I think.Stefan
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
On Fri, May 17, 2019 at 07:28:22PM -0700, Liu Bo wrote:
[..]
Quoted 30 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Stefan,
Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?
Quoted 9 lines
Quoted 2 lines
This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.
Yes, interesting idea!
When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.
It is probably not easy to do.
Imagine the following case,
// foo1 is on a dax virtiofs, foo2 is on a nondax virtiofsp =
mmap(foo1, ...);write(foo2, p, ...);
Quoted 7 lines
virtiofsd where foo2 is using needs to interpret gpa from virtiofs where foo1 exists along with fd being foo1, but a write
fuse_reqdoesn't have foo1's fd.And are you suggesting that qemu goes to read the data on gpa and returns via vhost-user message? or let this virtiofsd (foo2) do mmap on foo1 directly?
Hi Liu Bo,
Not sure I understand the question. Assuming foo2 virtiofs instances is mounted with cache=none, write(foo2), will trigger FUSE_WRITE request to daemon with source address being in dax window of first virtiofs instance.
This request should be forwarded to qemu which can just read from foo1 mmaped area in qemu address space and write to foo2. I am assuming before FUSE_WRITE comes in, mapping for foo1 has already been establied.
IOW, foo1 is already mmaped in qemu address space so there is no need to pass fd of foo1 in FUSE_WRITE(foo2) request. Did I miss something?
Vivek
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
Quoted 48 lines · Vivek Goyal
On Fri, May 17, 2019 at 07:28:22PM -0700, Liu Bo wrote:
[..]
Quoted 30 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Stefan,
Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?
Quoted 9 lines
Quoted 2 lines
This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.
Yes, interesting idea!
When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.
It is probably not easy to do.
Imagine the following case,
// foo1 is on a dax virtiofs, foo2 is on a nondax virtiofsp =
mmap(foo1, ...);write(foo2, p, ...);Quoted 7 lines
virtiofsd where foo2 is using needs to interpret gpa from virtiofs where foo1 exists along with fd being foo1, but a write
fuse_reqdoesn't have foo1's fd.And are you suggesting that qemu goes to read the data on gpa and returns via vhost-user message? or let this virtiofsd (foo2) do mmap on foo1 directly?
Hi Liu Bo,
Not sure I understand the question. Assuming foo2 virtiofs instances is mounted with cache=none, write(foo2), will trigger
FUSE_WRITErequest to daemon with source address being in dax window of first virtiofs instance.
Quoted 4 lines
This request should be forwarded to qemu which can just read from foo1 mmaped area in qemu address space and write to foo2. I am assuming before
FUSE_WRITEcomes in, mapping for foo1 has already been establied.
Bofore sending any message to qemu, fv_queue_thread() will fail at vu_queue_pop()->vu_queue_map_desc()->virtqueue_map_desc()->vu_gpa_to_va(), because virtiofsd (where foo2 is on) doesn't have the information of foo1's mapping within its address space.
It could be reproduced by fstests/generic/413.
thanks,
-liubo
Quoted 6 lines
IOW, foo1 is already mmaped in qemu address space so there is no need to pass fd of foo1 in
FUSE_WRITE(foo2)request. Did I miss something?Vivek
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
Quoted 54 lines · * Liu Bo ([email protected]) wrote:
Quoted 48 lines · Vivek Goyal
On Fri, May 17, 2019 at 07:28:22PM -0700, Liu Bo wrote:
[..]
Quoted 30 lines
Quoted 12 lines
Quoted 12 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Stefan,
Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?
Quoted 9 lines
Quoted 2 lines
This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.
Yes, interesting idea!
When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.
It is probably not easy to do.
Imagine the following case,
// foo1 is on a dax virtiofs, foo2 is on a nondax virtiofsp =
mmap(foo1, ...);write(foo2, p, ...);Quoted 7 lines
virtiofsd where foo2 is using needs to interpret gpa from virtiofs where foo1 exists along with fd being foo1, but a write
fuse_reqdoesn't have foo1's fd.And are you suggesting that qemu goes to read the data on gpa and returns via vhost-user message? or let this virtiofsd (foo2) do mmap on foo1 directly?
Hi Liu Bo,
Not sure I understand the question. Assuming foo2 virtiofs instances is mounted with cache=none, write(foo2), will trigger
FUSE_WRITErequest to daemon with source address being in dax window of first virtiofs instance.Quoted 4 lines
This request should be forwarded to qemu which can just read from foo1 mmaped area in qemu address space and write to foo2. I am assuming before
FUSE_WRITEcomes in, mapping for foo1 has already been establied.
Quoted 4 lines
Bofore sending any message to qemu,
fv_queue_thread()will fail at vu_queue_pop()->vu_queue_map_desc()->virtqueue_map_desc()->vu_gpa_to_va(), because virtiofsd (where foo2 is on) doesn't have the information of foo1's mapping within its address space.
The trick is to bounce the problem back to qemu; as soon as the mapping fails I add a flag to say the mapping failed; then that causes the request to be sent to qemu via a slave command - and qemu has all of the mappings so can read/write to all of the fs's.
Dave
Quoted 11 lines
It could be reproduced by fstests/generic/413.
thanks,
-liuboQuoted 6 lines
IOW, foo1 is already mmaped in qemu address space so there is no need to pass fd of foo1 in
FUSE_WRITE(foo2)request. Did I miss something?Vivek
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Virtio-fs] [PATCH 3/4] virtiofsd: use file-backend memory region for virtiofsd's cache area
In reply to Liu BoQuoted 82 lines · * Liu Bo ([email protected]) wrote:
Quoted 63 lines · Stefan Hajnoczi
Quoted 62 lines · Vivek Goyal
Quoted 49 lines · Stefan Hajnoczi
Quoted 48 lines · Liu Bo
Quoted 47 lines · Stefan Hajnoczi
Quoted 45 lines · Dr. David Alan Gilbert
Quoted 37 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
When running xfstests test case generic/413, we found such issue:
1, create a file in one virtiofsd mount point with dax enabled 2, mmap this file, get virtual addr: A
3,write(fd, A, len), here fd comes from another file in another
virtiofsd mount point without dax enabled, also note here write(2) is direct io.
4, this direct io will hang forever, because the virtiofsd has crashed.
Here is the stack:[ 247.166276] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 247.167171] t_mmap_dio D 0 2335 2102 0x00000000 [ 247.168006] Call Trace: [ 247.169067] ? __schedule+0x3d0/0x830 [ 247.170219] schedule+0x32/0x80 [ 247.171328] schedule_timeout+0x1e2/0x350 [ 247.172416] ? fuse_direct_io+0x2e5/0x6b0 [fuse] [ 247.173516] wait_for_completion+0x123/0x190 [ 247.174593] ? wake_up_q+0x70/0x70 [ 247.175640] fuse_direct_IO+0x265/0x310 [fuse] [ 247.176724] generic_file_read_iter+0xaa/0xd20 [ 247.177824] fuse_file_read_iter+0x81/0x130 [fuse] [ 247.178938] ? fuse_simple_request+0x104/0x1b0 [fuse] [ 247.180041] ? fuse_fsync_common+0xad/0x240 [fuse] [ 247.181136] __vfs_read+0x108/0x190 [ 247.181930] vfs_read+0x91/0x130 [ 247.182671] ksys_read+0x52/0xc0 [ 247.183454] do_syscall_64+0x55/0x170 [ 247.184200] entry_SYSCALL_64_after_hwframe+0x44/0xa9And virtiofsd crashed because
vu_gpa_to_va()can not handle guest physical address correctly. For a memory mapped area in dax mode, indeed the page for this area points virtiofsd's cache area, or rather virtio pci device's cache bar. In qemu, currently this cache bar is implemented with an anonymous memory and will not pass this cache bar's address info to vhost-user backend, sovu_gpa_to_va()will fail.Quoted 2 lines
To fix this issue, we create this vhost cache area with a file backend memory area.
Thanks,
I know there was another case of the daemon trying to access the
buffer that Stefan and Vivek hit, but fixed by persuading the kernel not to do it; Stefan/Vivek: What do you think?Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
That case happened with cache=none and the dax mount option.
The general problem is when FUSE_READ/FUSE_WRITE is sent and the buffer is outside guest RAM.
Stefan,
Can this be emulated by sending a request to qemu? If virtiofsd can detect that source/destination of READ/WRITE is not guest RAM, can it forward message to qemu to do this operation (which has access to all the DAX windows)?
Quoted 9 lines
Quoted 2 lines
This probably will mean introducing new messages like setupmapping/removemapping messages between virtiofsd/qemu.
Yes, interesting idea!
When virtiofsd is unable to map the virtqueue iovecs due to addresses outside guest RAM, it could forward READ/WRITE requests to QEMU along with the file descriptor. It would be slow but fixes the problem.
It is probably not easy to do.
Imagine the following case,
// foo1 is on a dax virtiofs, foo2 is on a nondax virtiofsp =
mmap(foo1, ...);write(foo2, p, ...);
Quoted 7 lines
virtiofsd where foo2 is using needs to interpret gpa from virtiofs where foo1 exists along with fd being foo1, but a write
fuse_reqdoesn't have foo1's fd.And are you suggesting that qemu goes to read the data on gpa and returns via vhost-user message? or let this virtiofsd (foo2) do mmap on foo1 directly?
I have a patchset I'm just tidying up that passes this case back to qemu to handle. I intend to post it by the end of the week.
What it does is that when the virtiofsd receives a read/write to an area of memory that it doesn't have a mapping for, it forms a new slave message back to qemu together with the fd asking qemu to read/write at the given GPA. Then it's upto QEMU to deal with it.
That should work even if there are two separate daemons.
It's not a pretty solution; but I think it should work.
Dave
Quoted 9 lines
thanks,
-liuboQuoted 6 lines
Implementing this is a little tricky because the libvhost-user code probably fails before
fuse_lowlevel.cis able to parse the FUSE request header. It will require reworking libvhost-user andfuse_virtio.ccode, I think.Stefan
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
PATCH 4/4 virtiofsd: use fallocate(2) instead posix_fallocate(3)
In reply to Liu BoFrom: Xiaoguang Wang <[email protected]>
This is because posix_fallocate(3) does not support FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE well, so this change will be ok.
--- contrib/virtiofsd/passthrough_ll.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)
contrib/virtiofsd/passthrough_ll.c +3 -8
@@ -1568,15 +1568,10 @@ static void lo_fallocate(fuse_req_t req, fuse_ino_t ino, int mode, off_t offset, off_t length, struct fuse_file_info *fi) { int err; - (void ) ino; - -if (mode) { -fuse_reply_err (req,EOPNOTSUPP ); -return ; - } - err =posix_fallocate (lo_fi_fd (req, fi), offset, - length); + err =fallocate (lo_fi_fd (req, fi), mode, offset, length); +if (err <0 ) + err = errno;fuse_reply_err (req, err); }
Signature · 1 line
1.8.3.1
Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)
Quoted 8 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
This is because
posix_fallocate(3)does not supportFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supportsFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLEwell, so this change will be ok.Signed-off-by Xiaoguang Wang <[email protected]>
We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?
Miklos: Do you know?
Dave
Quoted 34 lines
--- contrib/virtiofsd/passthrough_ll.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-)contrib/virtiofsd/passthrough_ll.c +3
-8@@ -1568,15 +1568,10 @@ static void lo_fallocate(fuse_req_t req, fuse_ino_t ino, int mode, off_t offset, off_t length, struct fuse_file_info *fi) { int err; - (void ) ino; - -if (mode) { -fuse_reply_err (req,EOPNOTSUPP ); -return ; - } - err =posix_fallocate (lo_fi_fd (req, fi), offset, - length); + err =fallocate (lo_fi_fd (req, fi), mode, offset, length); +if (err <0 ) + err = errno;fuse_reply_err (req, err); }Signature · 6 lines
1.8.3.1 _______________________________________________ Virtio-fs mailing list [email protected] https://www.redhat.com/mailman/listinfo/virtio-fs
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
This is because
posix_fallocate(3)does not supportFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supportsFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLEwell, so this change will be ok.
Quoted 4 lines
Quoted 1 line
Signed-off-by Xiaoguang Wang <[email protected]>We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?
The patch looks good to me.
Fuse (the kernel part) supports FALLOC_FL_KEEP_SIZE and FALLOC_FL_PUNCH_HOLE, but returns EOPNOTSUPP for any other flag.
Even if, at a later time, fuse starts supporting additional fallocate flags, then the passthrough implementation calling fallocate(2) should be fine.
Thanks,
Miklos
Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)
Quoted 16 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
This is because
posix_fallocate(3)does not supportFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supportsFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLEwell, so this change will be ok.Quoted 4 lines
Quoted 1 line
Signed-off-by Xiaoguang Wang <[email protected]>We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?
Quoted 8 lines
The patch looks good to me.
Fuse (the kernel part) supports
FALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLE, but returns EOPNOTSUPP for any other flag.Even if, at a later time, fuse starts supporting additional fallocate flags, then the passthrough implementation calling fallocate(2) should be fine.
OK, so then I'll take this patch; would it make sense for it to be sent to upstream fuse?
Dave
Quoted 2 lines
Thanks,
Miklos
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)
Quoted 23 lines · Dr. David Alan Gilbert
Quoted 21 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
This is because
posix_fallocate(3)does not supportFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supportsFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLEwell, so this change will be ok.Quoted 4 lines
Quoted 1 line
Signed-off-by Xiaoguang Wang <[email protected]>We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?The patch looks good to me.
Fuse (the kernel part) supports
FALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLE, but returns EOPNOTSUPP for any other flag.
Quoted 6 lines
Quoted 3 lines
Even if, at a later time, fuse starts supporting additional fallocate flags, then the passthrough implementation calling fallocate(2) should be fine.
OK, so then I'll take this patch; would it make sense for it to be sent to upstream fuse?
It would have to be wrapped in #ifdef HAVE_FALLOCATE to make it
portable. Otherwise I don't see why not.
Thanks,
Miklos
Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)
Quoted 29 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 27 lines · Dr. David Alan Gilbert
Quoted 24 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
This is because
posix_fallocate(3)does not supportFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supportsFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLEwell, so this change will be ok.Quoted 4 lines
Quoted 1 line
Signed-off-by Xiaoguang Wang <[email protected]>We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?The patch looks good to me.
Fuse (the kernel part) supports
FALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLE, but returns EOPNOTSUPP for any other flag.Even if, at a later time, fuse starts supporting additional fallocate flags, then the passthrough implementation calling fallocate(2) should be fine.
Quoted 5 lines
Quoted 2 lines
OK, so then I'll take this patch; would it make sense for it to be sent to upstream fuse?
It would have to be wrapped in #ifdef
HAVE_FALLOCATEto make it
portable. Otherwise I don't see why not.
OK, I'll take it for now.
Liu Bo: Can you make it portable please and post it upstream on the fuse list?
Dave
Quoted 2 lines
Thanks,
Miklos
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Re: [Virtio-fs] [PATCH 4/4] virtiofsd: use fallocate(2) instead posix_fallocate(3)
Quoted 33 lines · Dr. David Alan Gilbert
Quoted 32 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 29 lines · Dr. David Alan Gilbert
Quoted 16 lines · * Miklos Szeredi ([email protected]) wrote:
Quoted 9 lines · Dr. David Alan Gilbert
Quoted 7 lines · * Liu Bo ([email protected]) wrote:
From: Xiaoguang Wang <[email protected]>
This is because
posix_fallocate(3)does not supportFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLE. Our underlying host filesystem is ext4 and ext4 supportsFALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLEwell, so this change will be ok.Quoted 4 lines
Quoted 1 line
Signed-off-by Xiaoguang Wang <[email protected]>We need to check what 'fuse' expects - is it defined what
fallocate features it has, and what the semantics are?Quoted 8 lines
The patch looks good to me.
Fuse (the kernel part) supports
FALLOC_FL_KEEP_SIZEandFALLOC_FL_PUNCH_HOLE, but returns EOPNOTSUPP for any other flag.Even if, at a later time, fuse starts supporting additional fallocate flags, then the passthrough implementation calling fallocate(2) should be fine.
OK, so then I'll take this patch; would it make sense for it to be sent to upstream fuse?
Quoted 8 lines
Quoted 2 lines
It would have to be wrapped in #ifdef
HAVE_FALLOCATEto make it
portable. Otherwise I don't see why not.OK, I'll take it for now.
Liu Bo: Can you make it portable please and post it upstream on the fuse list?
No problem, will do it.
thanks,
-liubo